tables, table.unpack and nil values

Ketho · 05-07-12, 05:56 PM

I'm confused why the first example does print the nil values, but the latter does not.
Aren't these tables the "same"?
1. Lua Code:
```
local t = {nil, nil, 3, 7, nil, 4} 
 
print(#t) -- 6
print(unpack(t)) -- nil, nil, 3, 7, nil, 4
```
2. Lua Code:
```
local t = {}
t[3] = 3
t[4] = 7
t[6] = 4
 
print(#t) -- 0
print(unpack(t)) -- nil
```
I was always thinking that table.unpack can't iterate over "holes" in the same way as an ipairs-loop
(I'm not a programmer, and I don't know what's the proper terminology)

On another note, I'm also wondering about this (4 months old) post from Lur,
concerning tables with a nil value at the end of a table

Originally Posted by Lur

Thanks a lot. But why

Lua Code:

local t1 = {1, 2, nil, "wwew", nil, 3} -- #t1 = 6
local t2 = {1, 2, nil, "wwew", 3, nil} -- #t2 = 2

What's wrong with all of that? Why tables with a nil as the last value are truncated to the first nil occurrence while with any non-nil end are counted correct?

Foxlit · 05-07-12, 07:05 PM

unpack(t, i, j) returns t[i], t[i+1], ..., t[j]; if you do not specify i or j, they're defaulted to 1 and #t respectively -- so whether unpack will return values from an array with holes in it depends on the behavior of #t.

The Length Operator

The length of a table t is defined to be any integer index n such that t[n] is not nil and t[n+1] is nil; moreover, if t[1] is nil, n can be zero. For a regular array, with non-nil values from 1 to a given n, its length is exactly that n, the index of its last value. If the array has "holes" (that is, nil values between other non-nil values), then #t can be any of the indices that directly precedes a nil value (that is, it may consider any such nil value as the end of the array).

If your array has holes in it, there are no guarantees as to which value #t (of those described in the quote above) will return, as illustrated by your examples. There's nothing wrong with the code you've provided as far as the language is concerned. You simply should not expect or rely on #t to return specific values for such arrays.

On a tangential note, ipairs explicitly iterates over t[1], t[2], t[3], ... stopping at the first nil value rather than using the length operator, so it'll never make it past the first nil in the array.

SDPhantom · 05-07-12, 07:25 PM

There are quirks in how Lua stores tables internally. Every table is created with 2 data arrays, one for sequential data and another for associative data. Sequential data is stored when you assign a non-nil value to the next index in a table, this index is equal to #t+1. If you assign data to a key that is greater than the next index or is a non-number, the data is stored in the associative array. Both the length operator and ipairs() operate exclusively off the sequential array.

The table constructor { } breaks this rule in its own way in which undefined keys are written directly into the sequential array and may include embedded nils. The table constructor also trims any nils from the right side of the list.

There is another quirk in Lua that allows you to manually force nils to be embedded into the sequential array of a table. This is done by inserting two non-nil values then overwriting the first with nil.

d87 · 05-07-12, 07:50 PM

http://www.lua.org/source/5.2/ltable.c.html#luaH_getn

{x,y,z} seems to be an "array" notation, which makes sense to have (for example to store function arguments with holes in a table).
When you construct table that way, sizearray variable is filled without checking about holes. Later, when you request table length, it just checks if last element according to sizearray is nil, and if it is, tries to find boundary(non-nil value followed by nil value) manually.
Like in t2 example, it might not be the the boundary you want, so you better strip nils yourself in this case.

Code:

local t3 = {nil, nil, 4, 7, nil, 4}
t3[5]= 2   -- #t3 = 6
t3[7]= 2   -- #t3 = 0

Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.

Edit: i'm late

SDPhantom · 05-08-12, 12:13 AM

Originally Posted by d87

Code:

local t3 = {nil, nil, 4, 7, nil, 4}
t3[5]= 2   -- #t3 = 6
t3[7]= 2   -- #t3 = 0

Also, if you try to expand your array, sizearray is recalculated to first boundary it founds.

I've tested this with the following function, sizearray doesn't ever seem to decrease like said.

Lua Code:

function tAppend(tbl,...)
--  This should allow unpack() to work with embedded nils as if {...} was used
--  We need to keep the array allocation intact, Lua ignores setting the next index to nil
--  and creates gaps, pouring the next values into the non-array section of the table
    local tlen,previsnil=#tbl,false;
    for i=1,select("#",...) do
        local id,val=tlen+i,select(i,...);
        tbl[id]=val or false;-- Force to an actual value
        if previsnil then tbl[id-1]=nil; end
        previsnil=(val==nil);
    end
    if previsnil then tbl[#tbl]=nil; end
    return tbl;
end

Running the following code allows it to resize the table a few times to give it a chance to recalculate the sizearray value, but the embedded nils are still counting towards the size.

Code:

local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8

To make the process better understood, this is the same as the following:

Code:

local t={};

t[1]=1;
t[2]=false;
t[3]=false;	t[2]=nil;
t[4]=4;		t[3]=nil;
t[5]=false;
t[6]=6;		t[5]=nil;
t[7]=7;
t[8]=8;

print(#t);	-- Prints 8

d87 · 05-08-12, 01:44 AM

Code:

local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8

It's 1 on 5.1. But on 5.2 it is indeed 8...

SDPhantom · 05-08-12, 03:14 PM

Originally Posted by d87

Code:

local t=tAppend({},1,nil,nil,4,nil,6,7,8);
print(#t);	-- Prints 8

It's 1 on 5.1. But on 5.2 it is indeed 8...

WoW has been running 5.1 since BC, they have not updated to 5.2 as of now.
All of my observations have been tested on the current implementation of Lua running on WoW.

Ketho · 05-08-12, 06:22 PM

Well this answers a lot for me. Thank you Foxlit, SDPhantom and d87 for the, very clear explanation

Although I had some difficulty trying to understand about the length operator and the "embedded nils" in the table constructor, and the difference with ipairs

But especially Lur's example was one of the weirdest things about Lua that prompted me to ask this question