How does next behave in terms of table references/how do I quickly clone a table

local array = {} -- this is a variable in my actual code, so I can't make new_array = {}
local new_array = array

for index,value in next,array do
    table.insert(new_array, index + 1, array)
end

The above code becomes infinite and gives the following error:

Exception while populating local variables window: bad allocation (x2)

I thought you could clone a table simply by referencing it in a new variable, and evidently I’m wrong.
But this has worked for me in the past - just not using next, it seems.

Can someone please explain this behaviour, and also let me know about any ways to clone a table without making an annoying function for it (there probably aren’t any but I can hope)?

setmetatable(tableToCopy,{})

I’m not sure. Can’t test atm.

1 Like

Doesn’t seem to work - didn’t run a proper test but the tables had the same ID when I printed them.

Next is iterating over the new entries too and re-adding them in your case, which means it’ll just keep creating entries forever, since you specified the same table to have its contents iterated over and added onto itself.

2 Likes

Yeah I think I figured that out

Both variables will point to the same table. Making changes to array also makes changes to new_array and vice versa. It’s not a clone:

local array = {}
local new_array = array

array[1] = "hello"
print(new_array[1])  -- prints hello

new_array[2] = "goodbye"
print(array[2])  -- prints goodbye

What’s happening is that every iteration of your loop, you add to the table, so you make it bigger and it never ends.

I’ll use a while loop to demonstrate:

local array = {1}

local index = 1
while index <= #array do
    table.insert(array, array[index])
    index = index + 1
end

If you think about it, this example will lead to the while loop never completing. Because you added a value to the table, #array is now 2, and your index is 2, so it’s going to go again, which will increase the length and index to 3. This will continue forever. It doesn’t even matter what index you insert into.

This is similar to using pairs or next, which will continue as long as the next index or key exists. Because you keep adding to the same array, there will always be a next index/key.

In comparison, numerical for loops (for var = min, max, increment do) evaluates the max once, so if you increase the length of the array it will still use the original length rather than the new length.


If you want to clone an array (indexes from 1 to length), then here is the easy-to-understand option:

local array = {}
local new_array = {}

for index = 1, #array do
    table.insert(new_array, index, array[index])
end

If you want to clone a dictionary or any table, here is an option using next:

local tbl = {}
local new_tbl = {}

for index, value in next, tbl do
    new_tbl[index] = value
end

This will also work for arrays.


If you want to clone an array, here is the quick option:

local new_array = {unpack(array)}

unpack turns the array into a tuple, or a list of values (e.g. 1, 2, "hello", workspace, 5), then {tuple} sticks them in a new table/array (e.g. {1, 2, "hello", workspace, 5})

This will not work for dictionaries (e.g. {["key"] = value, ["hello"] = 5, another_key = 7})

4 Likes

Thanks

setmetatable actually returns a reference to the first argument, and not to a new table.

Worth noting that all the methods I’ve mentioned only clone the top level of the table. This is also called a shallow copy. These “copy” methods work by literally taking all of the fields in array and putting them in a new table, new_array.

For example:

local array = {
    [1] = {a = "hello"}
}
local new_array = {unpack(array)}

print(new_array[1].a)  -- prints hello

new_array[1].b = "goodbye"
print(array[1].b)  -- prints goodbye

This is because you only made a new table for the outermost table (array in this case). You did not make a new table for any of the inner tables ([1] in this case).


If you want to copy inner tables too, it’s a bit more complicated. This is called a deep copy. The basic idea is to call your copier function on any contained tables, so that way you copy the outer table, the inner table, and any tables inside those recursively forever!

I'll take you through writing a deep copier and some common pitfalls.

Here is our first iteration:

local function deepCopy(tbl)
	local new_tbl = {}
	for key, value in next, tbl do
		local new_value
		if type(value) == 'table' then
			new_value = deepCopy(value)
		else
			new_value = value
		end
		new_tbl[key] = new_value
	end
	return new_tbl
end

In most cases this works! Inner tables are copied, and you have a fancy new table with everything different, it seems.

We have neglected the fact that keys can be anything. Check this out:

local tbl = {}

local tbl_key = {}
tbl[tbl_key] = "hello!"
print(tbl[tbl_key])  -- prints hello!

In this first version, tbl_key won’t be deep copied, it will be shallow copied because we never use deepCopy on keys. Let’s fix that.


local function deepCopy(tbl)
	local new_tbl = {}
	for key, value in next, tbl do
		local new_key, new_value
		if type(key) == 'table' then
			new_key = deepCopy(key)
		else
			new_key = key
		end
		if type(value) == 'table' then
			new_value = deepCopy(value)
		else
			new_value = value
		end
		new_tbl[new_key] = new_value
	end
	return new_tbl
end

Great! Now tables-as-keys works perfectly!

It still won’t work in all cases. Consider this:

local tbl = {}

tbl[1] = tbl
deepCopy(tbl)

That will result in an infinite loop. deepCopy will call itself on tbl over and over. It sees a table it needs to copy, but doesn’t consider that it’s already been copied!


To solve this, we need a way to “remember” that we already copied a table. Furthermore, we need to “remember” what the new_tbl for the already-copied-table is so that we can use it.

We just discussed the solution a moment ago! We can accomplish this with table keys.

In the following scenario, I’m going to copy tbl1 but not tbl2, then I’ll show that tbl1 was copied and tbl2 was not. Consider:

local already_copied = {}

local tbl1 = {"hello"}
local tbl2 = {"goodbye"}

local new_tbl1 = {unpack(tbl1)}
already_copied[tbl1] = new_tbl

print(already_copied[tbl1])  -- prints table: ###### which refers to new_tbl1
print(already_copied[tbl2])  -- prints nil

By using tables as keys, and setting the value to the new_tbl, we can check if a table has already been copied and get the new_tbl all in one operation!

To implement this in deepCopy, we’ll have to somehow reference the same already_copied through all of our recursive calls while copying a single table.

This is actually pretty easy. We can just pass the table on as a parameter to each call!

Let’s implement it:

local function deepCopy(tbl, already_copied)
	if already_copied == nil then  -- let it be called with `deepCopy(tbl)` instead of `deepCopy(tbl, {})`
		already_copied = {}
	end
	local new_tbl = {}
	already_copied[tbl] = new_tbl
	for key, value in next, tbl do
		local new_key, new_value
		if type(key) == 'table' then
			local existing_copy = already_copied[key]
			if existing_copy == nil then
				new_key = deepCopy(key, already_copied)
			else
				new_key = existing_copy
			end
		else
			new_key = key
		end
		if type(value) == 'table' then
			local existing_copy = already_copied[value]
			if existing_copy == nil then
				new_value = deepCopy(value, already_copied)
			else
				new_value = existing_copy
			end
		else
			new_value = value
		end
		new_tbl[new_key] = new_value
	end
	return new_tbl
end

This should work perfectly. Both keys and values are deep copied, and it won’t get stuck in a recursive loop. Awesome! It can be used as easy as local new_table = deepCopy(some_table_here)

This does not copy Instances or any other type that saves state (like Random). This only copies tables.



Now that deep copying is covered, let’s cover metatables.

Metatables are great and neat. They also can’t always be copied, or shouldn’t be copied.

  • The __metatable metamethod can be used to prevent getmetatable from working as you would expect – __metatable can cause getmetatable to return literally any value except nil
  • The metamethods might be written specifically with reference to the original table and won’t work with others. e.g.
local function new_object(a)
	local tbl = {[1] = a}
	local metatable = {
		__index = {
			print = function()
				print(tbl[1])
			end
		}
	}
	setmetatable(tbl, metatable)
	return tbl
end

local obj1 = new_object(5)

local obj2 = {unpack(obj1)}
setmetatable(obj2, getmetatable(obj1))

obj1:print()  -- prints 5
obj1[1] = 7
obj1:print()  -- prints 7

obj2[1] = 29
obj2:print()  -- prints 7, because it's still referring to obj1 in its metatable.

If an object has a metatable, then you probably should not be copying it this way.


Let’s implement that in our deep copier!

local function deepCopy(tbl, already_copied)
	if already_copied == nil then  -- let it be called with `deepCopy(tbl)` instead of `deepCopy(tbl, {})`
		already_copied = {}
	end
	local meta = getmetatable(tbl)
	if meta ~= nil then
		-- uh oh, it has a metatable!
		-- let's just keep a reference to it instead :shrug:
		already_copied[tbl] = tbl
		return tbl
	end
	local new_tbl = {}
	already_copied[tbl] = new_tbl
	for key, value in next, tbl do
		local new_key, new_value
		if type(key) == 'table' then
			local existing_copy = already_copied[key]
			if existing_copy == nil then
				new_key = deepCopy(key, already_copied)
			else
				new_key = existing_copy
			end
		else
			new_key = key
		end
		if type(value) == 'table' then
			local existing_copy = already_copied[value]
			if existing_copy == nil then
				new_value = deepCopy(value, already_copied)
			else
				new_value = existing_copy
			end
		else
			new_value = value
		end
		new_tbl[new_key] = new_value
	end
	return new_tbl
end

You could, alternatively, allow deep copying by implementing a :clone method. Keep in mind though that because it has a metatable, indexing __index might error! We’ll have to wrap it in a pcall…

local function deepCopy(tbl, already_copied)
	if already_copied == nil then  -- let it be called with `deepCopy(tbl)` instead of `deepCopy(tbl, {})`
		already_copied = {}
	end
	local meta = getmetatable(tbl)
	if meta ~= nil then
		local copy
		pcall(function()
			if tbl.clone then
				copy = tbl:clone()
			end
		end)
		if copy ~= nil then
			already_copied[tbl] = copy
			return copy
		else
			already_copied[tbl] = tbl
			return tbl
		end
	end
	local new_tbl = {}
	already_copied[tbl] = new_tbl
	for key, value in next, tbl do
		local new_key, new_value
		if type(key) == 'table' then
			local existing_copy = already_copied[key]
			if existing_copy == nil then
				new_key = deepCopy(key, already_copied)
			else
				new_key = existing_copy
			end
		else
			new_key = key
		end
		if type(value) == 'table' then
			local existing_copy = already_copied[value]
			if existing_copy == nil then
				new_value = deepCopy(value, already_copied)
			else
				new_value = existing_copy
			end
		else
			new_value = value
		end
		new_tbl[new_key] = new_value
	end
	return new_tbl
end

Right now, :clone can’t return nil. Let’s fix that. We’ll need some value to stand in for nil

local Nil = {}
local function deepCopy(tbl, already_copied)
	if already_copied == nil then  -- let it be called with `deepCopy(tbl)` instead of `deepCopy(tbl, {})`
		already_copied = {}
	end
	local meta = getmetatable(tbl)
	if meta ~= nil then
		local copy
		pcall(function()
			if tbl.clone then
				copy = tbl:clone()
				if copy == nil then
					copy = Nil
				end
			end
		end)
		if copy ~= nil then
			already_copied[tbl] = copy
			return copy
		else
			already_copied[tbl] = tbl
			return tbl
		end
	end
	local new_tbl = {}
	already_copied[tbl] = new_tbl
	for key, value in next, tbl do
		local new_key, new_value
		if type(key) == 'table' then
			local existing_copy = already_copied[key]
			if existing_copy == nil then
				new_key = deepCopy(key, already_copied)
			else
				if existing_copy == Nil then
					new_key = nil
				else
					new_key = existing_copy
				end
			end
		else
			new_key = key
		end
		if type(value) == 'table' then
			local existing_copy = already_copied[value]
			if existing_copy == nil then
				new_value = deepCopy(value, already_copied)
			else
				if existing_copy == Nil then
					new_value = existing_copy
				else
					new_value = existing_copy
				end
			end
		else
			new_value = value
		end
		if new_key ~= nil then
			new_tbl[new_key] = new_value
		end
	end
	return new_tbl
end


That got really long really fast! That should pretty much covers everything.

Biggest thing to take away here is not this deepCopy method, but instead all of the possible pitfalls and workaround we had to write.

Ideally you won’t be deep-copying things often. If you ever are deep-copying things, then you want to understand your data structure well enough to know what you need to include and ideally use something simpler than this. Once we consider metatables, we realize that there’s no catch-all way to copy a table in Lua.

13 Likes