Error handler in xpcall called unexpectedly from memory error

local c = 0
print(xpcall(
	function()
		local t = {}
		for i=1,10000 do
			local ti = {}
			t[i] = ti
			for j=1,0X7FFFFFFF do
				ti[j] = j
				c = c+1
			end
		end
	end,
	function(e)
		print("error handler called!",e)
		return "error handler return"
	end
))
print(c)

On my machine with 16gb of ram (might be important) the output of the script was:

error handler called! ServerScriptService.Script:9: table overflow
error handler called! error handler return
false not enough memory
134217728

This program run using Lua 5.1.5 produced the following output (different iteration count is probably due to compiling as a 32 bit program):

false   not enough memory
33554432

Firstly, the function passed to xpcall should result in a memory error. Lua 5.1.5 doesn’t call the error handler as the error was a memory error (and future versions of the Lua manual specify that memory errors don’t result in the error handler being called), if this is the expected behavior then it shouldn’t call the error handler (and it also shouldn’t call the error handler a second time). Because the Lua 5.1 manual doesn’t specify this, it could be that the intention was to make memory errors also call the error handler. If this was the intention, it still wouldn’t explain the second call to the error handling function, or that the result of xpcall was not enough memory and not the result of the error handler function. It also seems weird that the result of the first call to the error handler was used as the argument to the second call of the error handler.

I’m not sure how it got table overflow, that might be part of the bug? I insert only up to 2^31-1 elements into the arrays, so it shouldn’t be generated? The last line that was printed also indicates that only 2^27 elements were inserted, so it shouldn’t generate a table overflow error. If the array part of a table doubles every time it becomes full, perhaps it could be because the size in bytes of the new array can’t fit into an unsigned 32 bit integer (2^27*2*16 is 2^32, where 16 is the size of a value and the associated tag plus alignment)? If this is what caused the error, then shouldn’t it generate a memory error and not a table overflow error? Table overflows errors are for when inserting an entry would result in there being more than INT_MAX entries are in the table, if I remember correctly.

Edit: Recompiling Lua with a smaller limit on the size of tables (MAXBITS = 14 in ltable.c) causes the output to become:

error handler called!   Active.lua:9: table overflow
false   error handler return
32768

As there is no memory error, it calls the error handler normally and uses the result of it as the result to xpcall. MAXBITS in ltable.c also explains the 2^27 limit I encountered before.

1 Like

I took the liberty of testing this on my machine with 32gb ram and got the same result

  23:41:10.057  error handler called! not enough memory  -  Server - Script:22
  23:41:10.057  false error handler return  -  Server - Script:4
  23:41:10.058  134217728  -  Server - Script:26

I don’t have any sort of solution for your issue here but I do want to ask what you were planning to do here? Just stress testing or is there an application for gargantuan arrays in Roblox gaming?

Ugh. Congrats, you found a horrible edge case in some code that was due for rewrite anyway :smiley:

“table overflow” is as designed - tables in Luau have limits on array & hash portions that are currently set to 2^26 elements each. We can maybe increase this to 2^27 but there are some potential problems in the future with this (we don’t have the full 32 bits available in some places where table indices are stored.

The problem you have highlighted otherwise is real in that the error handler should not have been called twice. This happens because of this patch: Lua: bugs

This is a bad patch. The reason why we thought it was bad and why this code was due for rewrite was efficiency, but it’s bad for a deeper reason: it results in the error handler getting called during handling the table overflow error (in this case), and then the second thrown error triggers the handler again. (I don’t know if this patch made it into 5.3.x; I believe 5.4 rewrites this code without using this patch).

On an unrelated note, we call the error handler even in case of out-of-memory error. It’s not quite clear if that’s a good or a bad idea - I understand the intention behind not-calling it, but we actually sometimes see out of memory errors on production due to large allocations and it’s not difficult to recover from these.

10 Likes

P.S. I believe you hit table overflow when the table reaches 2^26 array elements + 2^26 hash elements which adds up to 2^27 elements that together take 3 GB. We can actually increase the array limit to 2^30 or something along these lines with no issue, it’s the hash portion that needs a 2^26 limit.

2 Likes