Table being sent from the server, isn't the same on the client

So, recently I discovered a bug where, if I send a fairly complex table from the server to the client, when the client receives it, I print the table on the client, but for some reason the table isn’t the same on the client and the server.

Here is what I mean:

Here is the table that I’m attempting to send:

local tbl = {[-1] = {[2] = {[1] = true}}, [0] = {[2] = {[1] = true}}, [2] = {[2] = {[1] = true}}, [1] = {[2] = {[1] = true}}}

Script:

wait(5)
local tbl = {[-1] = {[2] = {[1] = true}}, [0] = {[2] = {[1] = true}}, [2] = {[2] = {[1] = true}}, [1] = {[2] = {[1] = true}}}

print(tbl)

game:GetService("ReplicatedStorage").RemoteEvent:FireAllClients(tbl)

Localscript:

local remote = game:GetService("ReplicatedStorage"):WaitForChild("RemoteEvent")

remote.OnClientEvent:Connect(function(tbl)
	print(tbl)
end)

What in the world is causing this, and how can I fix it? (Unless this is a bug with roblox)

3 Likes

There was a thread on here earlier where the problem was that the extended output, or whatever it’s called, was not accurate. Just as a sanity check, maybe try manually checking the contents of the table.

Like on the client add this:

function deepcompare(t1, t2, ignore_mt)
    local ty1 = type(t1)
    local ty2 = type(t2)
    if ty1 ~= ty2 then return false end
    -- non-table types can be directly compared
    if ty1 ~= 'table' and ty2 ~= 'table' then return t1 == t2 end
    -- as well as tables which have the metamethod __eq
    local mt = getmetatable(t1)
    if not ignore_mt and mt and mt.__eq then return t1 == t2 end
    for k1, v1 in pairs(t1) do
        local v2 = t2[k1]
        if v2 == nil or not deepcompare(v1, v2) then return false end
    end
    for k2, v2 in pairs(t2) do
        local v1 = t1[k2]
        if v1 == nil or not deepcompare(v1, v2) then return false end
    end
    return true
end

local shouldBe = {[-1] = {[2] = {[1] = true}}, [0] = {[2] = {[1] = true}}, [2] = {[2] = {[1] = true}}, [1] = {[2] = {[1] = true}}}

print(deepcompare(tbl, shouldBe))

deepcompare returns false, when I compare them:

Here is what I’m doing:

local remote = game:GetService("ReplicatedStorage"):WaitForChild("RemoteEvent")

function deepcompare(t1, t2, ignore_mt)
    local ty1 = type(t1)
    local ty2 = type(t2)
    if ty1 ~= ty2 then return false end
    -- non-table types can be directly compared
    if ty1 ~= 'table' and ty2 ~= 'table' then return t1 == t2 end
    -- as well as tables which have the metamethod __eq
    local mt = getmetatable(t1)
    if not ignore_mt and mt and mt.__eq then return t1 == t2 end
    for k1, v1 in pairs(t1) do
        local v2 = t2[k1]
        if v2 == nil or not deepcompare(v1, v2) then return false end
    end
    for k2, v2 in pairs(t2) do
        local v1 = t1[k2]
        if v1 == nil or not deepcompare(v1, v2) then return false end
    end
    return true
end
local sb = {[-1] = {[2] = {[1] = true}}, [0] = {[2] = {[1] = true}}, [2] = {[2] = {[1] = true}}, [1] = {[2] = {[1] = true}}}

remote.OnClientEvent:Connect(function(tbl)
	local xD = deepcompare(tbl, sb) 
	print(xD)
end)

Someone, please help me.

So, I think I found my issue. I did a bit of research, and the devforum says that you aren’t supposed to send mixed tables through remote events:

And there is a post on the devforum that shows you how to detect if a table is mixed or not here

So I decided to check and see if the table is mixed on the server using the function in that post that I mentioned above

And sure enough the table I’m trying to send is indeed a mixed table:

Now, the real question I is, how can I convert tables to a “Non-Mixed” table, then after it’s received convert it back into a mixed table?

2 Likes

Interesting. It might be considered “mixed” because of the [-1] index. Does it work without that? Or without any indices <1?

I believe Lua stores contiguous integers from 1…n in the “array part” of the table, and anything else goes in the “hash part”. Having both might be what roblox means by “mixed”.

Edit: Also note that the expressive output window is a beta feature. I think there are still a lot of bugs with it, I would probably not trust it.

2 Likes

I think it’s still mixed:

It still prints true, even though I changed -1 to 1.

OK, the docs are just wrong here. You piqued my interest. It’s much more complicated.

Summary

Docs:

Avoid passing a mixed table (some values indexed by number and others by key), as only the data indexed by number will be passed .

Reality:

Only numeric keys >= 1 will be passed. They also have to be contiguous. Except when they don’t have to be. And also sometimes it matters if you implicitly or explicitly declare the index.

Basically, lua is doing things behind the scenes, and I think “mixed” is more complicated than ROBLOX makes it out to be.

Setup

Server script
local tests = {
	{1, 2, 3},
	{[0]=0, 1, 2, 3},
	{1, 2, [4]=4},
	{1, 2, 3, 4}, -- [3] = nil
	{[2]=2, [3]=3, [4]=4},
	{1, 2, 3, [4]=4},
	{1, 2, 3, [4]=4}, -- [3] = nil
	{1, 2, 3, [4]=4, [6]=6},
	{1, 2, 3, [6]=6},
	{1, 2, 3, 4, [6]=6},
	{[-1]=-1, [0]=0},
	{[-1]=-1, [0]=0, [1]=1},
	{[-1]=-1, [0]=0, [2]=2},
	{[-1]=-1, [0]=0, [2]=2, 1},
}

tests[4][3] = nil
tests[7][3] = nil

wait(4)

for idx, tbl in ipairs(tests) do
	wait(.1)
	
	-- print all keys sending
	local keys = {}
	for k, v in pairs(tbl) do
		table.insert(keys, k)
	end
	print(idx, "sending keys", unpack(keys))
	
	-- print #tbl and ipairs keys
	local ikeys = {}
	for i, v in ipairs(tbl) do
		table.insert(ikeys, i)
	end
	print("...", #tbl, "ikeys", unpack(ikeys))
	
	-- send
	game:GetService("ReplicatedStorage").RemoteEvent:FireAllClients(tbl)
end
Client script
-- client

game:GetService("ReplicatedStorage").RemoteEvent.OnClientEvent:Connect(function(tbl)
	local keys = {}
	for i,v in pairs(tbl) do
		table.insert(keys, i)
	end
	print("received", unpack(keys))
	print()
end)

Testing Results

Show results
tbl #tbl ipairs keys (server) keys received (client) notes
{1, 2, 3} 3 1, 2, 3 1, 2, 3 duh
{[0]=0, 1, 2, 3} 3 1, 2, 3 1, 2, 3 0 is not sent. the array part of tables starts at 1.
{1, 2, [4]=4} 2 1, 2 1, 2 4 is not sent. the array part should be contiguous. 4 must be in the hash part of the table
{1, 2, 3, 4}; tbl[3]=nil 4 1, 2, 3, 4 1, 2, 4 4 is sent, because it was initially in the array part of the table? #tbl == 4 but ipairs only sees the first two elements. Roblox must just care about what’s in the array part.
{[2]=2, [3]=3, [4]=4} 0 4, 3, 2 all sent in random order. all keys are probably put in the hash part of the table (so, not mixed), and are sent in a random order.
{1, 2, 3, [4]=4} 4 1, 2, 3, 4 1, 2, 3, 4 all sent. numeric keys are contiguous. doesn’t really make sense though, since explicitly doing [4] means it should go in the hash part?
{1, 2, 3, [4]=4}; tbl[3]=nil 2 1, 2 1, 2 sort of makes sense. 4 is in the hash part because it was explicit?
{1, 2, 3, [4]=4, [6]=6} 6 1, 2, 3, 4 1, 2, 3, 4, 6 ??? bizarre. non-contiguous hash keys still sent
{1, 2, 3, [6]=6} 3 1, 2, 3 1, 2, 3 ??? … except when they aren’t. Possibly something to do with how lua is dynamically resizing its array part. So maybe…
{1, 2, 3, 4, [6]=6} 4 1, 2, 3, 4 1, 2, 3, 4 now I’m just totally lost :slight_smile:
{[-1]=-1, [0]=0} 0 -1, 0 makes sense, all hash
{[-1]=-1, [0]=0, [1]=1} 1 1 1 sort of makes sense, but [1] is put in the array part. must be some optimization or something
{[-1]=-1, [0]=0, [2]=2} 0 0, -1, 2 makes sense, all hash.
{[-1]=-1, [0]=0, [2]=2, 1} 2 1, 2 1, 2 okay…

Wonder if @Osyris or someone who knows more about this could weigh in.

4 Likes

Just as a side question, what are you trying to use this for?

There might be better ways to go about what you want to do without sending complex tables.

Can you please show me a method?

I’m making a terrain generator, and right now what I’m attempting to do is send the chunk data from the server to all the clients in the game, when the client receives it, they receive the chunk coordinates, and the data about each chunk, then it gets added to a queue. Then I have a custom “Heartbeat” / tick system, and iirc they fire 20 times per second, and every “tick” / “Heartbeat” the client decides to build a random chunk out of the queue of chunks waiting to be created.

But, I run into an issue, because the chunk data that is sent from the server to the client is a mixed table, so the client doesn’t generate the chunk properly.

The server decides what chunks generate, and when. It also generates the heightmap, and what blocks are solid, and which aren’t. After that, the server sends that chunk data to all the clients. The server does not do any of the visual stuff :stuck_out_tongue: , but the client does. When the clients receive it, they receive a “broken mixed table” so when they try to make the chunk appear the chunk doesn’t appear properly. So I did so many tests to see what was wrong with the terrain generator, to see what was wrong / how I could fix it. But I never could figure out how to fix it. So I did an experiment, and eventually found out that you can’t send complex tables (or “mixed tables”) through remote events.

Here is what the data looks like on the server, and the client:

EDIT: Also, Idk If I gave too much information :man_shrugging:.

Your table is only a valid Lua Array if it has non-nil values for a contiguous range of whole-number keys from 1 up to some number N, with no missing entries and no other keys.

Trying to use -1, or 0 as a key will make your table have a Dictionary part. If you have entries for keys 1, 2, 3, 5, 7, 11… i.e. there are whole numbers missing because you either didn’t assign values to those indices, or the value you assigned was nil, that can fail too.

The solutions are simple: either use all string keys so that you have a pure dictionary, or add a positive offset to all your indices, which you subtract back out on the receiving end. If you know your array is going to start at -4, for example, just add 5 to all the indices. If it’s a variable number, you can also send it in the array. Again with the -4 example, you can add 6 to all the indices instead of 5, and set array[1] = 6. then, on the receiving end, you read the offset from the array element [1], and then loop over elements 2…N, subtracting 6 to recover the true index.

For what you’re trying to do, I think a pure array-based solution is the better option, performance wise.

4 Likes

Why doesn’t roblox automatically do this and then turn it back into a number on the other end?

There would be a big performance hit to convert a large array to a dictionary. You generally want RemoteEvents to go from Client to Server and back as quickly as possible, so taking time to do some new encoding and decoding is not desirable. You can’t just make all array indices of a mixed table into strings either, for example, a Lua table can have [1] and [“1”] as two different keys already.

1 Like

Ah the fun of #tbl – Looks like it just sends 1…# if # > 0, else sends all.

You can tostring() the keys to turn it into a full dictionary.

I want to know which one has more performance, this one:

Or this one:

I’m going to assume that adding a positive offset to all of my indices is better for performance.

Yes. I’m guessing you already have contiguous integer indicies that are the voxel chunk coordinates or something, so keeping things as integer arrays is WAY better than converting to a dictionary with string keys.

2 Likes

Agreed with @EmilyBendsSpace. It’s going to be a huge memory hog to store your voxels like that.

Here’s a cool article detailing how ROBLOX went about storing their terrain (in memory and on disk).

tl;dr:

Their first pass at memory:

When we originally developed the system, we decided to settle on a simple sparse storage format: all voxels would be stored in chunks 32x32x32 with each chunk represented as a linear 3D array […] and the entire voxel grid is simply a hash map, mapping the index of the chunk to the chunk contents itself:

HashMap<Vector3int32, Chunk> chunks;

Inside the chunk we ended up storing several 3D arrays - we call a 3D array of voxels “box” - of different sizes, forming a mipmap pyramid for our voxel content. We have 32^3 Box for the original content, and 16^3, 8^3 and 4^3 mipmaps that contain downsampled voxel data

They later improved with some compression:

  • Each row (32^3 chunk has 32^2 rows) is either allocated or not.
  • For unallocated rows, we store one byte - which represents the material value - and assume that all cells In this row are filled with this material and have “default” occupancy (1 for solid materials and 0 for air).
  • For allocated rows, we store an offset into cell data, which is a linear array that contains data for all allocated rows in an uncompressed fashion.

The article goes on to detail things like how they compressed the data for disk, which may be interesting if you plan to datastore stuff.

Of course, you’ll be doing all this in lua, and have little to no control over how the bytes are actually sent over the network, but there’s still cool info in the article.

2 Likes