How do you properly store `nil` values when packing tables?

Help and Feedback Scripting Support
1

I’m trying to pack entity data into an array using a predefined key list, but some of the values can be nil, and obviously those get lost when storing/unpacking but when unpacking parameters I need those values to not be lost.

Below is my current best attempt at this:

-- Sentinel value to represent nil in packed tables
local NIL_SENTINEL = newproxy(true) -- or: local NIL_SENTINEL = {}

local function EncodeCreationEntity(Entity)
	local encoded: EncodedEntity = {
		InitData = {},
	}

	local InitList = {"a", "b", "c"}

	local PackedInit = {}

	for i, key in ipairs(InitList) do
		local val = Entity[key]
		PackedInit[i] = (val == nil) and NIL_SENTINEL or val
	end

	PackedInit.n = #InitList
	encoded.InitData = PackedInit

	return encoded
end

local encodedEntity = EncodeCreationEntity(
	{
		a = 1,
		b = nil,
		c = 3,
	}
)

print(table.unpack(encodedEntity.InitData)) -- the goal is for this to print 1, nil, 3

Any help would be appreciated :folded_hands:

2 Likes
2

I believe table.unpack (like the rest of the table library) is nil-terminated, meaning you can’t do this with nil. You must use a different sentinel value to represent absence, like an empty string.

2 Likes
3

Arrays by nature are incremental.
This means that if you have array {"a", "b", "c"} and remove “b” at position 2, “c” will shift from position 3 into position 2 (only when encoding).
It should also be worth noting arrays are relatively more performant compared to dictionaries due to their nature (read more here).

To preserve numerical keys, you have two options:

  1. Use string keys (e.g. {"1" = "a", "2" = "b", "3" = "c"}. The downside is that you will have to access your table using strings as well, and array operations using the table library will no longer be possible, since it’s a dictionary.
  2. Use an “empty” value. This could be anything, as long as in your system it denotes with guarantee that it represents “nothing”. I recommend simply using an empty string "", or a number like -1! This is my go-to option when preserving indexes is a must.
2 Likes
4

This still wouldn’t preserve nil values though, as setting key = nil inside a dictionary is essentially the same thing as saying “delete this key”. So the non-nil sentinel value is the only way to make this work.

2 Likes
5

Yep, you’re absolutely right.
Somehow I misremembered this even though I’ve tinkered with this logic so many times (perhaps the reason why!)

Thanks for the callout.

2 Likes
6

local NIL_SENTINEL = "nil"

though you will have problems storing strings which values are “nil” but that’s another question.

2 Likes
7

Adding onto this, you could use a blank proxy table instead since they’re guaranteed unique when compared by value:

local NIL_SENTINEL = newproxy(false)

Though this is kind of a reinvention of Symbol libraries, it’d be better to just use one.

1 Like
8

this all works well, but the problem occurs when I attempt to unpack the table into it’s parameters, in which nil is lost.

1 Like
9

As a side note, when there are any requirements, i consider these are requirements:

then there should be a purpose too.
And there’s no purpose of it.

And since OP have not elaborated into purposes
local NIL_SENTINEL = "nil"
perfectly solves the problem.

That’s fantastic.
But I am not sure what’s the point of differentiation one nil from another nil. Because, you now, nil is just a nil.

this can be solved by:

local nilHolder = {}
setmetatable(nilHolder, {
	__tostring = function()
		return "nil"
	end,
})

local function createPackagedValue(value)
	if nil ~= value then return value end
	return nilHolder
end

PackedInit[i] = createPackagedValue(val)

checked with

local InitList = {"a", "b", "c", "d"}

local encodedEntity = EncodeCreationEntity(
	{
		a = 1,
		b = nil,
		c = 3,
		d = "nil"
	}
)

for i, e in {table.unpack(encodedEntity.InitData)} do
	print(i, `[{e}]`, `typeof[{typeof(e)}]`)
end

prints

  1 [1] typeof[number]
  2 [nil] typeof[table]
  3 [3] typeof[number]
  4 [nil] typeof[string]

where 2 is a nil and 4 is a “nil” string

1 Like
10 
-- Sentinel value to represent nil in packed tables
local NIL_SENTINEL = newproxy(false) -- or: local NIL_SENTINEL = {}

local function EncodeCreationEntity(Entity)
	local encoded: EncodedEntity = {
		InitData = {},
	}

	local InitList = {"a", "b", "c"}

	local PackedInit = {}

	for i, key in ipairs(InitList) do
		local val = Entity[key]
		PackedInit[i] = (val == nil) and NIL_SENTINEL or val
	end

	PackedInit.n = #InitList
	encoded.InitData = PackedInit

	return encoded
end

local encodedEntity = EncodeCreationEntity(
	{
		a = 1,
		b = nil,
		c = 3,
	}
)

print(table.unpack(encodedEntity.InitData))

as of right now if I were to run this code the print statement would output 1 userdata: 0x523f9ccfe9471cd9 3 when I need it to output nil directly (1, nil, 3).

1 Like
11

You can add a __tostring method to the symbol, like this:

-- Sentinel value to represent nil in packed tables
local NIL_SENTINEL = newproxy(false) -- or: local NIL_SENTINEL = {}

setmetatable(NIL_SENTINEL, {
  __tostring = function()
    return "nil"
  end
})
1 Like
12

you do not need newproxy
create new table and set metatable with __tostring to it
so it prints what the __tostring returns

local nilHolder = {}
setmetatable(nilHolder, {
	__tostring = function()
		return "nil"
	end,
})
1 Like
13

This is to counter collision with concrete values like how using "nil" as a sentinel might collide with legitimate data with the same value. A symbol wouldn’t have this issue because it is compared by address, not value

1 Like
14

== is true for the same values only.
this usage works correctly without creating separate instances for each nil

local nilHolder: any = {}
local function initInitHolder(t)
	setmetatable(t, {
		__tostring = function()
			return "nil"
		end,
	})
end

local function createPackagedValue(value)
	if nil ~= value then return value end
	return nilHolder
end

print("nilHolder == nilHolder", createPackagedValue(nil) == createPackagedValue(nil))
print("nilHolder == some value", createPackagedValue(nil) == createPackagedValue("some value"))
print("nil == nil", nil == nil)

prints

  nilHolder == nilHolder true
  nilHolder == some value false
  nil == nil true
15

Serialization and deserialization are common practices on Roblox.

This is a way to handle nil in Luau when you need to serialize data (DataStores/JSON) or just in general.

Standard tables delete keys assigned to nil. By using a holder object (the NIL_SENTINEL string), you provide a placeholder that retains the array index during JSON encoding, without excluding the entry.

The length operator # is unreliable in arrays with holes. Storing PackedInit.n ensures that we know exactly how many elements exist, even if the last values are nil.

Using table.unpack(result, 1, packedTable.n) is basically the main ingredient to this. Providing the third argument forces Luau to unpack the full range. Without it, the unpack would truncate at the first nil it hits.

Unlike using tables or newproxy as sentinels, a string sentinel survives HttpService:JSONEncode. This makes it viable for saving data or sending it across RemoteEvents.
However, RemoteEvents encodes data on its own (binary serialization), so it’s up to whatever you’re using it for.
If a legitimate data value could ever be the string “NIL_SENTINEL”, the decode would incorrectly treat it as nil, which can be solved with type scoping, GUIDs, and null terminators.

local HttpService = game:GetService("HttpService")
local NIL_SENTINEL = "__NIL_SENTINEL__"
local InitList = {"a", "b", "c"}

local function EncodeCreationEntity(Entity)
	local PackedInit = {}

	for i, key in ipairs(InitList) do
		PackedInit[i] = (Entity[key] == nil) and NIL_SENTINEL or Entity[key]
	end

	PackedInit.n = #InitList
	return { InitData = PackedInit }
end

local function UnpackEntity(packedTable)
	local result = {}
	for i = 1, packedTable.n do
		result[i] = (packedTable[i] == NIL_SENTINEL) and nil or packedTable[i]
	end
	return table.unpack(result, 1, packedTable.n)
end

local encodedEntity = EncodeCreationEntity({
	a = 1,
	b = nil,
	c = 3,
})

local serialized = HttpService:JSONEncode(encodedEntity)
local deserialized = HttpService:JSONDecode(serialized)

print(UnpackEntity(deserialized.InitData)) -- 1  nil  3

This was a pretty big discussion back when Lua was in its early stages. You can read up on some older wikis on the official page to understand it further.

16

Could you please clarify how that works with

local encodedEntity = EncodeCreationEntity({
	a = 1,
	b = nil,
	c = "__NIL_SENTINEL__",
})

?
Thank you!

17

Oh, and in addition,
Luau has a built-in table.pack(…) which automatically creates the .n field. While the manual loop is better for mapping an entity to an array, table.pack is the official version of that logic basically.

18

Why would you construct multiple sentinels anyways?

19

I would not.

local nilHolder: any = {}
local function initInitHolder(t)
	setmetatable(t, {
		__tostring = function()
			return "nil"
		end,
	})
end

local function createPackagedValue(value)
	if nil ~= value then return value end
	return nilHolder
end

createPackagedValue returns same nilHolder for nils.

But this

does.

1 Like
20

I would love to use it except I can’t figure out a way to use with the key array I have, anyways im bout to test your solution