What is the best way to serialize strings, numbers (userids) and cframes to take the least amount of data possible in a datastore? (like those notes games that have hundreds of thousands of them)
If you have know any methods or modules please link them!!!
I recommend taking a look at Packet (mainly for its batching function). I personally haven’t used it all that much, but maybe it can be of some help to you. Packet - Networking library
You might want to look into file compression algorithms. Serialization does not compress data.
But, a few tips:
Can I ask you what makes you worry about storage? Are you working on a game with a very large amount of data? Max data per datastorage key is 4 megabytes if I remember correctly. That’s enough for most cases.
Take a look at Suphi's DataStore Module it has compression
some things to take note: serializing for the datastore is different then serializing for lets say remote events this is because the datastore data is saved into JSON and JSON will escape some characters causing them to consume more then one byte of data you can test this by doing HttpService.JSONEncode(character) and making sure it only has a length of 3
and here is the code from SDM
local characters = {[0] = "0","1","2","3","4","5","6","7","8","9","a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","!","$","%","&","'",",",".","/",":",";","=","?","@","[","]","^","_","`","{","}","~"}
local bytes = {} for i = (0), #characters do bytes[string.byte(characters[i])] = i end
local base = #characters + 1
Compress = function(value, level, decimals, safety)
local data = {}
if type(value) == "boolean" then
table.insert(data, if value == false then "-" else "+")
elseif type(value) == "number" then
if value % 1 == 0 then
table.insert(data, if value < 0 then "<" .. Encode(-value) else ">" .. Encode(value))
else
table.insert(data, if value < 0 then "(" .. Encode(math.round(-value * decimals)) else ")" .. Encode(math.round(value * decimals)))
end
elseif type(value) == "string" then
if safety == true then value = value:gsub("", " ") end
table.insert(data, "#" .. value .. "")
elseif type(value) == "table" then
if #value > 0 and level == 2 then
table.insert(data, "|")
for i = 1, #value do table.insert(data, Compress(value[i], level, decimals, safety)) end
table.insert(data, "")
else
table.insert(data, "*")
for key, tableValue in value do table.insert(data, Compress(key, level, decimals, safety)) table.insert(data, Compress(tableValue, level, decimals, safety)) end
table.insert(data, "")
end
end
return table.concat(data)
end
Decompress = function(value, decimals, index)
local i1, i2, dataType, data = value:find("([-+<>()#|*])", index or 1)
if dataType == "-" then
return false, i2
elseif dataType == "+" then
return true, i2
elseif dataType == "<" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return -Decode(data), i2
elseif dataType == ">" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return Decode(data), i2
elseif dataType == "(" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return -Decode(data) / decimals, i2
elseif dataType == ")" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return Decode(data) / decimals, i2
elseif dataType == "#" then
i1, i2, data = value:find("(.-)", i2 + 1)
return data, i2
elseif dataType == "|" then
local array = {}
while true do
data, i2 = Decompress(value, decimals, i2 + 1)
if data == nil then break end
table.insert(array, data)
end
return array, i2
elseif dataType == "*" then
local dictionary, key = {}, nil
while true do
key, i2 = Decompress(value, decimals, i2 + 1)
if key == nil then break end
data, i2 = Decompress(value, decimals, i2 + 1)
dictionary[key] = data
end
return dictionary, i2
end
return nil, i2
end
Encode = function(value)
if value == 0 then return "0" end
local data = {}
while value > 0 do
table.insert(data, characters[value % base])
value = math.floor(value / base)
end
return table.concat(data)
end
Decode = function(value)
local number, power, data = 0, 1, {string.byte(value, 1, #value)}
for i, code in data do
number += bytes[code] * power
power *= base
end
return number
end
this code was written before buffers existed so you might be able to improve CPU performance by converting it to use buffers
saving strings into the datastore if you selected the correct characters can go up to if I remember correctly base 94
if you decide to save a buffer into the datastore you can use the full range of the byte [0-127] but I believe this gets converted to base 64 but might also have built in compression
I’m not 100% sure what one of the 2 would end up having the best outcome
base94 vs base64 + compression
I personally think it would be best to use a buffer to build the string using the special 94 characters then converting the buffer to a string before saving it to the datastore
this video should also help you
the game permanently saves player placed objects in a single datastore key and each one takes up about 120 bytes, capping it at like 30k max objects that can be saved, which isnt a lot since the map goes on forever and theyre loaded in chunks