Hello,
I have a few questions about DataStore, as I have don’t have much experience in working with DataStoreService.
Thank you!
Here’s the announcement stating all the datastore changes that are taking place soon.
What kind of string are you trying to save? I did find this:
Any string being stored in a data store must be valid UTF-8. In UTF-8, values greater than 127 are used exclusively for encoding multi-byte codepoints, so a single byte greater than 127 will not be valid UTF-8 and the GlobalDataStore:SetAsync() attempt will fail.
If you are not having millions of players there is nothing to worry about as of right now. If you are over the limit you might have to pay more. I am not fully sure but that’s what I heard.
I know this, but what I wanted to ask is if there was a limit to the length of the string that I can pass into GlobalDataStore:SetAsync(), or does the length not matter as long as that string does not allocate more than 1 MB of storage?
I don’t mean the overall limit of DataStores, I meant the limit of a single data that I am passing to GlobalDataStore:SetAsync()
You can read about the limits here: Error codes and limits | Documentation - Roblox Creator Hub
as we can see from here each key can have a max of 4,194,304 characters
to measure how many characters your using you do
local data = {
MyValue = 2,
}
local json = HttpService:JSONEncode(data)
print(#json) -- print how many characters data uses
another limit is the experience limit and that will scale you can see your current limit at the dashboard
Yes, each DataStore entry can store up to 1 MB of data.
There is no fixed character length limit for strings, only the total data size matters. You can save any string as long as its serialized size stays under the 1 MB limit.
If you save 3 different keys, each can have up to 1 MB, so total you can store about 3 MB spread across those keys.
The limit is per key, not for the whole DataStore combined.
buffers are not bad at base 64 but you can make custom compression at base 94 I believe is the max for datastore/json
here is a video on compression
and you can also checkout SDM that has compression
and there you will find
local characters = {[0] = "0","1","2","3","4","5","6","7","8","9","a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","!","$","%","&","'",",",".","/",":",";","=","?","@","[","]","^","_","`","{","}","~"}
local bytes = {} for i = (0), #characters do bytes[string.byte(characters[i])] = i end
local base = #characters + 1
Compress = function(value, level, decimals, safety)
local data = {}
if type(value) == "boolean" then
table.insert(data, if value == false then "-" else "+")
elseif type(value) == "number" then
if value % 1 == 0 then
table.insert(data, if value < 0 then "<" .. Encode(-value) else ">" .. Encode(value))
else
table.insert(data, if value < 0 then "(" .. Encode(math.round(-value * decimals)) else ")" .. Encode(math.round(value * decimals)))
end
elseif type(value) == "string" then
if safety == true then value = value:gsub("", " ") end
table.insert(data, "#" .. value .. "")
elseif type(value) == "table" then
if #value > 0 and level == 2 then
table.insert(data, "|")
for i = 1, #value do table.insert(data, Compress(value[i], level, decimals, safety)) end
table.insert(data, "")
else
table.insert(data, "*")
for key, tableValue in value do table.insert(data, Compress(key, level, decimals, safety)) table.insert(data, Compress(tableValue, level, decimals, safety)) end
table.insert(data, "")
end
end
return table.concat(data)
end
Decompress = function(value, decimals, index)
local i1, i2, dataType, data = value:find("([-+<>()#|*])", index or 1)
if dataType == "-" then
return false, i2
elseif dataType == "+" then
return true, i2
elseif dataType == "<" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return -Decode(data), i2
elseif dataType == ">" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return Decode(data), i2
elseif dataType == "(" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return -Decode(data) / decimals, i2
elseif dataType == ")" then
i1, i2, data = value:find("([^-+<>()#|*]*)", i2 + 1)
return Decode(data) / decimals, i2
elseif dataType == "#" then
i1, i2, data = value:find("(.-)", i2 + 1)
return data, i2
elseif dataType == "|" then
local array = {}
while true do
data, i2 = Decompress(value, decimals, i2 + 1)
if data == nil then break end
table.insert(array, data)
end
return array, i2
elseif dataType == "*" then
local dictionary, key = {}, nil
while true do
key, i2 = Decompress(value, decimals, i2 + 1)
if key == nil then break end
data, i2 = Decompress(value, decimals, i2 + 1)
dictionary[key] = data
end
return dictionary, i2
end
return nil, i2
end
Encode = function(value)
if value == 0 then return "0" end
local data = {}
while value > 0 do
table.insert(data, characters[value % base])
value = math.floor(value / base)
end
return table.concat(data)
end
Decode = function(value)
local number, power, data = 0, 1, {string.byte(value, 1, #value)}
for i, code in data do
number += bytes[code] * power
power *= base
end
return number
end
but this reserves some special characters so you might be able to compress a little better then whats in SDM for more specific cases
Right the code was not for you to just copy and paste but to show that its possible to compress to base 94 over using buffers base 64 and to answer the question:
Q: would it be better to create a buffer object by passing the string to buffer.fromstring and then storing that buffer to DataStore directly, or would it be better to grab the compressed zbase64
A: it would be better to make your own custom base 94 compressor
But if you don’t want to do that I think it might be better to just save the buffer directly while it will use more data the problem is when you try to extract the base64 string using JSONEncode Roblox does not guarantee the format of the JSON will always stay the same and a change could break your code so it feels risky to me and also very hacky
also as a side note my computer does not support --!native
your encodedData is not correct the reason we only have base 94 over base 128 is because some characters consume more then 1 character in json but your not selecting the correct characters so your encoding to base 94 for no reason not only do you need to encode to base 94 you must also only use the 94 characters that use 1 character each
to test what character are ok to use you do
local HttpService = game:GetService("HttpService")
for index = 0, 127 do
local character = string.char(index)
print(index, character, "|||", #HttpService:JSONEncode(character) - 2)
end
you want to only use the characters that end with 1
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
