i want to add a feature to my game where changes to server are saved and loaded when someone joins that server
the current problem that i have is the data size
roblox’s data store has a limit for saved data
is there a way i can fix that and is there a compression formula i can use?
convert your data to a string and store that, almost like a manual encryption of sorts. for example like coins = c and its value is 100 so as a string its c100. you just gotta make sure when you read that info from the datastore you have to convert it out of that form.
2 Likes
You can split the data into multiple Data Stores.
1 Like
You can use Squash to serialize your data (including converting numbers to strings in base 256, but be careful, datastores only accept 93 of the 256 characters). 1 “normal” string character takes up 1 byte. So converting base 10 numbers to base 93 strings saves up space. Other characters like emojis, etc, take up more than 1 byte
You can also use buffers to write to bytes directly, and datastore can now save buffers. I didn’t have the chance to use them yet, but I assume they will give you much tighter control on memory, and also probably will allow us to use the whole 256 possible values of a byte
There are also these luau implementations of compression algorithms you can find on the forum, such as this one LZW text-compression
1 Like
If you convert the coins to higher base you can turn write that 100 with fewer bytes. And instead of using c as the header you can use another less common character, which allows your alphabet to be more continuous.
1 Like
This post is too vague … I need to know what the data you’re saving is…
Is it like this:
local items = {“Item1”, “Item2”, “Item3”}
or like this:
local items = {Item1 = 2, Item2 = 5, Item3 = 3}
Compressing data don’t work out well in Roblox due to the fact it’s hard to have a large enough amount to compress. Small data amounts actually turn out larger “Compressed”.
Let’s say I have 12 items or options a player could have.
This is a completely different way to save data…
- 01 - Sniper
- 02 - Gum
- 03 - Boots
- 04 - Shield
- 05 - Sword
- 06 - Armor
- 07 - Backpack
- 08 - Grappling Hook
- 09 - Magic Wand
- 10 - Flashlight
- 11 - Map
- 12 - Key
Right now I have: Gum, a Sword and a Map.
I could save that to the datastore as “020511”
Then read back 2 digits (strings) at a time to get: 02 05 11. Those would be the numbers (strings) to the items. This way we can have a many items or options by all kinds of names, each with their own number. So when we save to the datastore the data is just a short string, making it super small. When we read it back for use, it would need a conversion function to decode the save back to the items or options.
Could even use 4 digits for item and number of them. Either way you will cut down data needed to save by a mile. 4 digits would be: Up to 99 things and up to 99 of them.
Well without a reply I have no choice but to just guess.
This is a bit hardcore … ya that is a compression Module.
This will need some love to get to work for you the way you wish.
-- ModuleScript in ReplicatedStorage named Compress
local HttpService = game:GetService("HttpService")
local Compress = {}
local dict, len = {}, 0
for i = 32, 127 do
if i ~= 34 and i ~= 92 then
local c = string.char(i)
dict[c], dict[len] = len, c
len = len + 1
end
end
local escMap = {}
for i = 1, 34 do
i = ({34, 92, 127})[i - 31] or i
local c, e = string.char(i), string.char(i + 31)
escMap[c], escMap[e] = e, c
end
local function escape(s)
return (s:gsub("[%c\"\\]", function(c)
return "\127" .. escMap[c]
end))
end
local function unescape(s)
return (s:gsub("\127(.)", function(c)
return escMap[c]
end))
end
local function toBase93(n)
local val = ""
repeat
local rem = n % 93
val = dict[rem] .. val
n = (n - rem) / 93
until n == 0
return val
end
local function toBase10(val)
local n = 0
for i = 1, #val do
n = n + 93^(i - 1) * dict[val:sub(-i, -i)]
end
return n
end
function Compress.compressTable(tbl)
local serializedData = HttpService:JSONEncode(tbl)
local org = #serializedData
local data = Compress.compress(serializedData)
return data, org
end
function Compress.compress(txt)
local d = {}
for k, v in pairs(dict) do
d[k] = v
end
local key, seq, size = "", {}, #d
local width, spans, span = 1, {}, 0
local function listKey(k)
local v = toBase93(d[k])
if #v > width then
width, span, spans[width] = #v, 0, span
end
seq[#seq + 1] = (" "):rep(width - #v) .. v
span = span + 1
end
txt = escape(txt)
for i = 1, #txt do
local c = txt:sub(i, i)
local new = key .. c
if d[new] then
key = new
else
listKey(key)
key, size = c, size + 1
d[new], d[size] = size, new
end
end
listKey(key)
spans[width] = span
return table.concat(spans, ",") .. "|" .. table.concat(seq)
end
function Compress.decompress(txt)
local d = {}
for k, v in pairs(dict) do
d[k] = v
end
local seq, spans, content = {}, txt:match("(.-)|(.*)")
local groups, start = {}, 1
for span in spans:gmatch("%d+") do
local w = #groups + 1
groups[w] = content:sub(start, start + span * w - 1)
start = start + span * w
end
local prev
for w = 1, #groups do
for val in groups[w]:gmatch(('.'):rep(w)) do
local entry = d[toBase10(val)]
if prev then
if entry then
seq[#seq + 1] = entry
d[#d + 1] = prev .. entry:sub(1, 1)
else
entry = prev .. prev:sub(1, 1)
seq[#seq + 1] = entry
d[#d + 1] = entry
end
else
seq[1] = entry
end
prev = entry
end
end
return unescape(table.concat(seq))
end
function Compress.deserializeTable(str)
return HttpService:JSONDecode(str)
end
function Compress.printTable(tbl, indent)
indent = indent or ""
for key, value in pairs(tbl) do
if type(value) == "table" then
print(indent .. key .. ":")
Compress.printTable(value, indent .. " ")
else
print(indent .. key .. ": " .. tostring(value))
end
end
end
return Compress
-- ServerScript in ServerScriptService
local Players = game:GetService("Players")
local DataStoreService = game:GetService("DataStoreService")
local Compress = require(game:GetService("ReplicatedStorage"):WaitForChild("Compress"))
local playerDataStore = DataStoreService:GetDataStore("PlayerDataStore")
local function onPlayerAdded(player)
local playerId = player.UserId
local loadedData
local loadSuccess, loadData = pcall(function()
return playerDataStore:GetAsync(tostring(playerId))
end)
if loadSuccess and loadData then
local decompressedData = Compress.decompress(loadData)
loadedData = Compress.deserializeTable(decompressedData)
print("Loaded Data:")
Compress.printTable(loadedData)
else
print("No data found for player or failed to load data.")
loadedData = {}
end
player.AncestryChanged:Connect(function(_, parent)
if not parent then
loadedData = { score = 100, level = 5, inventory = {"sword", "shield"} }
local compressedData, org = Compress.compressTable(loadedData)
local com = #compressedData
print("Original Size: " .. org .. " bytes")
print("Compressed Size: " .. com .. " bytes")
local success, err = pcall(function()
playerDataStore:SetAsync(tostring(playerId), compressedData)
end)
if success then
print("Data saved successfully.")
else
warn("Failed to save data: " .. err)
end
end
end)
end
Players.PlayerAdded:Connect(onPlayerAdded)
ion even script you know whats up
i fw memory alocation heavy…
Roblox’s buffer library will be the fastest and easiest way to compress data.
I don’t recommend any other compression library (personal opinion and some facts you can research on your own)
Some of the reasons you’d use the buffer library and why you should, is simply just because it’s built for roblox’s engine and is performance optimized. It’s a roblox library so it’s constantly being updated and worked on behind the scenes, no external dependencies, and it’s memory efficient.
Of course, the option is yours, this is just my opinion. buffer | Documentation - Roblox Creator Hub
I also recommend watching this video if you’re new to using the buffer library. (or just read the documentation on it)
1 Like