Is storing UserData in Binary worth it?

I have as of recently been looking into the bit32 libary and I have understood it pretty well and I looked into bitpacking, now I am wondering if storing a binary sequence instead of a dictionary for the player’s data would be worth it.

For context this is how the player’s data is stored currently, in the dictionary form:

{
Currency={Coins=,Level=,Xp=,Playtime=, so on...},
Inventory={InventoryObjectID=NumberOfObject,...}
}

Here is how I might store it in binary:

1011_0111_1010_1110  1011_0110_1010

The underscores are to be replaced by the binary form of the number of that currency/InventoryObject that the player has.

The half byte sequences like 1011 would represent the value names. Although I might have them at 6 bits or even a byte so that more than 8 values can be stored.

I am not sure if it would be worth the effort though, but what are your ideas?

2 Likes

It actually might be worth it depending on how much keys you are storing AND UPDATING as every time you append something to that array it performs a costly sizing operation which could be detrimental on the server and you got lots of people. However, unless you are making the next Starscape AND are planning to have players of more than I want to say 30 it isnt worth it

4 Likes

The currency keys would amount to around 5-10 (currency is refering to the data that works as currency aswell as user data; such as playtime). But the Inventory keys may get into the dozens or even hundreds.

Should I store Currency in table and Inventory in Binary then?

2 Likes

What I really meant by updating was “appending” (adding one to the end of the list). If you just create them and just stick to using those indexs you’ll be fine but you might get some cache locality straining problems if you are “flushing” (setting table to nil) the contents and reappending them

1 Like

Well, it all comes up to how difficult it is to update my storage system.

Here is my script for loading, saving, and managing data changes. Basically the backbone of my datastorage system.

local RepStore = game:GetService("ReplicatedStorage")
local SvrStore = game:GetService("ServerStorage")
local DatStore = game:GetService("DataStoreService")
local Players = game:GetService("Players")
local PlayerDataFolder = RepStore.PlayerData

local InfoStore = DatStore:GetDataStore("PlayerData")
local ToCall = game.ServerStorage.Events.DataCall

local ServerStore = DatStore:GetDataStore("ServerStore")

local CurFormat = {"Level","Token","Xp","Coin","Playtime","Streak","GameJoinD"}
local Conversions = {Score="Xp"}

local function UnloadPlayerData(Player: Player)
	
	local IDList = ServerStore:GetAsync("IDlist")
	if not IDList then IDList = {} end
	if not table.find(IDList,Player.UserId) then
		table.insert(IDList,Player.UserId)
		ServerStore:SetAsync("IDlist",IDList)
	end
	local PlayerID = Player.UserId
	local suc, err = pcall(function()
		local PlayerData = InfoStore:GetAsync(PlayerID)
		if not PlayerData then -- adds player data
			warn("data empty")	
			PlayerData = {}
			PlayerData.Inventory = {}
			PlayerData.Currency = {}
			local Currency = PlayerData.Currency
			local Inventory = PlayerData.Inventory
			for i, v in pairs(CurFormat) do -- goes through the format
				if v == "GameJoinD" then
					Currency[v] = os.time() -- os date 
				else
					
					Currency[v] = 0
				end
			end
		end
		local PlayerFolder = Instance.new("Folder")
		local CurrencyFolder = Instance.new("Folder")
		local InventoryFolder = Instance.new("Folder")
		PlayerFolder.Name = PlayerID -- adds currency and inventory folder
		PlayerFolder.Parent = PlayerDataFolder
		CurrencyFolder.Name = "Currency"
		CurrencyFolder.Parent = PlayerFolder
		InventoryFolder.Name = "Inventory"
		InventoryFolder.Parent = PlayerFolder
		
		if not PlayerData.Currency and PlayerData.Inventory then -- adds ^
			PlayerData.Currency = {}
			PlayerData.Inventory = {}
		end
		local Currency = PlayerData.Currency
		local Inventory = PlayerData.Inventory
		
		
		for i, v in pairs(PlayerData) do -- converts old system to new 
			if table.find(CurFormat,i) then
				local suc, err = pcall(function()
					Currency[i] = v
					PlayerData[i] = nil
				end)
			end
		end
		
		for i, v in pairs(CurFormat) do
			if not Currency[v] then
				if v == "GameJoinD" then
					local timeS = os.time()
					Currency[v] = timeS
				else
				Currency[v] = 0
				end
			end
		end
		
		
		for i, v in pairs(Currency) do
			if Conversions[i] then
				local Conv = Conversions[i]
				Currency[Conv] = v
				Currency[i] = nil
			end
			if not table.find(CurFormat,i) then
				PlayerData[i] = nil
			end
		end
		
		for store, data in pairs(Currency) do
			local IntHolder = Instance.new("IntValue")
			IntHolder.Name = store
			IntHolder.Value = data
			IntHolder.Parent = CurrencyFolder
		end
		for object, amount in pairs(Inventory) do
			local IntHolder = Instance.new("IntValue")
			IntHolder.Name = object
			IntHolder.Value = amount
			IntHolder.Parent = InventoryFolder
		end

	end) if not suc then warn(err) end
end

local function PackPlayerData(Player: Player)
	local PlayerID = Player.UserId
	if PlayerID then
		local PlayerFolder = PlayerDataFolder:FindFirstChild(PlayerID)
		local success, err = pcall(function()
			if PlayerFolder then
				local PlayerData = {}
				
				local CurrencyFol = PlayerFolder:FindFirstChild("Currency")
				local InventoryFol = PlayerFolder:FindFirstChild("Inventory")
				
				PlayerData.Currency = {}
				PlayerData.Inventory = {}
				
				local CurrencyV = PlayerData.Currency
				local InventoryV = PlayerData.Inventory

				for i, v in pairs(CurrencyFol:GetChildren()) do
					CurrencyV[v.Name] = v.Value
				end
				for i, v in pairs(InventoryFol:GetChildren()) do
					InventoryV[v.Name] = v.Value
				end

				InfoStore:SetAsync(PlayerID,PlayerData)
			end
		end)
		if success then
			PlayerFolder:Destroy()
		else
			warn(err)
		end
	end
end

local function TranslateCall(ToTranslate: {})
	local callData = ToTranslate
	coroutine.wrap(function()
		local suc, err = pcall(function()
			for i, v in pairs(callData) do
				local PlrId = v.PlrId
				local Ttype = v.Type
				local value = v.Value
				local LPlrFol = PlayerDataFolder[PlrId]
				local SVal = LPlrFol.Currency[Ttype]
				SVal.Value += value
			end
		end) if not suc then warn(err) end
	end)()
end

Players.PlayerAdded:Connect(UnloadPlayerData)
Players.PlayerRemoving:Connect(PackPlayerData)
ToCall.Event:Connect(TranslateCall)
1 Like

Wait so you’re storing binary with numbers that are made from binary? What’s the point? You’re basically just increasing the data size?

2 Likes

bitpacking (storing data in binary) can significantly reduce the data that needs to be stored.

For example, the player’s storage size for my game may be around 2-3 KB, storing it in binary will make it possibly under 1000 bits or even 500 bits.

You can’t store in “Binary”? You do realise those numbers are made from unicode/ACSII. So apparently that’s binary??? What you are doing is not “bitpacking” you are adding more data.

He could encode the binary as a string then convert it to base 64, or really whatever base for that matter. its just that there is only 64 characters that dont give you the question mark symbol

1 Like

Yeah sure, you can use binary to reduce data needed to be stored, but you’ll still have to deal with Roblox servers limitation on pinging back and forth to the game server.

Is it worth saving that 2-3 KB just for 1 second faster? Probably not, you’re best off using normal datastore methods, because Roblox servers have limitations regardless.

If it were your own database, then it would be worth it.

2 Likes

You can’t write actual binary because it’s not an actual datatype in roblox. This means you’d have to use something external outisde roblox?(which would be inefficient)

2-3 KB is small enough, you’re trying to optimize for no reason.

Still, if you do want to try and make this as small as possible, you might want to try using a BitBuffer module.

https://dekkonot.github.io/bitbuffer/

1 Like

2-3 KB may be on one save, the player could have multiple saves on the datastore. And each save may get into the dozens of kilobytes. So upto 50-200KB (might) be saved.

200KB is still way below the 4MB limit.

There’s also this if you want some general compression.

The way the buffer is stored is as an unsigned integer.