Any way to optimize my object serializer?

I recently created a simple object serializer which stores objects in datastore’s. I am wondering if there is anyway to optimize my script. Here is the code (you can also find the post that has more info on it here):

-- Services
local players = game:GetService("Players")
local replicatedStorage = game:GetService("ReplicatedStorage")
local dataStoreService = game:GetService("DataStoreService")
local httpService = game:GetService("HttpService")
local dataStore = dataStoreService:GetDataStore("Example_Serialize_System")

local serializeE = replicatedStorage.remotes.events.serialize

local location = workspace.Plot.plane.itemHolder

local dataLoaded = false
local tries = 3

local rad = math.rad
local cframe = CFrame.new
local angles = CFrame.fromEulerAnglesXYZ
local vector3 = Vector3.new
local color = Color3.new

-- Saves models currently placed on the plane/plot
local function serialize(plr)
	if dataLoaded then
		local key = plr.UserId
		local data = {}
		
		-- Saves properties from all objects
		for _, objs in pairs(location:GetChildren()) do
			for i, obj in pairs(objs:GetDescendants()) do
				if obj:IsA("BasePart") then
					table.insert(data, {
						-- p = position
						["p"] = {obj.CFrame.X, obj.CFrame.Y, obj.CFrame.Z, obj.Orientation.X, obj.Orientation.Y, obj.Orientation.Z};
						["s"] = {obj.Size.X, obj.Size.Y, obj.Size.Z}; -- s = size
						["c"] = {obj.Color.R, obj.Color.G, obj.Color.B}; -- c = color
						["n"] = obj.Name; -- n = name
						["t"] = obj.Transparency;
						["mdlN"] = objs.Name; -- t = transparency
						["m"] = string.sub(tostring(obj.Material), 15, string.len(tostring(obj.Material))); -- m = material
						["isPri"] = objs.PrimaryPart == obj; -- isPri = isPrimaryPart
						["f"] = objs:GetDescendants()[1] == obj -- f = firstObject
					})
				end
			end
		end
		
		local count = 0
		local success, err
		
		-- To prevent errors and data loss
		repeat
			success, err = pcall(function()
				dataStore:SetAsync(key, data)
			end)
			
			count = count + 1
		until success or count >= tries
		
		if not success then
			warn("Failed to serialize data: Error code " .. tostring(err))
			
			return
		end
	end
end

-- Loads the data back into the game
local function deserialize(plr)
	local key = plr.UserId
	local serializedData
	
	local count = 0
	local success, err
	
	repeat
		success, err = pcall(function()
			serializedData = dataStore:GetAsync(key)
		end)
		
		count = count + 1
	until success or count >= tries
	
	if not success then
		warn("Failed to read data: Error code " .. tostring(err))
		
		return
	end
	
	if serializedData then
		local model
		
		-- Loads data
		for i, data in pairs(serializedData) do
			-- Makes sure a model is created only per model
			if data.f then
				model = Instance.new("Model")
				model.Name = data.mdlN
				
				model.Parent = location
			end
			
			local part = Instance.new("Part")
			part.Anchored = true
			part.CFrame = cframe(data.p[1], data.p[2], data.p[3])*angles(rad(data.p[4]), rad(data.p[5]), rad(data.p[6]))
			part.Size = vector3(data.s[1], data.s[2], data.s[3])
			part.Color = color(data.c[1], data.c[2], data.c[3])
			part.TopSurface = Enum.SurfaceType.SmoothNoOutlines
			part.BottomSurface = Enum.SurfaceType.SmoothNoOutlines
			part.LeftSurface = Enum.SurfaceType.SmoothNoOutlines
			part.RightSurface = Enum.SurfaceType.SmoothNoOutlines
			part.Name = data.n
			part.Material = Enum.Material[data.m]
			part.Transparency = data.t
			part.Parent = model
			
			-- Handles primary parts
			if data.isPri and model then
				model.PrimaryPart = part
				model.PrimaryPart.CanCollide = false
			end
			
			wait(0.1) -- this can be removed
		end
		
		dataLoaded = true
	else
		dataLoaded = true
		
		serialize(plr)
	end
end

-- calls
players.PlayerAdded:Connect(deserialize)
players.PlayerRemoving:Connect(serialize)
serializeE.OnServerEvent:Connect(serialize)

game:BindToClose(function()
	for i, plr in pairs(players:GetChildren()) do
		serialize(plr)
	end
end)

One change you might consider is compressing numbers info Base64. Also, it might be more memory efficient to convert everything into a string. I know that tables for instance have a memory overhead in addition to whatever you put in them. Instead, using the characters “{}” is 2 bytes.

I agree with AstroCode on both points. Base64 instead of decimal for integers is a lot more efficient as the integers get stored in string form.

I also would make it a string instead of a table, especially as you have it as an array of objects, and just use a single character delimiter, e.g. ; between each array item to use string.split on later when decompressing the data.

Finally, keys like isPri and mdlN is a waste of characters. You’ve got other chars available so I’d use them.

I’m not convinced by this. Using a base64 string instead of a number already adds two extra characters (opening and closing quotes).

For integers, a base 64 representation (as opposed to base 10) only starts being more space efficient for integers greater than 99999. Using a base-64 encoded floating-point format also wouldn’t be especially helpful because every single-precision float (4 bytes) would be expanded out to 8 characters anyway.


Once saved in a Datastore, the data is converted into a string representation anyway. The overhead for tables is minimal (a few bytes). This means that converting the data to a string and adding delimiter characters etc. offers no benefit in terms of conserving space, because you’re just adding your own overhead of the same size. You can check this by looking at a JSON-encoded version of the data.


I think the main thing you can do is tailor your saving/loading system to the requirements of your game/project. It’s very unlikely that you actually need to save all of these properties of every part in a certain area.

I noticed you linked to a general model serializer - in which case you would need store a lot more information about arbitrary parts - but I think this just indicates that a general solution for saving data isn’t ever going to be optimal.

For example, a game where you can place down pre-existing objects should not be saving every property of every part of each placed object; it should be saving the names of the objects placed down and the position/rotation, if applicable.

2 Likes

Depends on the data, especially the nested tables with string keys here (each key being 3 times the bytes needed for a single character key once encoded with quotes). Personally I found on my city building game I reduced the byte count by nearly 1/3 by making it a single string with smarter keys and delimiters than when it was JSON encoded straight from it’s table form.

Also base64 doesn’t add extra opening and closing characters if it’s already a single string, which is why that was preferable in my case over representing coordinates as vectors of decimals. I agree if it’s still in table form, representing as a base64 string instead of a literal number worsens it in many cases. My recommendations from personal experience were to be considered in conjunction with one another rather than separately.


If anyone's curious how much I save with a few other tips:

Before:

  • Lua table encoded straight to JSON.
  • Single character keys, similar to the post example.
  • All numbers and integers left as literal numbers and integers. All booleans left as literal booleans.
  • Coordinates represented by tables of 2 entries (as Vector2 does not encode).
  • 2,151 bytes in total.
{"s":1,"b":[[3,2,[20,4],0,0,0,false,false],[35,1,[21,14],0,0,0,false,false],[3,2,[18,4],0,0,0,false,false],[3,2,[16,4],0,0,0,false,false],[3,4,[13,4],3,0,0,false,false],[3,1,[11,5],2,0,0,false,false],[3,1,[19,1],2,0,0,false,false],[3,5,[17,1],2,0,0,false,false],[11,1,[16,9],1,0,5,false,false],[3,3,[12,8],0,0,0,false,false],[3,3,[10,8],0,0,0,false,false],[3,2,[13,11],2,0,0,false,false],[3,3,[11,11],2,0,0,false,false],[3,3,[9,11],2,0,0,false,false],[3,3,[8,8],0,0,0,false,false],[35,1,[18,14],0,0,20,false,false],[34,1,[26,13],1,0,15,false,false],[4,1,[16,11],1,0,8,false,false],[3,3,[19,11],2,0,0,false,false],[3,5,[21,11],2,0,0,false,false],[3,4,[18,8],0,0,0,false,false],[3,5,[20,8],0,0,0,false,false],[3,5,[23,11],2,0,0,false,false],[3,1,[22,8],0,0,0,false,false],[13,1,[22,4],0,0,15,false,false],[14,1,[7,4],3,0,7,false,false],[12,1,[7,3],0,0,0,false,false],[42,1,[15,1],2,0,6,false,false],[3,5,[13,1],2,0,0,false,false],[3,5,[11,1],2,0,0,false,false],[3,4,[9,1],2,0,0,false,false],[3,1,[5,8],3,0,0,false,false],[3,2,[5,10],3,0,0,false,false],[3,5,[26,6],1,0,0,false,false],[3,5,[26,8],1,0,0,false,false],[53,1,[11,14],0,0,23,false,false]],"k":559372,"d":"Wonderland","l":5,"r":[[0,[18,6],0,false,false],[0,[8,4],0,false,false],[0,[6,12],0,false,false],[0,[14,4],0,false,false],[0,[8,12],0,false,false],[0,[12,6],0,false,false],[0,[24,4],0,false,false],[0,[20,6],0,false,false],[0,[8,2],0,false,false],[0,[22,6],0,false,false],[0,[10,2],0,false,false],[0,[22,12],0,false,false],[0,[14,6],0,false,false],[0,[12,12],0,false,false],[0,[24,12],0,false,false],[0,[20,12],0,false,false],[0,[16,12],0,false,false],[0,[24,2],0,false,false],[0,[24,6],0,false,false],[0,[24,10],0,false,false],[0,[6,8],0,false,false],[0,[24,8],0,false,false],[0,[10,6],0,false,false],[0,[16,6],0,false,false],[0,[6,10],0,false,false],[0,[14,2],0,false,false],[0,[22,2],0,false,false],[0,[20,2],0,false,false],[0,[14,8],0,false,false],[0,[16,2],0,false,false],[0,[18,2],0,false,false],[0,[14,12],0,false,false],[0,[14,10],0,false,false],[0,[10,12],0,false,false],[0,[12,2],0,false,false],[0,[18,12],0,false,false],[0,[8,6],0,false,false],[0,[6,6],0,false,false]],"c":[12]}

After:

  • Single string compressed using my own delimiters and built-in assumptions about given items (i.e. what sircfenner said about tailoring the system to the requirements of your project).
  • Values that are equal to the default (e.g. a coordinate placement of 0 in either x or y, default rotation, 0 citizens in the building, 0 workers, etc.) omitted entirely - you can spot these by places where two vertical bars are next to each other, or where there is no data between a bar and a colon or semicolon.
  • All integers encoded to a base 64 numbering system where the integer number can exceed 9. Where integers are used as enumerators with no more than 10 items, they are kept as decimal.
  • All booleans converted to 0 and 1.
  • Same single character keys, unchanged from the “before” table.
  • 810 bytes in total.
:s:1:r:A|S,G||;A|I,E||;A|G,M||;A|O,E||;A|I,M||;A|M,G||;A|Y,E||;A|U,G||;A|I,C||;A|W,G||;A|K,C||;A|W,M||;A|O,G||;A|M,M||;A|Y,M||;A|U,M||;A|Q,M||;A|Y,C||;A|Y,G||;A|Y,K||;A|G,I||;A|Y,I||;A|K,G||;A|Q,G||;A|G,K||;A|O,C||;A|W,C||;A|U,C||;A|O,I||;A|Q,C||;A|S,C||;A|O,M||;A|O,K||;A|K,M||;A|M,C||;A|S,M||;A|I,G||;A|G,G||:d:Wonderland:c:M:b:D|C|U,E||||;j|B|V,O||||;D|C|S,E||||;D|C|Q,E||||;D|E|N,E|3|||;D|B|L,F|2|||;D|B|T,B|2|||;D|F|R,B|2|||;L|B|Q,J|1|A|F|;D|D|M,I||||;D|D|K,I||||;D|C|N,L|2|||;D|D|L,L|2|||;D|D|J,L|2|||;D|D|I,I||||;j|B|S,O||A|U|;i|B|a,N|1|A|P|;E|B|Q,L|1|A|I|;D|D|T,L|2|||;D|F|V,L|2|||;D|E|S,I||||;D|F|U,I||||;D|F|X,L|2|||;D|B|W,I||||;N|B|W,E||A|P|;O|B|H,E|3|A|H|;M|B|H,D||||;q|B|P,B|2|A|G|;D|F|N,B|2|||;D|F|L,B|2|||;D|E|J,B|2|||;D|B|F,I|3|||;D|C|F,K|3|||;D|F|a,G|1|||;D|F|a,I|1|||;1|B|L,O||A|X|:k:CIkM:l:F

This is of course one specific example, but even in the absolute worst case of no default values to omit from the data, it still reduces the byte count quite a lot. I haven’t encountered any cases where the byte count is the same or higher.

No benefit from defaults: 992 bytes total
  • 992 bytes total.
:s:1:r:A|S,G|0|0;A|I,E|0|0;A|G,M|0|0;A|O,E|0|0;A|I,M|0|0;A|M,G|0|0;A|Y,E|0|0;A|U,G|0|0;A|I,C|0|0;A|W,G|0|0;A|K,C|0|0;A|W,M|0|0;A|O,G|0|0;A|M,M|0|0;A|Y,M|0|0;A|U,M|0|0;A|Q,M|0|0;A|Y,C|0|0;A|Y,G|0|0;A|Y,K|0|0;A|G,I|0|0;A|Y,I|0|0;A|K,G|0|0;A|Q,G|0|0;A|G,K|0|0;A|O,C|0|0;A|W,C|0|0;A|U,C|0|0;A|O,I|0|0;A|Q,C|0|0;A|S,C|0|0;A|O,M|0|0;A|O,K|0|0;A|K,M|0|0;A|M,C|0|0;A|S,M|0|0;A|I,G|0|0;A|G,G|0|0:d:Wonderland:c:M:b:D|C|U,E|0|A|A|0;j|B|V,O|0|A|A|0;D|C|S,E|0|A|A|0;D|C|Q,E|0|A|A|0;D|E|N,E|3|A|A|0;D|B|L,F|2|A|A|0;D|B|T,B|2|A|A|0;D|F|R,B|2|A|A|0;L|B|Q,J|1|A|F|0;D|D|M,I|0|A|A|0;D|D|K,I|0|A|A|0;D|C|N,L|2|A|A|0;D|D|L,L|2|A|A|0;D|D|J,L|2|A|A|0;D|D|I,I|0|A|A|0;j|B|S,O|0|A|U|0;i|B|a,N|1|A|P|0;E|B|Q,L|1|A|I|0;D|D|T,L|2|A|A|0;D|F|V,L|2|A|A|0;D|E|S,I|0|A|A|0;D|F|U,I|0|A|A|0;D|F|X,L|2|A|A|0;D|B|W,I|0|A|A|0;N|B|W,E|0|A|P|0;O|B|H,E|3|A|H|0;M|B|H,D|0|A|A|0;q|B|P,B|2|A|G|0;D|F|N,B|2|A|A|0;D|F|L,B|2|A|A|0;D|E|J,B|2|A|A|0;D|B|F,I|3|A|A|0;D|C|F,K|3|A|A|0;D|F|a,G|1|A|A|0;D|F|a,I|1|A|A|0;1|B|L,O|0|A|X|0:k:CIkM:l:F

The % benefit decreases slightly as the number of buildings and roads represented in the datasave increase, and likewise the benefit increases as larger integers are used where only one base 64 character is needed. There is still a consistent benefit per building of 50-60%.


That said, there are still things you can do to make your storage smarter, so I agree just making it a string and converting some numbers isn’t enough. Consider what is essential, vs what can be calculated/inferred. Consider default values and only saving things that are non-default.

Another point to consider is if I have a house type called Small House and the user has placed 10 of these around the map. It’s not great to save it as 10 discrete items in your array, instead each item could have the ability to provide an array for each item inside it, which will save you bytes for every clone in the user’s save file. I don’t personally do this as my compression is good enough that the user runs out of land + all available expansion areas before they can possibly fill their datasave, even in the worst case, most densly-packed city possible, with all non-default values.

1 Like