What's the best thing for this scenario?

[gravitycoil828 (Gravity)]

This is a bit of an awkward question, but I’m making an open world building game and I made a serialiser (compressed also with JSON.Encode).

Sometimes, the strings spat out can be more then 1 million characters [even compressed, 52 instances is averaging 12000 characters] sometimes so I don’t know about using Datastore to save the serialised builds.

Should I stick to datastore and compress it more [through some method] or use a third party (and if so what one?)

local HttpService = game:GetService("HttpService")
local serialiser = {}

function serialiser.Serialise(obj)
	local function serialiseInstance(instance)
		local data = {
			ClassName = instance.ClassName,
			Name = instance.Name,
			Properties = {},
			Children = {}
		}

		local hasPosition = pcall(function() return instance.Position end)

		if hasPosition then
			local propertiesToSerialize = {"Position", "Rotation", "Material", "Color", "Size", "Reflectance", "Transparency", "CanCollide", "Name"}
			for _, propertyName in ipairs(propertiesToSerialize) do
				local success, propertyValue = pcall(function() return instance[propertyName] end)
				if success then
					local propertyData = {}
					if typeof(propertyValue) == "BrickColor" then
						propertyData.Type = "BrickColor"
						propertyData.Value = propertyValue.Name
					elseif typeof(propertyValue) == "Vector3" then
						propertyData.Type = "Vector3"
						propertyData.Value = {propertyValue.X, propertyValue.Y, propertyValue.Z}
					elseif typeof(propertyValue) == "Color3" then
						propertyData.Type = "Color3"
						propertyData.Value = {propertyValue.R, propertyValue.G, propertyValue.B}
					elseif typeof(propertyValue) == "Vector2" then
						propertyData.Type = "Vector2"
						propertyData.Value = {propertyValue.X, propertyValue.Y}
					elseif typeof(propertyValue) == "UDim2" then
						propertyData.Type = "UDim2"
						propertyData.Value = {propertyValue.X.Scale, propertyValue.X.Offset, propertyValue.Y.Scale, propertyValue.Y.Offset}
					elseif typeof(propertyValue) == "CFrame" then
						propertyData.Type = "CFrame"
						propertyData.Value = {propertyValue:components()}
					elseif typeof(propertyValue) == "UDim" then
						propertyData.Type = "UDim"
						propertyData.Value = {propertyValue.Scale, propertyValue.Offset}
					elseif typeof(propertyValue) == "EnumItem" then
						propertyData.Type = "EnumItem"
						propertyData.EnumType = tostring(propertyValue.EnumType)
						propertyData.Value = propertyValue.Name
					elseif typeof(propertyValue) == "number" then
						propertyData.Type = "Number"
						propertyData.Value = tostring(propertyValue)
					else
						propertyData.Type = typeof(propertyValue)
						propertyData.Value = propertyValue
					end
					data.Properties[propertyName] = propertyData
				end
			end
		else
			local success, properties = pcall(function() return APIService:GetProperties(instance) end)

			if success and properties then
				for propertyName, propertyValue in pairs(properties) do
					local propertyData = {}
					if typeof(propertyValue) == "BrickColor" then
						propertyData.Type = "BrickColor"
						propertyData.Value = propertyValue.Name
					elseif typeof(propertyValue) == "Vector3" then
						propertyData.Type = "Vector3"
						propertyData.Value = {propertyValue.X, propertyValue.Y, propertyValue.Z}
					elseif typeof(propertyValue) == "Color3" then
						propertyData.Type = "Color3"
						propertyData.Value = {propertyValue.R, propertyValue.G, propertyValue.B}
					elseif typeof(propertyValue) == "Vector2" then
						propertyData.Type = "Vector2"
						propertyData.Value = {propertyValue.X, propertyValue.Y}
					elseif typeof(propertyValue) == "UDim2" then
						propertyData.Type = "UDim2"
						propertyData.Value = {propertyValue.X.Scale, propertyValue.X.Offset, propertyValue.Y.Scale, propertyValue.Y.Offset}
					elseif typeof(propertyValue) == "CFrame" then
						propertyData.Type = "CFrame"
						propertyData.Value = {propertyValue:components()}
					elseif typeof(propertyValue) == "UDim" then
						propertyData.Type = "UDim"
						propertyData.Value = {propertyValue.Scale, propertyValue.Offset}
					elseif typeof(propertyValue) == "EnumItem" then
						propertyData.Type = "EnumItem"
						propertyData.EnumType = tostring(propertyValue.EnumType)
						propertyData.Value = propertyValue.Name
					elseif typeof(propertyValue) == "number" then
						propertyData.Type = "Number"
						propertyData.Value = tostring(propertyValue)
					else
						propertyData.Type = typeof(propertyValue)
						propertyData.Value = propertyValue
					end
					data.Properties[propertyName] = propertyData
				end
			end
		end

		for _, child in ipairs(instance:GetChildren()) do
			local childData = serialiseInstance(child)
			if childData then
				table.insert(data.Children, childData)
			else
				warn("Skipping child: " .. child.Name)
			end
		end

		return data
	end

	local serialize = serialiseInstance(obj)
	print(HttpService:JSONEncode(serialize))
	return HttpService:JSONEncode(serialize)
end

function serialiser.Deserialise(data)
	local function deserialiseInstance(data)
		local blockType = game.ReplicatedStorage.Files.BlockTypes:FindFirstChild(data.Name)
		local instance

		if blockType then
			instance = blockType:Clone()
		else
			instance = Instance.new(data.ClassName)
		end

		instance.Name = data.Name

		local parent = nil
		local success = pcall(function()
			for propertyName, propertyData in pairs(data.Properties) do
				local propertyValue
				if propertyData.Type == "BrickColor" then
					propertyValue = BrickColor.new(propertyData.Value)
				elseif propertyData.Type == "Vector3" then
					propertyValue = Vector3.new(unpack(propertyData.Value))
				elseif propertyData.Type == "Color3" then
					propertyValue = Color3.new(unpack(propertyData.Value))
				elseif propertyData.Type == "Vector2" then
					propertyValue = Vector2.new(unpack(propertyData.Value))
				elseif propertyData.Type == "UDim2" then
					propertyValue = UDim2.new(unpack(propertyData.Value))
				elseif propertyData.Type == "CFrame" then
					propertyValue = CFrame.new(unpack(propertyData.Value))
				elseif propertyData.Type == "UDim" then
					propertyValue = UDim.new(unpack(propertyData.Value))
				elseif propertyData.Type == "Number" then
					local convertedValue = tonumber(propertyData.Value)
					if convertedValue then
						propertyValue = convertedValue
					else
						propertyValue = propertyData.Value
					end
				elseif propertyData.Type == "EnumItem" then
					local enumType = Enum[propertyData.EnumType]
					if enumType then
						propertyValue = enumType[propertyData.Value]
					else
						warn("Failed to find Enum type: " .. propertyData.EnumType)
					end
				elseif propertyName == "Parent" then
					parent = propertyData.Value
				else
					propertyValue = propertyData.Value
				end
				if instance[propertyName] ~= nil and propertyName ~= "Parent" and propertyName ~= "DataCost" and propertyName ~= "ClassName" then
					local success, errorMsg = pcall(function()
						instance[propertyName] = propertyValue
					end)
					if not success then
						warn("Failed to set property: " .. propertyName .. " - " .. errorMsg)
					end
				elseif propertyName ~= "DataCost" and propertyName ~= "ClassName" and propertyName ~= "className" then
					warn("Property not found: " .. propertyName)
				end
			end
		end)

		if not success then
			warn("Failed to set properties for " .. data.Name)
		end

		for _, childData in ipairs(data.Children) do
			local childInstance = deserialiseInstance(childData)
			if instance.Name == "SpawnLocation" then
				if instance:FindFirstChild("SpawnDecal") then
					local a = 0
					for _, decal in pairs(instance:GetChildren()) do
						if instance:FindFirstChild("SpawnDecal") then
							a = a + 1
						end
						if a == 2 then
							instance:FindFirstChild("SpawnDecal"):Destroy()
							a = a - 1
						end
					end
				end
			end
			childInstance.Parent = instance
		end

		if parent then
			instance.Parent = game.Workspace:FindFirstChild(parent)
		end

		return instance
	end

	local deserialisedInstance = deserialiseInstance(HttpService:JSONDecode(data))
	print(deserialisedInstance)
	return deserialisedInstance
end

return serialiser```

[happya_x (Happya)]

(i think) when you store stuff to datastore it gets converted to json anyways, so you can save space by doing your own compression

so for example if you had 255 then it would be stored as “255” ( 3 bytes )
if you compress 255 to FF ( 2 bytes ) in hex then you can save space

[gravitycoil828 (Gravity)]

how would i do this though?

[i have no experience with compression]

[tlr22 (FusionOak)]

It’s basically writing it in such a way you can store values directly in binary inside the strings. So instead of using C to represent C, you use C to represent 01000011 (67 which is what ‘means’ C to computers).
This means you basically need to come up with a way to directly store data in binary and you’re just using a string because I don’t think datastores accept generic binary (though if they do, just use that). The above solution recommends storing HEX in strings which will work, but it’s more efficient to just pack the data directly into the characters. Though there are potentially edge cases. Like for example, strings often know when the string ends by having the last char be the number 0. That’s the null terminator. As such this hypothetical binary system could break if you ever use the number 0 directly because the string might prematurely end

But the moral is you need to find ways to reduce the amount of information.

If you are storing property names in the json, dont. Give each name a number. That way you save on all that data. If you are storing numbers, try to store them in a more data dense way like described above by both our posts instead of directly storing them in strings. These methods are a bit complex but overall you just need to take care to design a system that finds exploitable patterns to reduce the amount of information you need to completely reproduce the original data from the smallest possible state.

[junekept]

lua strings are not null-terminated & roblox has a built-in buffer type that can convert itself into a byte string afaik. So yeah you can pack whatever you want into a string like you said and just send it through with no problems.

Compression is a bit of a weird thing to learn but this is unfortunately the only real workaround to datastore limitations; going third party just causes a whole other headache and is not worth it (you have to pay for a server now, and if that server goes down your entire game does too). There’s probably also plenty of compression methods on the devforums already if you look around because this isn’t a new problem

[pullman45]

Worst comes to worst, why not divide data between multiple keys? Since each key can store up to 4 megabytes, that gives you 4 million characters per key assuming each character takes up exactly one byte.

[gravitycoil828 (Gravity)]

wait, do datastores have a particular limit on how much you can have for the entire thing?
originally i was thinking of having like a seperate datastore for each world [which ID is stored in a datastore key]

e.g.

local worldindex = JSON.Decode(worldidstore:GetAsync("Index"))

local page = worldidstore:GetAsync(worldindex)

local worldid = page[wid]

local id = dss:GetDataStore(id)

local index2 = JSON.Decode(id:GetAsync("Index"))

for _, page in pairs(index:ListKeysAsync(worldid) do

cont...

[happya_x (Happya)]

4mb per key

you can have infinite keys but theres an overall rate limit so you wouldnt be able to practically use infinite keys

[gravitycoil828 (Gravity)]

wdym by rate limit?

y’mean how many keys you can save at once, or how many keys that can be saved in a datastore?

[tlr22 (FusionOak)]

There is a limit to how many keys per minute you can save to and load from. The limit is a base number plus a constant times the amount of players. You need to ensure you stay within that threshold. So loading a giant map would take several minutes depending on player count and how much of that budget is being used elsewhere.

I can’t find the docs for it because they’ve changed so much otherwise I would include it.