How to compress big tables (Bypassing unrealiable remotes byte limit)

This tutorial is about compressing tables and how buffers work in general.

Say you got a big tables that either too big for networking or you just wanna save space, luckily buffer can help us, this works for every luau table as we encode it to a JSON table.

Step 1: Encode your table first.

I’m gonna use my experience’s (Private since still W.I.P) badges tables that I got from roproxy due to how many badges it has, it exceeds the 900 byte limit when I try to send it to client ! And we’ll get a warning !

local HttpService = game:GetService("HttpService")

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local Joueur = game:GetService("Players").PlayerAdded:Wait()

-- ATTENTION: MON TABLEAU  EST DEJA ON JSON DONC IL N'EST PAS NECESSAIRE DE LA FORMATTER MAIS DEPUIS LE VALEUR "data" EST IMPORTANTE NOUS NOUS FAIRE LE

-- WARNING: MY TABLE IS ALREADY ON JSON SO IT IS NOT NECESSARY TO FORMAT IT BUT SINCE THE “data” VALUE IS IMPORTANT WE WILL DO IT

local _mesBadges = HttpService:JSONDecode(HttpService:GetAsync("https://badges.roproxy.com/v1/universes/6206562942/badges")).data

ReplicatedStorage._envoyerLesInformations:FireClient(Joueur, _mesBadges)

We can now convert this table into JSON which essentially converts it to a string

ReplicatedStorage._envoyerLesInformations:FireClient(Joueur, HttpService:JSONEncode(_mesBadges))

But that makes it bigger! Which is why we’re gonna use buffer in the next step.

STEP 2: Compressing the table, utilizing buffer.

To make our table incredibly smaller and to bypass the byte limit (in my case scenario) we’ll convert the encoded table to a buffer since it is a string!

local _buffer = buffer.fromstring(HttpService:JSONEncode(_mesBadges))

print(_buffer)

ReplicatedStorage._envoyerLesInformations:FireClient(Joueur, _buffer)

It works, we successfully sent the information to the client, you’ll also notice that the IDs are different but do not fret, the content stays the same.

image

Our then-big table is now much smaller and can be used for various uses! But to make it a normal table again we’ll have to use it buffer.tostring(b:buffer)

STEP 3: Decoding.

Since we have compressed our table, it’s time to convert it back to how it was!

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local HttpService = game:GetService("HttpService")

ReplicatedStorage._envoyerLesInformations.OnClientEvent:Connect(function(_lesInformations)
	local _buffer = buffer.tostring(_lesInformations)
	local _tableau = HttpService:JSONDecode(_buffer)
	print(_tableau)
end)

It works! Our table is now back to how it was!

image

We’ve successfully compressed and decompressed a big table to send to the client!

Server:

local HttpService = game:GetService("HttpService")

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local Joueur = game:GetService("Players").PlayerAdded:Wait()

-- ATTENTION: MON TABLEAU  EST DEJA ON JSON DONC IL N'EST PAS NECESSAIRE DE LA FORMATTER MAIS DEPUIS LE VALEUR "data" EST IMPORTANTE NOUS NOUS FAIRE LE

-- WARNING: MY TABLE IS ALREADY ON JSON SO IT IS NOT NECESSARY TO FORMAT IT BUT SINCE THE “data” VALUE IS IMPORTANT WE WILL DO IT

local _mesBadges = HttpService:JSONDecode(HttpService:GetAsync("https://badges.roproxy.com/v1/universes/6206562942/badges")).data

local _buffer = buffer.fromstring(HttpService:JSONEncode(_mesBadges))

print(_buffer)

ReplicatedStorage._envoyerLesInformations:FireClient(Joueur, _buffer)

Client:

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local HttpService = game:GetService("HttpService")

ReplicatedStorage._envoyerLesInformations.OnClientEvent:Connect(function(_lesInformations)
	local _buffer = buffer.tostring(_lesInformations)
	local _tableau = HttpService:JSONDecode(_buffer)
	print(_tableau)
end)

Thank you ! !

16 Likes

In addition to this topic, I recomend to use EncodingService for greater buffer compression, and vice versa. For example, the CompressBuffer and DecompressBuffer methods.

Please never do any of that in a real game
You should never be structuring remotes to accept tables ever
Instead, pass primitive deterministic data types (buffers preferable)
They aren’t that hard to understand as people make them to be

Plus, UnreliableRemoteEvent is usually used often for frequently sent data, so you can’t really afford all this JSON or compression
You shouldn’t be using JSON to begin with, honestly
And you should not be using “serializers” either because you should be serializing or storing data yourself without middleware guessing your intent while burning every frame…

JSON encoding is just a heavy way to avoid writing a proper serializer. If you’re already using buffers, you might as well just pack the raw bytes yourself instead of relying on string conversion middleware.

A question here, are you sure that wouldn’t cause data loss? I tried and found out that CFrame and vector are lost when trying to send data.

1 Like