This how do i extremely reduced data packet size

Resources Community Tutorials
, , , ,
#1

Okay so i want to share to you guys a bit about reducing data packet size (standard option).

Commonly firing remotes with 1 arg string that costed upto 32-70+ bytes (scaled with length and may more) so how do you reduced it? so i used some method to reduce it.

  1. string.pack
  • using string pack can reduce size upto 2-3x (or more) like 32 to 16.
  1. Use external encode text strings reduce sizer
  • Depend to the encoder system work to request packet data to be compressed
  1. JSON Encode
  • i use this method too for reduce size upto 2x smaller.

Finish

if you use these 2 method in row that able make your data size compressed upto 2/4 bytes

  • Important for using JSON Encode

using JSON Encode on multiple times in short time can limit you for while, so while your limited your not able to use JSON API

if you got your self opinion or source, fell free to share it to us!

Thanks

5 Likes
Save your player data with ProfileService! (DataStore Module)
#2

Can you explain more? I do not understand a thing, the title is unclear and I don’t get what the tutorial is about.

2 Likes
#3

Depending on what you’re sending, you can also use serialization. For example:

RemoteEvent, Server vs Client
“BlueCar”, “OpenLeftDoor”, “OpenRightDoor” → “0001”, “0002”, “0003”

Then in some conversion table, you can store the definitions of these serialized IDs. Like so:
“0001” = “BlueCar”
“0002” = “OpenLeftDoor”
“0003” = “OpenRightDoor”

With that, you get the same deserialized arguments but in way less bytes. Bonus points if you decide to call string.pack, JSONEncode, and possibly a custom text compressor on top of all that.

Obviously this way of serialization only works depending on what you’re sending, if you send high resolution Vector3s, it might be conflicting performance to serialize it to reconstruct it a millisecond later. Or possibly if you’re sending non-unique text, like a player typed message.

1 Like
#4

Personally, I use waffles’s text compressor to compress my strings. I can get tens of thousands of characters of text compressed down to a small fraction of its original size(~20%) then decompress back exactly how it originally was without any difficulty. From what I understand about it, the more patterns in the string the more compressed it will be.

You can find more information on what I’m talking about in this topic: Text compression

If you just wanna copy and paste the code into a module without checking it out well, here ya go:

local dictionary, length = {}, 0
for i = 32, 127 do
	if i ~= 34 and i ~= 92 then
		local c = string.char(i)
		dictionary[c], dictionary[length] = length, c
		length = length + 1
	end
end

local escapemap = {}
for i = 1, 34 do
	i = ({34, 92, 127})[i-31] or i
	local c, e = string.char(i), string.char(i + 31)
	escapemap[c], escapemap[e] = e, c
end
local function escape(s)
	return (s:gsub("[%c\"\\]", function(c)
		return "\127"..escapemap[c]
	end))
end

local function unescape(s)
	return (s:gsub("\127(.)", function(c)
		return escapemap[c]
	end))
end

local function copy(t)
	local new = {}
	for k, v in pairs(t) do
		new[k] = v
	end
	return new
end

local function tobase93(n)
	local value = ""
	repeat
		local remainder = n%93
		value = dictionary[remainder]..value
		n = (n - remainder)/93
	until n == 0
	return value
end

local function tobase10(value)
	local n = 0
	for i = 1, #value do
		n = n + 93^(i-1)*dictionary[value:sub(-i, -i)]
	end
	return n
end

local function compress(text)
	local dictionary = copy(dictionary)
	local key, sequence, size = "", {}, #dictionary
	local width, spans, span = 1, {}, 0
	local function listkey(key)
		local value = tobase93(dictionary[key])
		if #value > width then
			width, span, spans[width] = #value, 0, span
		end
		sequence[#sequence+1] = (" "):rep(width - #value)..value
		span = span + 1
	end
	text = escape(text)
	for i = 1, #text do
		local c = text:sub(i, i)
		local new = key..c
		if dictionary[new] then
			key = new
		else
			listkey(key)
			key, size = c, size+1
			dictionary[new], dictionary[size] = size, new
		end
	end
    listkey(key)
	spans[width] = span
    return table.concat(spans, ",").."|"..table.concat(sequence)
end

local function decompress(text)
	local dictionary = copy(dictionary)
	local sequence, spans, content = {}, text:match("(.-)|(.*)")
	local groups, start = {}, 1
	for span in spans:gmatch("%d+") do
		local width = #groups+1
		groups[width] = content:sub(start, start + span*width - 1)
		start = start + span*width
	end
	local previous;
	for width = 1, #groups do
		for value in groups[width]:gmatch(('.'):rep(width)) do
			local entry = dictionary[tobase10(value)]
			if previous then
				if entry then
					sequence[#sequence+1] = entry
					dictionary[#dictionary+1] = previous..entry:sub(1, 1)
				else
					entry = previous..previous:sub(1, 1)
					sequence[#sequence+1] = entry
					dictionary[#dictionary+1] = entry
				end
			else
				sequence[1] = entry
			end
			previous = entry
		end
	end
	return unescape(table.concat(sequence))
end
return {compress = compress, decompress = decompress}

I highly suggest checking it out though, whether you’re going to use it or not.

1 Like