BufferEncoder - Very efficient table to buffer serializer that doesn’t use schemas

Resources Community Resources
#1

BufferEncoder
BufferEncoder4096×2252 140 KB

BufferEncoder

Github | Wally | Current Release (.rbxm)

BufferEncoder is a table to buffer serializer that is designed to be as performant as possible while being able to serialize any type of table you want with all datatypes realistically used in tables.
It automatically serializes the table for you without having to define a schema for how the table is serialized - unlike most other serializers publicly available right now - and has very simple API to use.

Features

  • Very optimized and space-efficient
  • Supports any type of table (array, dictionary, mixed), cyclic tables, and any value for keys in dictionaries
  • Supports encoding every datatype that you’d realistically put in a table
  • Can encode custom values into 2 bytes if registered beforehand using encoder.enums.register() (eg. userdata from newproxy())
  • Can deduplicate repeated tables, strings, numbers, vectors, and enumitems to reduce data size
  • Has syncing from server to client for networking purposes
  • Has simple encryption that relies on psuedo-random number generation
  • Fully typed

Performance

It performs very fast compared to all serializers I’ve found publicly available
It may in some cases be faster than JSONEncode and JSONDecode, depending on the structure of your data

Benchmark

i was only able to find one module that works similar to mine, benchmarking with ones that use schemas isn’t really viable since they dont work the same as mine :expressionless:

Comparison between MessagePack and HTTPService

image
image1221×635 79.5 KB

Code 
--!optimize 2
--!native

local HTTP = game:GetService("HttpService")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local benchmarkmodules = ReplicatedStorage.benchmarkmodules

local BufferEncoder = require(ReplicatedStorage.BufferEncoder:Clone())
local MessagePack = require(benchmarkmodules.msgpack)

local c = 50

local tab, tab2, tab3, tab4, tab5, tab6 = {}, {}, {}, {}, {}, {}
for i = 1, c do tab[i] = {i, tostring(i), string.rep("a", i), true, false, nil, 4, "ok", true, 5436} end

for i = 1, c do tab2[i] = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" end

for i = 1, c do tab3[i] = i ^ 2 end

for i = 1, c do tab4[i] = {a = 'b', b = 'c', c = 'd', e = {a = 1, b = 2, c = "ok"}} end

for i = 1, c do 
	tab5[i] = {math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000), math.random(1, 1000)}
end

for i = 1, c do tab6[tostring(i)] = tostring(i) end

local tabs = {
	tab,
	tab2,
	tab3,
	tab4,
	tab5,
	tab6
}

return {
	ParameterGenerator = function()
		return BufferEncoder, HTTP, MessagePack, tabs
	end,

	Functions = {
		["BufferEncoder"] = function(Profiler, BufferEncoder, HTTP, MessagePack, tab) 
			local t = BufferEncoder.write(tab, nil, nil, nil, nil, Profiler)
			Profiler.Begin("Read")
			BufferEncoder.read(t)
			Profiler.End()
		end,

		["JSONEncode"] = function(Profiler, BufferEncoder, HTTP, MessagePack, tab) 
			Profiler.Begin("Write")
			local t = HTTP:JSONEncode(tab)
			Profiler.End()
			Profiler.Begin("Read")
			HTTP:JSONDecode(t)
			Profiler.End()
		end,
		
		["MessagePack"] = function(Profiler, BufferEncoder, HTTP, MessagePack, tab) 
			Profiler.Begin("Write")
			local t = MessagePack.encode(tab)
			Profiler.End()
			Profiler.Begin("Read")
			MessagePack.decode(t)
			Profiler.End()
		end,
	},
}

Installation

You can install it on Wally or from the github page, Links are at the top of the post.

Basic Example

API can be found in the github repository

local BufferEncoder = require(path_to_encoder)
local parts = {}
local Properties = {'Name', 'ClassName', 'CFrame', 'Color', 'Transparency', 'Material', 'Anchored'}
for _, part in workspace:GetDescendants() do 
   if part:IsA("BasePart") then
      local t = {}
      for _, prop in Properties do t[prop] = part[prop] end
      table.insert(parts, t)
   end
end

local buff = BufferEncoder.write(parts)
-- you could then save that buffer or send it in a remote or whatever you want

Credit

3 Likes