BufferConverter2 | Blazingly Fast Schema-Based Buffer Serialization

[athar_adv (athar_adventure)]

Update:

• Replaced counter loops with iterator loops where I could because of this optimization Luau makes:
  
  
• Made SchemaData.fullinst() accept a dictionary of SchemaData so that less table indexing is done

[1Xayd1 (gh_xayd)]

The cframe is still bugging out lemme record it real quick>

Edit:
Try constantly sending random cframes and compressing + decompressing them thorugh the module. You will see the issue quickly.

[1Xayd1 (gh_xayd)]

I am unable to record since i am currently on laptop but just try what i said and print out stuffs @athar_adv

[athar_adv (athar_adventure)]

It looks fine to me

[1Xayd1 (gh_xayd)]

erm, make it more frequient like 1/60s i am using module for head movement replication

[athar_adv (athar_adventure)]

I myself use this module for character replication, so I would know if it was buggy. Are you sure you don’t have an older version and/or it’s a problem with your replication?

[1Xayd1 (gh_xayd)]

no its not a problem with my replication; it literally happened when i was passing cframe to an object

[athar_adv (athar_adventure)]

Well I dunno, the module works fine. 1 thing I would check is if you are setting the cframe on the server and the client at the same time, as that would cause stuttering. (My character replication doesn’t set the cframe on the server for that reason :V)

[1Xayd1 (gh_xayd)]

Its not even stuttering the literal cframe goes to nil; stuttering is other issue

[athar_adv (athar_adventure)]

I wonder, can you send a test place for me to replicate and fix the issue?

[athar_adv (athar_adventure)]

Update:

• Added special cases for CFrames which are axis aligned, making it so that they dont store rotation. This is determined by the new 3rd parameter of SchemaData.cframe() which if set to true will check for these special cases.
  
  
  The only caveat is that CFrames with the special cases turned on will have 1 byte of overhead to indicate wether or not the CFrame is special, however the benefits far outweigh as you can now serialize a CFrame in 8 bytes.
  
  
  Soon I will add a ton more special cases and increase the special cframe overhead to 2 bytes so that for most common cframes itll only take 9 bytes to serialize (half of the original 18)

Coming Soon:

• Preventing too many unique id assignments in a single frame

• SchemaData.normalNum(base: number, sizeType: SizeType?) which will serialize numbers relative to base, meaning less space

• SchemaData.obj() which can serialize objects using compressed object format (however at the cost of only being able to store primitive types)

[AB1BZ]

Would implementing this into my frameworks remote module be to any benefit?

[athar_adv (athar_adventure)]

Sure why not, it has quite a few useful types like unions, static types, full instances, instance references, bit arrays, etc that other buffer serdes don’t have

[AB1BZ]

I was speaking more or less benefits towards optimization and networking.

[athar_adv (athar_adventure)]

Like serialization times? It is quite fast, I can serialize 1k+ instances in less than 0.009s ast time I measured, even faster for the more primitive datatypes

[athar_adv (athar_adventure)]

Package Version 104 - 105:

• Fixed an offset miscalculation error, turns out the real size of the special CFrames are 7 bytes!
  
  

[KingBlueDash (A5TR0_N0VALIT3)]

I’ve never messed around with buffers, so sorry if this is a seemingly obvious question, but how would i serialize instances with this? and can i serialize all instances? as this module would be great help for my upcoming plugin, and would make the instance serializing much faster.

[athar_adv (athar_adventure)]

Hi, yes you can serialize all instances fully if they can be reconstructed with Instance.new(). If you wish to just serialize instance references you can do that too with SchemaData.instref, heres an example of fully serializing a part vs just a reference of a part in workspace:

-- Whatever path you have, this is mine
local Converter = require(game.ReplicatedStorage.Modules.Utility.Converter)
local data = Converter.data

local full = data.fullinst("Part", {
	["CFrame"] = data.cframe(),
	["Size"] = data.vec3(),
	["Color"] = data.color,
	["Name"] = data.string("u8")
})

local ref = data.instref

do
	print(workspace.Example.CFrame) -- 10, 20, 32, ...
	
	-- Serializes Example as a full instance with the properties specified in the schema, much bigger and slower but fully reconstructs the entire instance upon deserializing
	local buf, _, size = Converter.serialize(full, workspace.Example)
	print(size) -- 40
	local part = Converter.deserialize(full, buf)
	
	print(part.Name) -- Example
	print(part.CFrame) -- 10, 20, 32, ...
        print(part.Parent) -- nil
end

do
	-- Serializes Example as a reference, much smaller
	local buf, _, size = Converter.serialize(ref, workspace.Example)
	print(size) -- 5
	local part = Converter.deserialize(ref, buf)
	
	print(part.Parent) -- Workspace
	print(part.Name) -- Example
end

[athar_adv (athar_adventure)]

Package Version 106:

• Added a UniqueIdLength attribute to the module for you to specify the length of all unique ids assigned to Instances for the purpose of instance reference serialization
  

  
  
  This reduces SchemaData.instref overhead by 1 byte, which is massive if you have alot of instance references (For example, 400 bytes for an array of 400 instance references instead of 500 if the unique id length is 4)
  
  If your game has many instances, I would recommend increasing this limit to prevent collisions

• Made it so that if more than 100 unique id assignments are made in less than 1/30s, the next assignments are deferred to the end of the frame. (Hopefully this should work), if you wish to modify the thresholds simply access Converter -> UniqueIds -> Server

[athar_adv (athar_adventure)]

Package Version 108:

• Added the long awaited SchemaData.infer(argsList: {[string]: any}) which will infer datatypes preferring ones present within argsList.
  If the schema is a constructor function, the arguments passed to it will be determined by its entry in argsList
  
  Example:

  local schema = data.infer {
  	["number"] = "u8",
  }
  
  local buf, _, size = Converter.serialize(schema, 103)
  
  -- 1 byte of tag overhead
  print(size) -- 2
  print(Converter.deserialize(schema, buf)) -- 103, true, 2
  

  Compared to:

  local schema = data.infer {
  	["number"] = "u32",
  }
  
  local buf, _, size = Converter.serialize(schema, 103)
  
  -- 1 byte of tag overhead
  print(size) -- 5
  print(Converter.deserialize(schema, buf)) -- 103, true, 5
  

• Added SchemaData.localnum(base: number, sizeType: SizeType?) which is useful when your numbers are big but you know what range your numbers will be in, so this will use less space. Instead of serializing the full number, it serializes its difference from base

• Added SchemaData.s_bitarray(bitLength: number) which has a constant bit length. Useful if you don’t want to allocate much space for storing flags