BufferConverter2 | Blazingly fast schema-based buffer serde

Resources Community Resources
, ,
#1

This module is a complete redesign of my original BufferConverter module, this time utilizing a schema structure to speed up serialization times (serializing using BufferConverter was very slow, hence why I made this.)

Documentation:

SchemaData

SchemaData are what this module uses to know how you want to serialize your data.

There are quite a few SchemaDatas, so let’s go over all of them!

  1. SchemaData.string(lenSizeType = "u32": SizeType?)

    • This function creates a new SchemaData with its length byte being of size lenSizeType.
      For example, if you passed “u8” as lenSizeType, the string you serialize using this SchemaData is 255 characters long max.
      If you passed “u16”, it would be 65k characters long max, etc.
      The reason why it’s done this way (instead of say, using null terminators) is to utilize buffer.writestring() and buffer.readstring(), both of which are very fast.

  2. SchemaData.cframe(posSizeType="f32": SizeType?, rotSizeType="f16": SizeType?)

    • This function creates a new CFrame SchemaData with the position components being stored with size type posSizeType, and the rotation (axis, angle) components being stored as rotSizeType.
      With the default values (f32 and f16) the CFrame only takes 20 bytes to store! (albeit with a little loss in precision.)

  3. SchemaData.vec3(componentSizeType="f32": SizeType?)

    • This function creates a new Vector3 SchemaData, and stores the XYZ components with size type componentSizeType.

  4. SchemaData.vec2(componentSizeType="f32": SizeType?)

    • Same as SchemaData.vec3(), just for Vector2s.

  5. SchemaData.number(sizeType="f32": SizeType?)

    • This function creates a new number SchemaData, with the number being stored as size type sizeType.
      For example, if you passed “u32” as the sizeType, any numbers you serialize with this schema can only be integers between 0 and 65k.

  6. SchemaData.enum(enumType: Enum, itemSizeType="u8": SizeType?)

    • This function creates a new EnumItem SchemaData.
      You must pass what enumType the EnumItem will belong to, and you can also specify what size type you want to store the EnumItem value as.

  7. SchemaData.struct(struct: {[any]: SchemaData})

    • This function creates a new struct SchemaData.
      The way structs work is, basically every key in the struct has to be of 1 primitive datatype (for example, if you use numbers it should all be numbers, if you use strings it should all be strings), but the values can have differing datatypes.
      See the Examples section if you’re still confused.

  8. SchemaData.array(keySerde: SchemaData, valueSerde: SchemaData)

    • This function creates a new array SchemaData, using the keySerde for all keys, and the valueSerde for all values.
      This means that all of your keys must be the same datatype, same with values.
      This is unlike structs, which can have differing value datatypes.

  9. SchemaData.instref

    • This SchemaData simply indicates that the value here should be a reference to an Instance (as a UniqueId.)

  10. SchemaData.bool

    • Self explanatory.

All of these SchemaDatas are stored within BufferConverter2.data.

Examples

A script that serializes camera configurations and sends it over a remote event to all clients:

local Converter = require(path_to_bufferconverter2)
local data = Converter.data

local schema = data.struct {
	CFrame = data.cframe(),
	CamType = data.enum(Enum.CameraType)
}

local buf = Converter.serialize(schema, {
	CFrame = CFrame.identity,
	CamType = Enum.CameraType.Scriptable
})

game.ReplicatedStorage.RemoteEvent:FireAllClients(buf)

A random script that serializes using every single SchemaData:

local Converter = require(path_to_bufferconverter2)
local data = Converter.data

local schema = data.struct {
	Hello = data.cframe("f32", "f16"),
	Array = data.array(data.number("u8"), data.enum(Enum.CameraType, "u8")),
	Skibidi = data.struct {
		[1] = data.string("u16"),
		[2] = data.instref
	},
	Hi = data.bool,
	Poopy = data.vec3("f16"),
	Vec2 = data.vec2("u8"),
}

local buf = Converter.serialize(schema, {
	Hello = CFrame.new(10, 20, 3),
	Array = {
		Enum.CameraType.Track,
		Enum.CameraType.Fixed
	},
	Skibidi = {
		"Hello World!",
		workspace
	},
	Hi = true,
	Poopy = Vector3.new(10, 2, 10),
	Vec2 = Vector2.new(12, 28, 3)
})

local result = Converter.deserialize(schema, buf)
print(result) -- Same table as before

Download BufferConverter2 here! →
BufferConverter2.rbxm (6.7 KB)

If you’re confused about something or have some questions, ask away down in the replies.

2 Likes
#2

Update:

  • Tried fixing an issue related to UniqueId race conditions with instref SchemaData, if anyone still experiences it lmk
#3

Update:

  • Added SchemaData.class(classMetatable: any, struct: SchemaData) and SchemaData.color
    The way that SchemaData.class(...) works is that it accepts a struct SchemaData and will set a metatable to it once read.
    This is mostly just for convenience :sweat_smile:


    Example usage:
      local Converter = require("path_to_bufferconverter2")
  local data = Converter.data
  
  -- Example Person class
  local Person = require("path_to_class") -- Assuming the module returns the metatable for the class
  
  local struct = data.struct {
  	_age = data.number("u8"), -- Lets be honest, noones living past 100
  	_name = data.string("u8")
  }
  
  local classSchema = data.class(Person, struct)
  
  local person = Person.new(10, "Jimmy") -- Pass age and name
  
  local age = person:GetAge() -- Lets assume Person has a method called GetAge()
  
  local buf = Converter.serialize(classSchema, person)
  
  print(Converter.deserialize(classSchema, buf):GetAge()) -- 10!
  • SchemaData.color does what it says, just indicates a Color3 value.
#4
  • I also added SchemaData.vec4 which is just a quaternion.