BufferConverter2 | Blazingly Fast Schema-Based Buffer Serialization

Are you tired of slow serialization? BufferConverter2 is a complete redesign of my original BufferConverter module—built for speed.

Unlike the previous version, BufferConverter2 uses a schema-based approach to dramatically improve serialization performance. Whether you’re handling networking, save data, or structured buffers, this module ensures your data is compact, efficient, and fast!


:star2: Why Use BufferConverter2?

:white_check_mark: Blazing-Fast Serialization – Schema-based for maximum efficiency.
:white_check_mark: Compact & Optimized Data – Reduce size and improve performance.
:white_check_mark: Supports Complex Data Types – Serialize CFrames, Vectors, Enums, Instances, and more!
:white_check_mark: Perfect for Networking – Send data faster with smaller payloads.
:white_check_mark: Lightweight & Easy to Use – Just define a schema and go!


:open_book: How It Works

BufferConverter2 relies on SchemaData, which defines how data should be serialized. This allows for precise control over how your data is packed and unpacked.

:small_blue_diamond: Supported Schema Types:

:scroll: Primitive & Structured Data

:heavy_check_mark: Strings with configurable length sizes
:heavy_check_mark: Numbers with custom precision (e.g., u32, f32)
:heavy_check_mark: Booleans
:heavy_check_mark: Structs (Nested objects with mixed data types)
:heavy_check_mark: Arrays (Key-value serialization)
:heavy_check_mark: Unions (one of two datatypes)

:building_construction: Roblox-Specific Types

:heavy_check_mark: CFrames (Highly optimized, only 20 bytes per CFrame! (minimum))
:heavy_check_mark: Vector2s & Vector3s
:heavy_check_mark: Enums
:heavy_check_mark: Instance References
:heavy_check_mark: Color3s

:hammer: Custom Types

:heavy_check_mark: Vector4
:heavy_check_mark: Class OOP objects

:mag_right: Full SchemaData Reference :point_down:

SchemaData

SchemaData are what this module uses to know how you want to serialize your data.
There are quite a few SchemaDatas, so let’s go over all of them!
(All SchemaData are stored within BufferConverter2.data)

Primitive

  1. SchemaData.empty

    • Indicates that the value here is nil.

  2. SchemaData.number(sizeType="f32": SizeType?)

    • This function creates a new number SchemaData, with the number being stored as size type sizeType.
      For example, if you passed “u32” as the sizeType, any numbers you serialize with this schema can only be integers between 0 and 65k.

  3. SchemaData.bool

    • Self explanatory.

  4. SchemaData.enum(enumType: Enum, itemSizeType="u8": SizeType?)

    • This function creates a new EnumItem SchemaData.
      You must pass what enumType the EnumItem will belong to, and you can also specify what size type you want to store the EnumItem value as.

  5. SchemaData.string(lenSizeType = "u32": SizeType?)

    • This function creates a new SchemaData with its length byte being of size lenSizeType.
      For example, if you passed “u8” as lenSizeType, the string you serialize using this SchemaData is 255 characters long max. If you passed “u16”, it would be 65k characters long max, etc.
      The reason why it’s done this way (instead of say, using null terminators) is to utilize buffer.writestring() and buffer.readstring(), both of which are very fast.

  6. SchemaData.bitarray

    • Reads and writes an array of 0s and 1s bitpacked into the bytes, making it so that each byte can store 8 of these bits (duh).
      Very useful if you, for example, wanna represent something in binary more easily or you want to send alot of flags

  7. SchemaData.class(classMetatable: any, struct: SchemaData)

    • Indicates that the value stored here uses a metatable from a specific class, and when read the metatable should be set to it.
      Also requires a struct SchemaData for serializing the class members.

Spatial

  1. SchemaData.cframe(posSizeType="f32": SizeType?, rotSizeType="f16": SizeType?)
    • This function creates a new CFrame SchemaData with the position components being stored with size type posSizeType, and the rotation (axis, angle) components being stored as rotSizeType.
      With the default values (f32 and f16) the CFrame only takes 20 bytes to store! (albeit with a little loss in precision.)

  2. SchemaData.vec3(componentSizeType="f32": SizeType?)

    • This function creates a new Vector3 SchemaData, and stores the XYZ components with size type componentSizeType.

  3. SchemaData.vec2(componentSizeType="f32": SizeType?)

    • Same as SchemaData.vec3(), just for Vector2s.

  4. SchemaData.vec4(componentSizeType="f32": SizeType?)

    • Same as SchemaData.vec3(), just using a custom Vector4 data format.

Composite

  1. SchemaData.union(first: SchemaData, second: SchemaData)

    • This SchemaData indicates that the value here can be of datatype first OR second.
      Useful if you need stuff like options, also can be nested!

  2. SchemaData.struct(struct: {[any]: SchemaData})

    • This function creates a new struct SchemaData.
      The way structs work is, basically every key in the struct has to be of 1 primitive datatype (for example, if you use numbers it should all be numbers, if you use strings it should all be strings), but the values can have differing datatypes.
      See the Examples section if you’re still confused.

  3. SchemaData.map(keySerde: SchemaData, valueSerde: SchemaData, lenSizeType="u32": SizeType?)

    • This function creates a new array SchemaData, using the keySerde for all keys, and the valueSerde for all values.
      This means that all of your keys must be the same datatype, same with values.
      This is unlike structs, which can have differing value datatypes.

  4. SchemaData.mapnterm(keySerde: SchemaData, valueSerde: SchemaData)

    • This is just SchemaData.map() but without a size limit, at the cost of not being able to use \0 characters, the number 0 and nil.

  5. SchemaData.array(valueSerde: SchemaData, lenSizeType="u16": SizeType?)

    • This function creates a new array SchemaData, reading and writing all values in the array using the valueSerde.
      Useful if you don’t want to store all of the keys (unlike in a map).

  6. SchemaData.arraynterm(valueSerde: SchemaData)

    • Same as SchemaData.array(), just without a size limit and with the same limitations as SchemaData.mapnterm().

Other

  1. SchemaData.instref

    • This SchemaData simply indicates that the value here should be a reference to an Instance (as a UniqueId.)

  2. SchemaData.color

    • Indicates that the value stored here should be a Color3.


:hammer_and_wrench: Examples

:small_blue_diamond: Sending Camera Configurations Over RemoteEvent

local Converter = require(path_to_bufferconverter2)
local data = Converter.data

local schema = data.struct {
    CFrame = data.cframe(),
    CamType = data.enum(Enum.CameraType)
}

local buf = Converter.serialize(schema, {
    CFrame = CFrame.identity,
    CamType = Enum.CameraType.Scriptable
})

game.ReplicatedStorage.RemoteEvent:FireAllClients(buf)

:small_blue_diamond: Using Every SchemaData Type

local Converter = require(path_to_bufferconverter2)
local data = Converter.data

local schema = data.struct {
    Hello = data.cframe("f32", "f16"),
    Array = data.array(data.number("u8"), data.enum(Enum.CameraType, "u8")),
    Skibidi = data.struct {
        [1] = data.string("u16"),
        [2] = data.instref
    },
    Hi = data.bool,
    Poopy = data.vec3("f16"),
    Vec2 = data.vec2("u8"),
}

local buf = Converter.serialize(schema, {
    Hello = CFrame.new(10, 20, 3),
    Array = {
        Enum.CameraType.Track,
        Enum.CameraType.Fixed
    },
    Skibidi = {
        "Hello World!",
        workspace
    },
    Hi = true,
    Poopy = Vector3.new(10, 2, 10),
    Vec2 = Vector2.new(12, 28, 3)
})

local result = Converter.deserialize(schema, buf)
print(result) -- Same table as before

:small_blue_diamond: Replicating Camera CFrame To The Server Every Frame

local Converter = require(path_to_bufferconverter2)
local data = Converter.data

local schema = data.cframe()

local camera = workspace.CurrentCamera
local remote = game:GetService("ReplicatedStorage").CameraReplicate

game:GetService("RunService").Heartbeat:Connect(function()
    local buf = Converter.serialize(schema, camera.CFrame)
    remote:FireServer(buf)
end)

:arrow_down_small: Download & Try It Now!

:inbox_tray: BufferConverter2.rbxm (7.0 KB)
:speech_balloon: Questions? Feedback? Let me know in the replies! :rocket:

Latest Module Update:

7 Likes

Update:

  • Tried fixing an issue related to UniqueId race conditions with instref SchemaData, if anyone still experiences it lmk
2 Likes

Update:

  • Added SchemaData.class(classMetatable: any, struct: SchemaData) and SchemaData.color
    The way that SchemaData.class(...) works is that it accepts a struct SchemaData and will set a metatable to it once read.
    This is mostly just for convenience :sweat_smile:


    Example usage:
      local Converter = require("path_to_bufferconverter2")
      local data = Converter.data
      
      -- Example Person class
      local Person = require("path_to_class") -- Assuming the module returns the metatable for the class
      
      local struct = data.struct {
      	_age = data.number("u8"), -- Lets be honest, noones living past 100
      	_name = data.string("u8")
      }
      
      local classSchema = data.class(Person, struct)
      
      local person = Person.new(10, "Jimmy") -- Pass age and name
      
      local age = person:GetAge() -- Lets assume Person has a method called GetAge()
      
      local buf = Converter.serialize(classSchema, person)
      
      print(Converter.deserialize(classSchema, buf):GetAge()) -- 10!
    
  • SchemaData.color does what it says, just indicates a Color3 value.
1 Like
  • I also added SchemaData.vec4 which is just a quaternion.
2 Likes

Update:

  • Redesigned the post for more engagement, I’m not too used to this so I’ll just see how it pans out
1 Like

What does this differ from Sera by loleris.

It has structs, maps, arrays (soon), and also I’m currently working on adding more datatypes such as unions, run-length encoded arrays, bit-packed boolean arrays, delta arrays and options.

Unless loleris added something and I didn’t notice :face_with_monocle:

1 Like

Update:

  • Added SchemaData.array() for storing just the values of an array (with order being built into how its ordered)
  • Added a SchemaData.arraynterm() and SchemaData.mapnterm(), both of which use null terminators instead of length indicator bytes (SchemaData.array() and SchemaData.map() both still use length indicator bytes)
  • Added SchemaData.boolarray which can store an array of booleans very compactly using bitpacking (useful for sending a bunch of flags)
  • Added SchemaData.empty to represent nil

WIP:

  • SchemaData.union() (semi-implemented)
  • SchemaData.numarray() (bit-packed number array)
  • SchemaData.cstring() (compressed string, slower to serialize and deserialize but takes up less space)
  • Reformatting the Data module to make it more readable
  • Optimizations for existing SchemaData

Apart from that, does it differ from sera performance-wise?

Honestly I can’t really do any comparisons for the composite types (map, struct, array, union) since Sera doesn’t have those, but for datatypes that do exist on both Sera and BufferConverter2 I can confirm the difference is negligible at best

1 Like

Update

  • The SchemaData.union() composite SchemaData works now!
    You can do this:

    local schema = data.union(data.color, data.cframe())
    	
    	local value1 = Converter.serialize(schema, CFrame.identity)
    	local value2 = Converter.serialize(schema, Color3.fromRGB(1, 2, 3))
    	
    	print(Converter.deserialize(schema, value1)) -- CFrame
    	print(Converter.deserialize(schema, value2)) -- Color3
    

    It also works for the SchemaData.empty SchemaData, meaning you can have optional values
    You can also nest unions inside one another, I wouldn’t really recommend it though

  • I also divided the Data module sorted from most primitive to least primitive, and grouped by their uses/components (vec3, vec2, vec4 and cframe are grouped (spatial), map, array, struct, union, etc are grouped (composite), bool, number, string, etc are grouped (primitive))

Minor Update:

  • Changed SchemaData.boolarray to SchemaData.bitarray

You should benchmark both, and display the results on your post, that’ll help, also, from what im seeing on the replication of camera, for the schema, you’re passing a serialized cframe with no arguments, does that mean if you dont provide the arguments then it’s gonna serialize it based on the integers-range? other than that, nice module :+1:

1 Like

No, if you dont provide any arguments it defaults to f32, though I should probably add some kinda inference for those who want it :stuck_out_tongue:

1 Like

Update:

  • Changed SchemaData.struct() and SchemaData.array() to use counter loops instead of iterator loops for better read and write times

  • Made the idLookup for SchemaData.instref more reliably retrieve the UniqueIds of instances by yielding for a frame until it exists

WIP:

  • SchemaData.infer which will automatically do the read and writing of the value (albeit at the cost of speed)
  • SchemaData.cstring() which is a compressed string with slower read and write times but smaller size
  • SchemaData.numarray which is a bit-packed number array (Useful for small numbers)
  • SchemaData.buf() which will copy the contents of the buffer into b

Update:

  • Made SchemaData.cframe() use quaternions instead of axis angle, saving 2 bytes in exchange for a single sqrt operation when reading (is this worth it guys? :cry: ) (but now CFrames are 18 bytes :smiling_imp: )

Update:

  • I made SchemaData.union() accept a tuple instead of only 2 SchemaData, so you can do stuff like this now!:

    Finally, a dictionary :slight_smile:

WIP:

  • SchemaData.infer

  • SchemaData.fullinst() Serialize an Instances full properties

Cancelled:

  • SchemaData.numarray() - Already done by SchemaData.bitarray, use that instead

Update:

  • Added a asJSON: boolean? argument to the Converter.serialize() function, which serializes the buffer like this!:

    What’s cool about this is that it can reduce your network recv when using RemoteEvents, because the buffers are compressed into base64! (I would implement a base64 to base255 for further compression but idk how :confused: )

  • Converter.deserialize() now also can accept the serialized JSON buffer as buf.

Update:

  • Fixed SchemaData.instref not properly being replicated for clients, now it is!

  • Fixed the json compression from interpreting a zbase64 buffer as a base64 buffer (just had to switch the orders around : P )

  • Reduced computations required for retrieving uniqueids (hopeful :broken_heart: )

  • Fixed some off-by-one errors i was encountering related to SchemaData.instref

Update:

  • Added SchemaData.fullinst()! You must pass the ClassName of the Instance and what properties you wish to be able to read and write in the form of {name: string, s: SchemaData}

    Soon I will be adding support for tags and attributes too