Mem-weaver: A strictly-typed buffer serialization and deserialization module

Resources Community Resources
,
#1

mem-weaver

mem-weaver is a low(ish)-level, strictly-typed module for creating, writing to, reading from, serializing, and deserializing buffers over the network.

Buffers can make events fire faster due to using less bandwidth.

Toolbox link

Supported Types

The buffers fully support the following types:

  • float (32 and 64)

  • integer

    Supported with 8, 16, and 32 bit integers.
    64 bit integers are supported, but they have to be encoded as a string,
    which takes up an extra byte (9 bytes instead of 8).

  • string

    Strings only support ASCII characters.

  • boolean

  • tables

    Tables only support simple datatypes, Vector3, Vector2, and CFrame are not supported within tables.

  • Vector3

  • Vector3int16

  • Vector2

  • Vector2int16

  • CFrame

  • Color3

Usage

The module revolves around the creation of buffer objects.
Buffer objects hold a set amount of memory at runtime.

Usage example:

-- CLIENT --
local alloc = require(mem-weaver)

local buf = alloc(8) -- Creates a buffer with 8 bytes of space.
	:write(3000) -- 2 bytes, 16 bit integer
	:write("four") -- 4 bytes, 4 character string
	:write(true) -- 1 byte, boolean
	:write(3) -- 1 byte, 8 bit integer

print(buf:read()) -- {[1] = 3000, [2] = "four", [3] = true, [4] = 3}
print(buf:read_at(3)) -- true

-- Serialize the buffer into a network-sendable format.
game.ReplicatedStorage.ExampleRemote:FireServer(buf:serialize())
buf:free() -- Remember to free your buffer!

-- SERVER --
local alloc = require(mem-weaver)

game.ReplicatedStorage.ExampleRemote.OnServerEvent:Connect(function(plr, received)
	-- Deserialize the received buffer into a buffer object.
	local buf = alloc.deserialize(received)
	print(buf:read()) -- {[1] = 3000, [2] = "four", [3] = true, [4] = 3}
	buf:free()
end)

Documentation

Documentation can be found on the github repository.

16 Likes
#2

Hello, nice module, but shouldnt the string “four” be 5 characters long? You have to include the null-terminating character as well.

#3

Are you using JSON to serialize tables? Very unoptimal, the way I did it for the schemaless BufferConverter was to list keys and values one after another and end the table with a null terminator, though a more reliable way is a length byte (as null terminators run into the issue of not being able to tell if a null terminator is a natural part of the data like 0, \n, nil or false, or the null terminator)

#4

I am using CBOR to serialize tables, which is a much more compact format than JSON, but i’m sure it would be better to use that method yes

#5

As far as I can tell, the null-terminating character is not included/required in buffer strings(at least in the way i use them here), because the buffer does not overflow on a size 4 buffer with a 4 character string.

#6

I mean, I think its very optional when it comes to ending your buffer with a null-terminating, but it’s recommended, haven’t checked the module’s source yet, but would be nice if you can end the buffers with null terminating characters