BufferConverter2 | Blazingly Fast Schema-Based Buffer Serialization

Resources Community Resources
, , ,
#21

0.08s to serialize 10k Instances TOTAL! :exploding_head:

Screenshot 2025-02-02 124235
Screenshot 2025-02-02 1242351919×385 63.8 KB

#22

Minor Update:

  • Fixed some incorrect types for the SchemaData

  • Made JSON serializing faster by using a lookup table and concat instead of interpolated strings

#23

Update:

  • Fixed CFrames not working correctly for size types other than f32 and f16

  • Fixed unions not interpreting structs correctly

2 Likes
#24

you could also use:
–!native and --!optimize 2

1 Like
#25

hey am I be able to use a function that calculates which number primitive is the best based on the value?

local Floor = math.floor
local function ChoosePrimitive(Value: number): "u8" | "i8" | "u16" | "i16" | "u32" | "i32" | "f16" | "f32" | "f64"
	local IsInteger = Floor(Value) == Value
	if IsInteger then
		if Value >= 0 and Value <= 255 then
			return "u8"
		elseif Value >= -128 and Value <= 127 then
			return "i8"
		elseif Value >= 0 and Value <= 65535 then
			return "u16"
		elseif Value >= 32768 and Value <= 32767 then
			return "i16"
		elseif Value >= 0 and Value <= 4294967295 then
			return "u32"
		elseif Value >= -2147483648 and Value <= 2147483647 then
			return "i32"
		end
		
		return "f64"
	else
		if Value >= -65504 and Value <= 65504 then
			return "f16"
		elseif Value >= -16777216 and Value <= 16777216 then
			return "f32"
		end
		
		return "f64"
	end
end

i have a table in which the values arent always the same primitive and i gotta save it to data stores, so i would like to save as much space as possible

1 Like
#26

I am unsure if you are able to because this module requires the schema to be consistent for serializing and deserializing, however if you can figure that out then yeah this’ll work

However, I am currently working on a way to serialize schemas so you can store them alongside the data itself (this hopefully should still be quite memory efficient as each schema isn’t too large, however I would still recommend just using 1 schema for every player’s data in the datastore)

2 Likes
#27

image
They addded buffer writebits and readbits, planning to use it anytime soon?

1 Like
#28

Wait, whaaaat :exploding_head: Yes this will definitely be incorporated into the module!

#29

Update:

  • Removed iterator loops from the extypeof() function and optimized array detection to not use any loops at all (This means unions are faster!)

WIP:

  • SchemaData.infer
  • SchemaData.numarray() using buffer.writebits()
  • Optimizing SchemaData.bitarray using buffer.writebits()
#30

What does optimize 2 even do again, I forgot tbh

1 Like
#31

Update:

  • Removed SchemaData.arraynterm() and SchemaData.mapnterm() (they were clogging up the module)
  • Used preallocated tables made with table.create() in the array() and map() SchemaData. This increased read times by like 2x (atleast from short measurements)

Docs will be updated soon Docs updated!

#32

makes your script run as it would in studio in roblox client, its pretty useful for benchmarks too

1 Like
#34

The Data module is getting quite big (nearing 1000 lines now :cry: ), any suggestions to cut down on the lines without making it too fragmentized would be appreciated :sweat_smile:

#35

I cant update the module and it seems its still the first version could you send us a new one?

1 Like
#36

Alright I’ll refresh the download link once I finish some fixes

#37

UPDATE + PATCH:

  • Added the static datatypes s_array and s_string, both of which require you to pass a fixed size. Their time complexities are slightly better than their dynamic alternatives and have less overhead due to not having to store the size as a header.

  • Fixed s_array not really working properly :sweat_smile:

  • Removed the first version of s_map from the latest version because it was benchmarked to be slower than the dynamic version, more likely another version is to come which will stay

  • Made all schemas return a new offset instead of a new offset from the offset ( I know, confusing right? )

I also renewed the download link :slight_smile:

1 Like
#38

hey i have an issue here, if i use cframe it can sometimes turn into nil

#39

i had look at it a bit more i think its a precision issue.

#40

What CFrame exactly? If you wish to not use the custom f16 for rotation you can specify it to be f32

#41

havent tried that but i will rewrite my game to do almost everything on client cuz it was stuttery