Bitbuf: A fast, bit-level buffer for binary serialization

Resources Community Resources
#1

The Bitbuf module provides a buffer that can read and write individual bits. It is fast enough for serialization to DataStore keys, and can even be used to store data in memory.

Features

Several common primitives are provided

  • Booleans.
  • Unsigned integers.
  • Signed integers.
  • Floats (32 bit).
  • Doubles (64 bit).
  • Byte arrays.
  • Unsigned fixed-point numbers (experimental).
  • Signed fixed-point numbers (experimental).

These can be composed futher to encode any desired data type.

Arbitrary bit-level reading, writing, and seeking

Bitbuf buffers read and write at the bit level, enabling efficient packing of binary data. Data does not have to be sequential either; buffers support seeking to any location at any time.

Alignment and padding

Buffers support padding with zeros, for indicating reserved bits. There are also methods to ensure that values have a particular bit alignment.

Converting between a buffer and a raw data string

A buffer can be converted to and from a binary string for further processing, such as converting to Base64.

Efficient memory footprint

A buffer in Bitbuf is represented in memory as an array of numbers, where each number represents 32-bits. This format creates a balance between memory consumption and read/write speed.

Blazing fast

The largest possible DataStore key (4,194,303 bytes) can be converted to a buffer in around 0.06 seconds. Converting the buffer back to a string takes around 0.44 seconds.

Example usage

-- Create buffer.
local buf = Bitbuf.new(32*6 + 8 + 8*(2^8-1))

-- Encode part data.
buf:WriteFloat(32, part.Position.X)
buf:WriteFloat(32, part.Position.Y)
buf:WriteFloat(32, part.Position.Z)
buf:WriteFloat(32, part.Size.X)
buf:WriteFloat(32, part.Size.Y)
buf:WriteFloat(32, part.Size.Z)
buf:WriteUint(8, #part.Name)
buf:WriteBytes(part.Name)

-- Move cursor to start.
buf:SetIndex(0)

-- Decode part data.
local copy = Instance.new("Part")
copy.Position = Vector3.new(
	buf:ReadFloat(32),
	buf:ReadFloat(32),
	buf:ReadFloat(32)
)
copy.Size = Vector3.new(
	buf:ReadFloat(32),
	buf:ReadFloat(32),
	buf:ReadFloat(32)
)
copy.Name = buf:ReadBytes(buf:ReadUint(8))

Benchmarks

I’ve created some benchmarks to compare the performance of Bitbuf with several similar modules. The following modules are compared:

Each compared benchmark is written to be equivalent.

Notes

  • ns/op is nanoseconds per operation. A value can be converted to seconds by dividing by 1000000000 (1e9).
  • A _N%d benchmark indicates an operation that is run N times on the same buffer.
  • A _L%d benchmark runs an operation using a string of length L.
  • An _Unaligned benchmark indicates values that are read or written without optimal bit alignment. For example, strings might be faster when aligned to 8 bits, or 32-bit unsigned integers might be faster when aligned to 32 bits. All such benchmarks first read or write 1 bit before the main operation.

Results

Benchmark results
Module Benchmark Iterations ns/op Delta
FastBitBuffer New 48026791 270 16.52x
Bitbuf New 32883393 375 11.90x
BitBuffer New 2958463 4461 1.00x
BitBufferDek New 1314034 9382 0.48x
-
FastBitBuffer WriteBool_N1 39798013 300 4.22x
Bitbuf WriteBool_N1 35637928 336 3.77x
BitBuffer WriteBool_N1 9407210 1267 1.00x
BitBufferDek WriteBool_N1 7031630 1713 0.74x
-
FastBitBuffer WriteBool_N10 5136256 2326 5.08x
Bitbuf WriteBool_N10 4593002 2623 4.50x
BitBuffer WriteBool_N10 1021654 11806 1.00x
BitBufferDek WriteBool_N10 792238 15289 0.77x
-
FastBitBuffer WriteBool_N100 546936 21913 5.33x
Bitbuf WriteBool_N100 468428 25774 4.53x
BitBuffer WriteBool_N100 102745 116878 1.00x
BitBufferDek WriteBool_N100 77812 150387 0.78x
-
FastBitBuffer WriteBool_N1000 54964 217652 5.40x
Bitbuf WriteBool_N1000 47336 252958 4.65x
BitBuffer WriteBool_N1000 10207 1175841 1.00x
BitBufferDek WriteBool_N1000 8325 1482912 0.79x
-
FastBitBuffer ReadBool_N1 38578716 311 1.03x
Bitbuf ReadBool_N1 38198766 314 1.02x
BitBuffer ReadBool_N1 37241586 321 1.00x
BitBufferDek ReadBool_N1 7956685 1286 0.25x
-
BitBuffer ReadBool_N10 5522054 2193 1.00x
Bitbuf ReadBool_N10 4866388 2467 0.89x
FastBitBuffer ReadBool_N10 4715926 2508 0.87x
BitBufferDek ReadBool_N10 1146598 10761 0.20x
-
BitBuffer ReadBool_N100 582112 20457 1.00x
Bitbuf ReadBool_N100 496106 24168 0.85x
FastBitBuffer ReadBool_N100 490100 24513 0.83x
BitBufferDek ReadBool_N100 102769 105880 0.19x
-
BitBuffer ReadBool_N1000 57274 204318 1.00x
Bitbuf ReadBool_N1000 49629 240673 0.85x
FastBitBuffer ReadBool_N1000 49057 245264 0.83x
BitBufferDek ReadBool_N1000 11910 1041730 0.20x
-
Bitbuf WriteUint_N1 24631899 487 8.20x
Bitbuf WriteUint_Unaligned_N1 12000000 885 4.51x
BitBufferDek WriteUint_N1 7119897 1584 2.52x
FastBitBuffer WriteUint_N1 7144600 1667 2.40x
FastBitBuffer WriteUint_Unaligned_N1 6347857 1894 2.11x
BitBuffer WriteUint_N1 4306402 2800 1.43x
BitBufferDek WriteUint_Unaligned_N1 2805890 3571 1.12x
BitBuffer WriteUint_Unaligned_N1 2974741 3993 1.00x
-
Bitbuf WriteUint_N10 3324967 3607 7.95x
Bitbuf WriteUint_Unaligned_N10 2292256 5254 5.46x
BitBufferDek WriteUint_N10 865202 14055 2.04x
FastBitBuffer WriteUint_N10 760796 15760 1.82x
FastBitBuffer WriteUint_Unaligned_N10 744691 16069 1.79x
BitBufferDek WriteUint_Unaligned_N10 685279 17415 1.65x
BitBuffer WriteUint_N10 438170 27346 1.05x
BitBuffer WriteUint_Unaligned_N10 419101 28689 1.00x
-
Bitbuf WriteUint_N100 343903 34567 7.93x
Bitbuf WriteUint_Unaligned_N100 246656 48788 5.62x
BitBufferDek WriteUint_N100 84908 137710 1.99x
BitBufferDek WriteUint_Unaligned_N100 75578 154776 1.77x
FastBitBuffer WriteUint_N100 76366 157167 1.74x
FastBitBuffer WriteUint_Unaligned_N100 75592 157544 1.74x
BitBuffer WriteUint_N100 43732 272591 1.01x
BitBuffer WriteUint_Unaligned_N100 43648 274076 1.00x
-
Bitbuf WriteUint_N1000 34794 345558 7.93x
Bitbuf WriteUint_Unaligned_N1000 24787 484305 5.66x
BitBufferDek WriteUint_N1000 8436 1366185 2.01x
BitBufferDek WriteUint_Unaligned_N1000 7770 1519964 1.80x
FastBitBuffer WriteUint_Unaligned_N1000 7582 1562217 1.76x
FastBitBuffer WriteUint_N1000 7650 1578232 1.74x
BitBuffer WriteUint_N1000 4338 2734800 1.00x
BitBuffer WriteUint_Unaligned_N1000 4375 2741707 1.00x
-
Bitbuf ReadUint_N1 30717668 390 4.81x
BitBufferDek ReadUint_N1 14964087 800 2.34x
Bitbuf ReadUint_Unaligned_N1 14829028 806 2.33x
BitBuffer ReadUint_N1 6864446 1658 1.13x
BitBuffer ReadUint_Unaligned_N1 6405574 1875 1.00x
BitBufferDek ReadUint_Unaligned_N1 5393529 2237 0.84x
FastBitBuffer ReadUint_N1 4065189 2931 0.64x
FastBitBuffer ReadUint_Unaligned_N1 3816073 3160 0.59x
-
Bitbuf ReadUint_N10 4132351 2912 5.73x
Bitbuf ReadUint_Unaligned_N10 2799390 4297 3.88x
BitBufferDek ReadUint_N10 1941956 6192 2.69x
BitBufferDek ReadUint_Unaligned_N10 1321808 9088 1.83x
BitBuffer ReadUint_Unaligned_N10 733623 16151 1.03x
BitBuffer ReadUint_N10 717631 16672 1.00x
FastBitBuffer ReadUint_N10 422180 28281 0.59x
FastBitBuffer ReadUint_Unaligned_N10 411134 28925 0.58x
-
Bitbuf ReadUint_N100 409962 29417 5.85x
Bitbuf ReadUint_Unaligned_N100 295281 40420 4.26x
BitBufferDek ReadUint_N100 200019 59856 2.88x
BitBufferDek ReadUint_Unaligned_N100 155163 77149 2.23x
BitBuffer ReadUint_Unaligned_N100 74724 165649 1.04x
BitBuffer ReadUint_N100 72108 172092 1.00x
FastBitBuffer ReadUint_N100 42361 283538 0.61x
FastBitBuffer ReadUint_Unaligned_N100 42280 290426 0.59x
-
Bitbuf ReadUint_N1000 41218 288867 5.93x
Bitbuf ReadUint_Unaligned_N1000 30038 399770 4.29x
BitBufferDek ReadUint_N1000 19810 596636 2.87x
BitBufferDek ReadUint_Unaligned_N1000 15768 767120 2.23x
BitBuffer ReadUint_N1000 7514 1598375 1.07x
BitBuffer ReadUint_Unaligned_N1000 7334 1714259 1.00x
FastBitBuffer ReadUint_Unaligned_N1000 4222 2842050 0.60x
FastBitBuffer ReadUint_N1000 4144 2848164 0.60x
-
Bitbuf WriteInt 20351904 589 5.09x
FastBitBuffer WriteInt 6975181 1723 1.74x
BitBuffer WriteInt 4066648 2996 1.00x
BitBufferDek WriteInt 3032449 4129 0.73x
-
Bitbuf ReadInt 22609216 531 3.29x
BitBuffer ReadInt 6836493 1747 1.00x
BitBufferDek ReadInt 4309941 2720 0.64x
FastBitBuffer ReadInt 4097336 2919 0.60x
-
BitBufferDek WriteFloat32 10393789 1175 4.81x
Bitbuf WriteFloat32 9388868 1280 4.41x
FastBitBuffer WriteFloat32 5130964 2337 2.42x
BitBuffer WriteFloat32 2107804 5650 1.00x
-
BitBufferDek ReadFloat32 16900268 718 3.13x
Bitbuf ReadFloat32 10553856 1125 2.00x
BitBuffer ReadFloat32 5353840 2246 1.00x
FastBitBuffer ReadFloat32 3665166 3272 0.69x
-
Bitbuf WriteFloat64 11695104 1005 7.37x
BitBufferDek WriteFloat64 6495218 1840 4.03x
FastBitBuffer WriteFloat64 3051297 3913 1.89x
BitBuffer WriteFloat64 1608398 7411 1.00x
-
BitBufferDek ReadFloat64 11067909 1078 3.51x
Bitbuf ReadFloat64 9895226 1258 3.01x
BitBuffer ReadFloat64 3179137 3784 1.00x
FastBitBuffer ReadFloat64 1949074 6171 0.61x
-
Bitbuf WriteString_Unaligned_L1 9820274 1228 5.75x
Bitbuf WriteString_L1 8337523 1435 4.92x
FastBitBuffer WriteString_L1 7391632 1616 4.37x
BitBufferDek WriteString_L1 6909262 1660 4.25x
FastBitBuffer WriteString_Unaligned_L1 6557484 1831 3.85x
BitBufferDek WriteString_Unaligned_L1 2747910 3706 1.90x
BitBuffer WriteString_L1 2025330 5891 1.20x
BitBuffer WriteString_Unaligned_L1 1704037 7057 1.00x
-
Bitbuf WriteString_L10 5870937 2042 10.81x
Bitbuf WriteString_Unaligned_L10 3659462 3261 6.77x
BitBufferDek WriteString_L10 3433855 3312 6.66x
BitBufferDek WriteString_Unaligned_L10 1890525 5649 3.91x
FastBitBuffer WriteString_L10 1865576 6415 3.44x
FastBitBuffer WriteString_Unaligned_L10 1805342 6651 3.32x
BitBuffer WriteString_L10 575308 20952 1.05x
BitBuffer WriteString_Unaligned_L10 543894 22073 1.00x
-
Bitbuf WriteString_L100 3715216 3225 52.91x
BitBufferDek WriteString_L100 614280 19156 8.91x
Bitbuf WriteString_Unaligned_L100 506884 23618 7.22x
BitBufferDek WriteString_Unaligned_L100 472945 25556 6.68x
FastBitBuffer WriteString_L100 214549 55758 3.06x
FastBitBuffer WriteString_Unaligned_L100 213247 55970 3.05x
BitBuffer WriteString_L100 70383 170477 1.00x
BitBuffer WriteString_Unaligned_L100 69415 170633 1.00x
-
Bitbuf WriteString_L1000 518601 23168 71.97x
BitBufferDek WriteString_L1000 67388 197681 8.43x
BitBufferDek WriteString_Unaligned_L1000 52758 224264 7.43x
Bitbuf WriteString_Unaligned_L1000 52962 226379 7.37x
FastBitBuffer WriteString_Unaligned_L1000 21802 547062 3.05x
FastBitBuffer WriteString_L1000 21850 547414 3.05x
BitBuffer WriteString_L1000 7240 1666899 1.00x
BitBuffer WriteString_Unaligned_L1000 7136 1667386 1.00x
-
Bitbuf WriteString_L10000 53881 222955 74.83x
BitBufferDek WriteString_L10000 6760 1958149 8.52x
BitBufferDek WriteString_Unaligned_L10000 5445 2174652 7.67x
Bitbuf WriteString_Unaligned_L10000 5320 2249519 7.42x
FastBitBuffer WriteString_Unaligned_L10000 2193 5473978 3.05x
FastBitBuffer WriteString_L10000 2190 5481253 3.04x
BitBuffer WriteString_Unaligned_L10000 716 16676358 1.00x
BitBuffer WriteString_L10000 711 16682863 1.00x
-
Bitbuf WriteString_L100000 5409 2221853 75.51x
BitBufferDek WriteString_L100000 679 19730016 8.50x
BitBufferDek WriteString_Unaligned_L100000 529 21785098 7.70x
Bitbuf WriteString_Unaligned_L100000 526 22501246 7.46x
FastBitBuffer WriteString_L100000 216 54746125 3.06x
FastBitBuffer WriteString_Unaligned_L100000 212 55063409 3.05x
BitBuffer WriteString_L100000 70 166730618 1.01x
BitBuffer WriteString_Unaligned_L100000 70 167781751 1.00x
-
Bitbuf ReadString_Unaligned_L1 8295573 1358 1.33x
Bitbuf ReadString_L1 8861671 1364 1.33x
BitBuffer ReadString_L1 7502770 1601 1.13x
BitBufferDek ReadString_L1 7891028 1606 1.13x
BitBuffer ReadString_Unaligned_L1 6543798 1809 1.00x
FastBitBuffer ReadString_L1 5436841 2202 0.82x
FastBitBuffer ReadString_Unaligned_L1 4925478 2405 0.75x
BitBufferDek ReadString_Unaligned_L1 3920146 3152 0.57x
-
BitBufferDek ReadString_L10 4977184 2550 2.69x
Bitbuf ReadString_L10 4754926 2566 2.68x
Bitbuf ReadString_Unaligned_L10 3666562 3335 2.06x
BitBufferDek ReadString_Unaligned_L10 2748584 4476 1.54x
BitBuffer ReadString_L10 1792286 6626 1.04x
BitBuffer ReadString_Unaligned_L10 1759278 6872 1.00x
FastBitBuffer ReadString_L10 1202845 10000 0.69x
FastBitBuffer ReadString_Unaligned_L10 1165476 10312 0.67x
-
Bitbuf ReadString_L100 2504614 4818 12.38x
BitBufferDek ReadString_L100 1116735 10821 5.51x
BitBufferDek ReadString_Unaligned_L100 746460 15900 3.75x
Bitbuf ReadString_Unaligned_L100 526231 22889 2.61x
BitBuffer ReadString_L100 208359 58073 1.03x
BitBuffer ReadString_Unaligned_L100 206402 59634 1.00x
FastBitBuffer ReadString_L100 120000 90400 0.66x
FastBitBuffer ReadString_Unaligned_L100 120000 92312 0.65x
-
Bitbuf ReadString_L1000 316057 37863 21.21x
BitBufferDek ReadString_L1000 120000 92269 8.70x
BitBufferDek ReadString_Unaligned_L1000 90292 131081 6.13x
Bitbuf ReadString_Unaligned_L1000 55062 214843 3.74x
BitBuffer ReadString_L1000 15158 799001 1.01x
BitBuffer ReadString_Unaligned_L1000 15016 803102 1.00x
FastBitBuffer ReadString_L1000 10606 1132602 0.71x
FastBitBuffer ReadString_Unaligned_L1000 10634 1139231 0.70x
-
Bitbuf ReadString_L10000 31795 378054 49.10x
BitBufferDek ReadString_L10000 12663 949049 19.56x
BitBufferDek ReadString_Unaligned_L10000 9073 1338823 13.86x
Bitbuf ReadString_Unaligned_L10000 5506 2146776 8.65x
BitBuffer ReadString_Unaligned_L10000 670 17857384 1.04x
BitBuffer ReadString_L10000 645 18562715 1.00x
FastBitBuffer ReadString_L10000 561 21386237 0.87x
FastBitBuffer ReadString_Unaligned_L10000 559 21567243 0.86x
-
Bitbuf ReadString_L100000 2910 4161276 411.63x
BitBufferDek ReadString_L100000 1200 9778316 175.17x
BitBufferDek ReadString_Unaligned_L100000 900 13619741 125.77x
Bitbuf ReadString_Unaligned_L100000 564 21810310 78.54x
FastBitBuffer ReadString_L100000 12 1696367059 1.01x
FastBitBuffer ReadString_Unaligned_L100000 12 1704559474 1.00x
BitBuffer ReadString_L100000 12 1711241613 1.00x
BitBuffer ReadString_Unaligned_L100000 12 1712904305 1.00x
-
Bitbuf FromString_L1 20179684 615 9.62x
FastBitBuffer FromString_L1 10698159 1288 4.60x
BitBuffer FromString_L1 2343772 5919 1.00x
BitBufferDek FromString_L1 1282221 9761 0.61x
-
Bitbuf FromString_L10 12000000 875 14.26x
FastBitBuffer FromString_L10 1898974 6120 2.04x
BitBufferDek FromString_L10 1259013 9976 1.25x
BitBuffer FromString_L10 996063 12476 1.00x
-
Bitbuf FromString_L100 5702288 2075 34.65x
BitBufferDek FromString_L100 1045387 12301 5.85x
FastBitBuffer FromString_L100 232663 51222 1.40x
BitBuffer FromString_L100 166959 71909 1.00x
-
Bitbuf FromString_L1000 791175 15189 43.39x
BitBufferDek FromString_L1000 358172 33783 19.51x
FastBitBuffer FromString_L1000 24706 486146 1.36x
BitBuffer FromString_L1000 18357 659002 1.00x
-
Bitbuf FromString_L10000 81381 149516 48.69x
BitBufferDek FromString_L10000 45464 266594 27.31x
FastBitBuffer FromString_L10000 2218 5406240 1.35x
BitBuffer FromString_L10000 1648 7279534 1.00x
-
Bitbuf FromString_L100000 7914 1517467 47.34x
BitBufferDek FromString_L100000 4203 2854792 25.16x
FastBitBuffer FromString_L100000 226 53543297 1.34x
BitBuffer FromString_L100000 166 71833300 1.00x
-
BitBufferDek String_L1 31865580 387 2.81x
BitBuffer String_L1 10920808 1088 1.00x
Bitbuf String_L1 9138802 1102 0.99x
FastBitBuffer String_L1 9364008 1283 0.85x
-
BitBufferDek String_L10 25135957 487 9.76x
Bitbuf String_L10 8307032 1487 3.20x
BitBuffer String_L10 2517630 4752 1.00x
FastBitBuffer String_L10 2473148 4842 0.98x
-
BitBufferDek String_L100 9336999 1072 43.02x
Bitbuf String_L100 2982949 3958 11.65x
FastBitBuffer String_L100 286357 41954 1.10x
BitBuffer String_L100 260026 46117 1.00x
-
BitBufferDek String_L1000 1666552 7202 91.78x
Bitbuf String_L1000 331710 36349 18.18x
FastBitBuffer String_L1000 19545 611083 1.08x
BitBuffer String_L1000 18320 660966 1.00x
-
BitBufferDek String_L10000 120000 89269 180.98x
Bitbuf String_L10000 32263 367863 43.92x
FastBitBuffer String_L10000 784 15200873 1.06x
BitBuffer String_L10000 746 16156057 1.00x
-
BitBufferDek String_L100000 11343 1058298 1561.41x
Bitbuf String_L100000 2982 4062258 406.78x
FastBitBuffer String_L100000 12 1642126591 1.01x
BitBuffer String_L100000 12 1652439320 1.00x

Observations

  • BitBuffer and FastBitBuffer store data in units of 1 bit, which gives them an advantage for the bool type.
  • Bitbuf stores data in units of 32 bits, in order to maximize utilization of the bit32 library.
  • BitBufferDek stores data in units of 8 bits, which enables very efficient conversion between buffers and strings.

Benchmark file

bench-bitbuf.rbxl (113.1 KB)

Roadmap

  • Streaming: convert between buffers and strings incrementally.
  • First-class Base85 support. Base85 is an encoding similar to Base64 that has an efficiency of 0.8 (versus Base64’s 0.75).
  • Arbitrary-size floats. Bitbuf only supports 32 and 64 bit floats, while other modules support varying sizes.
47 Likes
#2

I must be missing something, because this module converts to a string form that uses chars using the full 0-255 range which isn’t supported by datastores. I just tried to use this to serialize some data for a datastore, and I could only get the following error: “104: Cannot store string in data store. Data stores can only accept valid UTF-8 characters.”

Am I missing something? I used Bitbuf:String() to get the stringified version of the buffer.

#3

Buffer:String() converts to a Lua string, not a JSON string. You’ll have to further process the string in order to save it to DataStores safely.

Eventually, I want to add first-class support for Base85, as mentioned above. This will produce strings that can be safely saved to DataStores. In the meantime, I have a Base85 module you can use.

1 Like
#4

how can we use this?

character limit

#5

Wow, this is actually really cool- I did not expect to see this on the devforum. Could this be put on Wally potentially? (http://wally.run/

This is really useful for compressing lots of data to be sent over the wire- thanks for the resource! I was super scared of making this myself, not gonna lie.

Two small criticisms though:

  • OOP isn’t idiomatic- really more of a knitpick
  • Doesn’t use generic iteration

Overall, great work! I can really easily see myself using this.

#6

I’ve added an example to show how it’s used. There’s another included on the documentation page.

I’ve also updated the module to use type annotations.

Wally seems to have very little documentation, so I’m not inclined to use it. If you want an alternative, there’s BitBuffer by rstk, which appears to have been inspired by Bitbuf. Not only is it faster in most cases, it also follows all of the “best practices” like Rojo and Wally. The only uncertainty I have over it is the test coverage.