Add mutable byte buffer type

Feature Requests Engine Features
#1

As a Roblox developer, it is currently too hard to represent mutable binary data with Luau efficiently.

If Roblox is able to address this issue, it would improve my development experience because I would be able to interface with binary formats in an efficient manner.

The current workarounds are very inefficient for this:

  • Tables can, at most, represent 64 bits per slot using expensive Vector3 packing.

  • Tables with or without packing severely bloat memory, as each array entry is subject to Luau value size and alignment.

  • Strings are immutable and can’t be used to efficiently construct binary data without exponential allocations.

  • Built in string.pack and string.unpack can’t cover more complex schemas on their own or formats which are edited mid-creation.

A new ByteBuffer (or similarly-named) object would solve this issue. It would represent a chunk of binary memory and provide an API for interfacing with it. Ideally, this API would allow for building, reading, and writing to the internal buffer. A particularly good example API that this could be derived from is the Java class by the same name.

While an API as extensive as the example might be good, it’s only really necessary to have methods for treating this block of data as a random access collections of bytes with potentially resizable backing. This could ideally be exposed as:

  • A constructor with predefined size and endianness.

  • A method for resizing the existing object while retaining data in bounds.

  • Methods for reading…

    • … a string given an offset and size.

    • … signed and unsigned integers of 8, 16, and 32 bits given an offset.

    • … floating point values of 32 and 64 bits given an offset.

  • Methods for writing…

    • … a string given an offset, size, and value.

    • … signed and unsigned integers of 8, 16, and 32 bits given an offset and value.

    • … floating point values of 32 and 64 bits given an offset and value.

As Luau can’t represent 64 bit integers in user code, the methods for these could be omitted or simply serve as a shortcut for writing what can be represented already without needing 2 calls to the 32 bit methods.

With this object, we ideally solve the use cases for binary format encoding and decoding, compression, and active/compact memory. This opens the door for plugin developers to work with file formats that might’ve been too large to represent with tables or to write to strings. It also allows for writing algorithms that deal with raw data often, such as compression or hashing. Web services that exchange data in packed formats could also benefit from this.

Alongside these example use cases, there’s a visible demand for this behavior as other developers have sought to implement versions of it as optimally as possible. These, of course, will suffer from the drawbacks listed at the start, which may be a deal breaker for lots of applications.

24 Likes