Hello Creators!
Today, we are releasing a new built-in Luau data type called ‘buffer’ under a Beta feature in Roblox Studio. This will provide a simple and faster way of working with compact binary data.
To enable this beta, head over to File, select Beta Features, and check Luau Buffer Type item in the list to opt-in.
Updated on 2023-12-13T00:00:00Z:
Updated on 2024-02-05T00:00:00Z:
The ‘buffer’ type represents a fixed-size block of memory and a new global ‘buffer’ library provides functions for the creation and manipulation of the data inside.
Like strings, buffers can contain arbitrary binary data; but unlike strings, data inside buffers can be freely mutated and the buffer library provides a way to interpret bytes inside the buffer as various small numeric types.
This extension was initially proposed in an RFC in our open-source Luau repository; we also received many requests to have a more low-level way to work with memory using fixed-size integer and floating-point types that are not just Luau numbers. Finding that buffers fit the use case very well, we refined the RFC and implemented it.
How do I use buffers?
To create a buffer, call buffer.create(size) to create a zero-initialized buffer with size bytes, or call buffer.fromstring(str) to convert an existing string with binary data into a buffer. From there on, you can use functions like buffer.readi16 or buffer.writef32 to work with data inside the buffer, using various common number representations; refer to buffer documentation for details.
For example, if you happen to have a lot of objects and you’d like to transmit their transforms over the network, you could do something like this:
-- each object takes 3*4 bytes for position and 3*2 bytes for orientation
local buf = buffer.create(#objects * (12 + 6))
for i, obj: Part in objects do
local offset = (i - 1) * (12 + 6) -- note: buffer offsets are 0-based
local pos, ori = obj.Position, obj.Orientation
-- 12 bytes for position, 4 bytes per component
buffer.writef32(buf, offset + 0, pos.X)
buffer.writef32(buf, offset + 4, pos.Y)
buffer.writef32(buf, offset + 8, pos.Z)
-- 6 bytes for orientation, 2 bytes per component
-- note: orientation components use degrees with -180..180 range
-- we're a little sloppy here and use a subset of the full i16 range
-- when reading this data, make sure to divide i16 by 100
buffer.writei16(buf, offset + 12, math.round(ori.X * 100))
buffer.writei16(buf, offset + 14, math.round(ori.Y * 100))
buffer.writei16(buf, offset + 16, math.round(ori.Z * 100))
end
Note: This is just one of many ways to represent translations and orientations.
What to watch out for
Buffers cannot currently be stored via attributes, but they can be passed using bindable/remote events, so you can simply pass the resulting buf object to BindableEvent.FireClient to replicate the data.
Keep in mind that just like a table, passing a buffer through Roblox APIs copies the data and doesn’t preserve the identity of that buffer reference.
Why are buffers useful?
Buffers are generally useful when working with binary data. Algorithms such as terrain serialization, audio and image processing, custom replication and compression, hashing as well as implementations of other virtual machines can benefit from a buffer type.
Compared to strings, buffers are easier and faster to extract data from (strings support string.unpack but this interface isn’t easy to use and isn’t very performant), and also support generating data whereas with strings it is difficult to produce packed data directly, requiring the use of temporary strings and table.concat.
Compared to number arrays, buffers are more compact in memory, take less space during storage or transmission, and can be faster to access in some cases. Large number arrays also can have a significant cost for garbage collection whereas buffers are fairly cheap in that regard.
Unlike number arrays, buffers are not automatically resizeable. When the buffer size is not known ahead of time, we recommend either estimating a reasonable upper bound and creating a slightly larger buffer, computing the precise size using another pass over the input data, or resizing the buffer adaptively using buffer.create/buffer.copy (this should be done with care to amortize the resize cost).
How do buffers interact with other engine features?
Buffers are supported by bindable and remote events and functions; importantly, buffers implement transparent on-the-wire compression. A sufficiently large buffer will be compressed before sending the data to the client/server and decompressed on the other side automatically, thereby reducing bandwidth; when buffers are used for replication, we recommend grouping smaller buffers into larger buffers, similar to the example above, to take maximum advantage of the compression.
Please keep in mind that network replication limits are smaller than the largest supported buffer size, and you should keep buffers that are sent over remote events under 50 MB.
We hope that buffers will be a good alternative for existing bit buffer libraries because of this: even though buffers require byte-based access instead of bit-based access, the compression can identify patterns where individual bytes can be compressed to occupy less than 8 bits each, as well as identify and collapse repetitive structures. Buffers should be significantly faster compared to bit buffer libraries as well.
Buffers also greatly benefit from native code generation; when –!native is used and the NCG preview is enabled, buffer accesses are fully optimized and inlined into the calling code, providing significant speedup compared to string access.
Please note that buffers currently are not supported by DataStore APIs; using buffer.tostring(buf) may not produce a string that can be stored in DataStores because it will often contain non-UTF8 bytes.
Looking for feedback
We plan to incorporate your feedback and make the feature officially available early next year. Please let us know if this new data type is useful for your experiences, if you’re missing any specific functionality, or if you find any bugs.
In the first release, we tried to focus on providing core functions, with the ability to build additional helpers on top of them. By gathering feedback we will get a better understanding of what to change/add in the future. We are also interested in code that is heavily reliant on buffers that we can use for our open-source benchmarks as this would help us ensure maximal performance for this feature.
Resources
Documentation for buffers can be found on the Roblox Creator Hub and also on Luau website.
Thanks to our open-source contributors to Luau on GitHub for the original proposal, @Dekkonot for early feedback, and to @WheretIB and @zeuxcg for refining the proposal and implementing the feature!
)
That’d be pretty sick if we could do that.