Bindings for bitwise operations

As a Roblox developer, it is currently slow and clunky to perform bitwise operations on numbers. This complicates what should be simple or quick tasks and makes them impractical or slow. We should have bindings (in the form of a library of C-implemented functions) that perform these operations quickly, akin to the bit32 library from Lua 5.2+.

Bitwise operations are useful for, among other things, compression, base 64 conversions, storing information in bit fields, and reading binary files. Each of these are achievable without bitwise operations, but they’re slower (sometimes by orders of magnitudes) and more convoluted without them.

Compression

Compression is no easy task without bitwise operations. There are a myriad of Roblox APIs that have limitations on their size. When large amounts of information has to be used with these APIs, those size limits can be reached easily without some compression. There are some truly incredible compression algorithms in the world, but nearly none of them are practical in Roblox at the moment simply due to the time required. It’s not worth compressing data if it takes a long time to do it. Compression becomes much easier with bitwise operations.

Base 64 Conversions

Another much more common problem is converting strings to base 64. Base 64 encoding is used to make sure a string of information won’t encounter problems when saved or sent over a web request. Roblox supports UTF-8 in strings, which is good, but lends itself to making Http requests that might encounter issues just because they have unicode characters in them. Base 64 encoding requires bitwise operations to do in a performant manner, as it’s based off of bits. Currently, pure Lua implementations of base 64 conversions are annoying to make. Bitwise would make these implementations dramatically shorter (see the Lua-Users wiki page on it).

Bit Fields

Bit fields are a way of storing multiple bools in one byte of data. While this could technically fall under compression, this is a more practical use case. Roblox currently gives us a few ways to store data. Most if not all of these use JSON to convert data to strings. This can be problematic when you’re trying to store a lot of information because JSON strings get long fast. Arrays of bools could be stored in one number using bitwise operations, which would reduce the overall size of stored data. Roblox actually does this with data types like Faces and Axes when they’re stored, where multiple (6 and 3 respectively) true or false values are stored in one byte.

Binary Files

At first, binary files may not seem like they’re important in Roblox. Almost everything people are going to be using is written in plain text, after all. However, there is actually a way to import local files using StudioService, and it supports binary files. In fact, the name of the method to get the contents of a returned file is GetBinaryContents. It seems only fitting then that we would be able to actually read the contents of binary files. This is most useful for things like Roblox’s rbmx or mesh file formats, but it also includes things like images or other mesh files. The current methods for translating this binary data are slow and impractical. With some sort of binding for bitwise operators, this would be sped up dramatically.

As a power user and a developer, my life would be made better by not having to use a slow bit library for things like bit fields or base 64 conversions. There’s at least one 3rd party company that would benefit from bitwise operators being built into Roblox (GameAnalytics) as well.

To be completely honest, it’s wild that we don’t have these yet. While you could argue that Roblox would be better served just implementing individual use cases like base 64 conversions or compression, I would much rather have the bitwise operations themselves since they solve a myriad of problems instead of just one.

17 Likes

Maybe a slight niche case but, bitwise operators or just a library would greatly enhance the performance in some other niche cases such as Lua interpreters during their deserialization phase and allow for easier interpreting of things like Lua 5.3 within the Roblox environment.

6 Likes

I experienced 2 second hangs on the server with the official GameAnayltics module and no extra metrics. A lower level bitwise feature would make a developer wiki recommended system viable to use.

Having an explicit way to pool operations to combat bridge latency would be helpful if that’s not how it would implicitly work/how that sort of thing works already.

3 Likes

And most importantly, they need to be efficient.

1 Like

Sorry for the 20 day delay; I saw your response, meant to write a reply, and it slipped my mind.

My main problem is that I haven’t actually tried to write anything that would need bitwise in Roblox because after some initial benchmarking I decided it wasn’t worth it. What I do know is that the GameAnalytics module that uses bitwise for base-64 conversions and hashing (SHA-256, HMAC) and it’s the source of a lot of the overhead of the module. Since it’s not practical for Roblox to implement every type of hashing, bitwise operators seems like a more broad solution to the problem, and that seems like it would be the case with compression as well.

I do have some very specific uses for bitwise operators (I’d like to read image files, for example), though I chose to go with a more general request since in my experience threads that open with niche reasons for wanting a feature are doomed to failure since they’re the first thing an engineer reads. My hope was that something that covered more ground would be more useful than something that was as specific as “I wish to read the header of an rbmx file”.

1 Like

Would definitely support something like this to allow us to compress, format and process our data in more efficient ways