So I’ve been looking a bit (no pun intended) into 32-bit library which I got interested in.
I haven’t done a lot with bit operations or manipulating data on the binary level but what got me interested is the fact that it can be used for compression.
I figured that I sometimes don’t really need more than 8 to 16 bits of precision on some numbers or data so I was wondering if I can send less data through remote events or save space in datastores by combining multiple integers into one.
As far as I know Lua always uses 32 bits for numbers regardless of their size? Unless I’m wrong of course.
And I’m still a little bit (no pun intended x2) new to bit operations.
I’ve done it in JavaScript before but never in Lua, let alone Roblox Lua.
What are some essential things I must know about bit manipulation?
Are there ways to control numbers in a more low-level manner?
My most important question really is, let’s say I have a vector3 but for each number I only need 10 bits of precision and want to send it through a remote event as a single integer, how would I do that?
local v3 = Vector3.new(RANDOM_NUMBERS)
v3 *= 100
local x, y, z = math.round(v3.X), math.round(v3.Y), math.round(v3.Z)
-- Combine X, Y and Z together into single big integer..?
To do this with floats I assume I’d just have to multiply by 100 and round it to keep 2 decimals of precision and divide it later.
Any help is greatly appreciated!
To convert them into 1 big integer, could you not concatenate them…?
For example, if we have 89.2, 200.1 and 12.34, we multiply by the maximum power of 10 required to get them all into a whole number. In this case, the power of 10 is 2. This gives us the numbers 8920, 20010 and 1234.
Next, we need to add 0s to the front of the numbers so that they all have the same length. So, we have 08920, 20019 and 01234.
Finally, we tack on the power of 10, which is 2, to the front. Concatenating it all, we get 2089202001901234.
To decode, we do it all in reverse.
Remove the first digit (2)
Divide the whole length by 3 to get the invidivual length of each number
Split the number according to these lengths.
Remove the 0s at the front
Divide by the power of 10, which is the first digit (2)
Thank you!
This gives me somewhat of an idea but I struggle slightly to “visualize” it in my mind.
Do you have a small code example by any chance?
There are a few steps I can’t quite figure out why they’re necessary.
Finally, we tack on the power of 10, which is 2
Would you mind explaining this? Do I have to put the final integer to the power of 10?
Do I have to convert the numbers to strings to add the zeros at the beginning?
I recall that in JavaScript when I tried combining 8 bit integers into a 24 or 32 bit integer, I converted them to binary strings and simply combined the strings together but my concern is that it might be too slow for a game where everything needs to be real time.
I dunno how fast strings are in Lua but I might have to do a lot of string manipulation within a single frame or second in the more chaotic scenarios.
I was thinking of compressing things like shotgun blasts in a game since a shotgun might fire many pellets and compressing their directions/positions might decrease how much data I have to send through remote events.
A automatic shotgun could send a lot of data through a remote event all at once, especially if it fires about 10 - 20 pellets per shot.
Of course. The power of 10 is the number you need to multiply by in order to get a whole number. So in the case of a number like 56.94, we need to multiply by 100 to get the lowest whole number, which is 5694. 100 is 10 ^ 2, so the power of 10 is 2.
To add 0s at the start, yes, you’d need to convert to a string.
It’d be easier to concatenate it all as a string, then use tonumber to convert it back to a number, and finally send that number off.
I initially thought this would be useless, but I suppose for something like an automatic shotgun, that is a hell of a lot of data to send.
Though it doesn’t exactly look like we’re putting bits together but instead just non-binary numbers and strings.
But wouldn’t this become a problem if the final integer ends up becoming larger than what a 32-bit integer can be?
If we put together the numbers 99999, 99999 and 99999 again we get a pretty large number that is 999999999999999.
If I tonumber() this it might exceed the 32 bit limit.
And we may also have to deal with negative integers as well.
I suppose I could make all integers positive to make it easier but that would then pose the problem of needing to make them negative again later.
I would’ve suggested hashing for crunching down large numbers but due to hash collisions, hashes should be used only 1 way.
I’m afraid I’m not quite too sure anymore on this. Perhaps you can send multiple numbers that are under 32-bits? The 32-bit limit is really large iirc, something like -2 billion to 2 billion, and since they are Vector3 numbers, it could be unlikely that these numbers get unreasonably high.
So the largest possible number with 32 bits seems to be 4,294,967,296 which has 10 digits.
The largest possible number with 10 bits is 1024 which has 4 digits.
If we combined 1024 together 3 times as a string, the final number would be 102,410,241,024 which has more digits than the largest 32 bit number which is also why it has to be binary data instead of non-binary numbers and strings.
And I now just came to realize that 1024 is actually a pretty small number and if you use 2 decimals of precision then the number can’t go much beyond 10.24 which is kiiiind of a problem.
The largest possible number with 16 bits seems to be 65,536 which gives more room.
So maybe I can combine two 16 bit integers into one 32 bit integer which would be enough for a Vector2 and still give decent range if used to reduce the amount of data needed for let’s say… compressing shotgun pellet data.
I could reduce 2 Vector3s (6 numbers total) down to half (3 integers) with that which can be useful for the many shotgun pellet data I might have to send through a remote event.
local bit = bit32
local function packValues(x, y, z)
local packedValue = 0
packedValue = bit.bor(bit.lshift(x, 20), packedValue)
packedValue = bit.bor(bit.lshift(y, 10), packedValue)
packedValue = bit.bor(z, packedValue)
return packedValue
end
local function unpackValues(packedValue)
local x = bit.band(bit.rshift(packedValue, 20), 1023)
local y = bit.band(bit.rshift(packedValue, 10), 1023)
local z = bit.band(packedValue, 1023)
return x, y, z
end
local v3 = Vector3.new(1.234, 5.678, 9.012)
v3 *= 100
local x, y, z = math.round(v3.X), math.round(v3.Y), math.round(v3.Z)
local packed = packValues(x, y, z)
print("packed value:", packed)
local unpackedX, unpackedY, unpackedZ = unpackValues(packed)
print("uunpacked values:", unpackedX, unpackedY, unpackedZ)
local function packValues(x: number, y: number, z: number): number
local packedValue: number = 0
packedValue = bit32.bor(bit32.lshift(x, 20), packedValue)
packedValue = bit32.bor(bit32.lshift(y, 10), packedValue)
packedValue = bit32.bor(z, packedValue)
return packedValue
end
local function unpackValues(packedValue: number): (number, number, number)
local x = bit32.band(bit32.rshift(packedValue, 20), 1023)
local y = bit32.band(bit32.rshift(packedValue, 10), 1023)
local z = bit32.band(packedValue, 1023)
return x, y, z
end
local v3 = Vector3.new(1.234, 5.678, 9.012)
v3 *= 100
local x: number, y: number, z: number = math.round(v3.X), math.round(v3.Y), math.round(v3.Z)
local packed: number = packValues(x, y, z)
print("Packed value:", packed)
local unpackedX, unpackedY, unpackedZ = unpackValues(packed)
print("Unpacked values:", unpackedX, unpackedY, unpackedZ)
Also just tidied it up, used bit library in previous because I was doing it on a modified lua compiler i made.
Not necessarily for securing arguments or remote events.
I just don’t want to spam remote events with lots of data every time someone fires a shotgun in-game that might fire let’s say 10 - 20 pellets.
That’s at least 1 vector3 start position, 10 vector3 hit positions, plus another 10 instances I would have to send through a remote.
That’s at least 21 arguments for a single shotgun blast, and on an auto-shotgun it might happen more than 2 - 3 times a second which is a lot of data compared to a machine gun which only has to send a start and end position and whatever might be hit.
So I’m looking for a more efficient way to send data without clogging up remotes and hitting limits too soon.
Thank you for this btw, I’m currently looking a bit into how exactly this code works so I know what I’m doing with it.
I don’t do bit masking often so it takes some time for me to fully understand but it doesn’t seem very complicated or convoluted so I’ll mark it as a solution if this is the method I end up using.
But I’ll still be on the lookout for compressing data even further if possible and not too hard to do.
Note that Luau only has a single number type, a 64-bit IEEE754 double precision number (which can represent integers up to 2^53 exactly),
which means that, if you need to use 10-bit numbers, an integer could fit five of those numbers.
To package the numbers you simply do a base conversion, to base 1024 (2^10) in this case, it’s as simple as that. Although I am not sure of the performance, it is likely to have little impact because it is pure arithmetic.
local base1024Number = x + y*1024 + z*1024^2 + v*1024^3 + w*1024^4
make sure that x, y, z,… are 10 bits to avoid overflow errors.
Lua uses 64 bit numbers?
Wow, that’s actually very useful to know, I might have to do a change of plans here then.
Should give room for putting 4 16-bit numbers in a single 64-bit number.
Using this method I could lossy compress a Vector3 plus still have 16 bits of room for additional data if needed which is also quite useful.
In theory I could give every player/entity in a game a unique ID and whenever a pellet from a shotgun hits them I can use the first 3 x 16 bits for position data and use the last 16 bits for additonal data like the ID of the entity that was hit or the ID of the material, etc.
16 bits of free space left over after the positional data is enough room for extra bonus data which might be really useful.
Though, I would have to know how to mash numbers together and be able to split them apart or even read a single bit of a 64-bit value.
To be fair, arithmetic and manipulating bits isn’t something I’m an expert at so I’d have to learn a bit more about it.
How in Roblox Lua for example would I read the 12th bit of a number?
How would I get the last 4 bits of a number or the 8 bits in the middle and convert it to something I can read?
I assume it would involve some bit masking and shifting, I somewhat understand bit shifting but I’m not entirely sure how bit masks work and how to use it to read a specific sequence of bits in a big number.
What functions like bor() or band() do sometimes still seems a bit like magic to me (I just know things like 1 + 0 = 0 and 1 + 1 = 1, etc) and I don’t see a bit64 library in Roblox Luau.
Unfortunately lua, and therefore luau, is known to be a lousy language for bit manipulation.
The best we have is arithmetic, mainly with power of 2 bases (2, 4, 8, …, 256, 512, 1024, …). For example in these bases multiplying/dividing a number by its base means a displacement of one digit. With the % modulo operator you can loop through each digit of the base and do whatever you want with them.
Basically it is all arithmetic with binary numbers, which can be extended to any base power of 2.
Regarding your problem, remember that the mantissa is 52bits which is used to represent all integers between -9.2x10^18 to 9.2x10^18 approximately. On the other hand, not all decimal numbers can be represented, since they are infinite (between 0 and 1 there are infinite decimals). When an operation is done with decimal numbers, it is almost always a rounding to the nearest reprecentable value (this does not happen with integers). This in turn makes the exponent a bit unpredictable, so in this case it is not advisable to use the 11 bits of the exponent.
local n = 15671
for _ = 1,11 do
n = math.floor(n/2)
end
print("12th bit:", n%2)
local n = 1219449499797184
for _ = 1, 48 do
n = math.floor(n/2)
end
print("the four last bits:", n)
local n = 1219449499797184
for _ = 1, 18 do
n = math.floor(n/2)
end
print("the 8 bits in the middle:", n%256)
forget to mention that in these examples it is assumed that the 52 bits of the mantissa are being used
