I’m currently working on a game where I need to send a heavy amount of information to the client and the best way I could think about sending a large amount of data is by bit packing my information into a 64 bit number.
The issue with this is that luau only has a bit32 library which creates many problems since I need to do a left shift on the bits in order to store more information. Is there a way for me to be able to preform bitwise operations on a 64 bit number using the bit32 library easily?
Here’s a snippet of my code for an example of what I’m trying to achieve:
-- Packs enemy info into 22 bits
local function pack_bits(enemy_id, enemy_info)
local result = 0
result += enemy_id
local next_grid_pos = math.floor(enemy_info.X + (enemy_info.Y - 1) * GRID_LENGTH)
result += bit32.lshift(next_grid_pos, 10)
return result -- returns a 22 bit number
end
local function pack_enemies(enemy_array)
local array_of_int64 = {}
local current_packer = 0
local bits_added = 0
for enemy_id, enemy_info in next, enemy_array do
local enemy_bits = pack_bits(enemy_id, enemy_info) -- create a 22 bit number packing the enemy info in it
current_packer += bit32.lshift(enemy_bits, bits_added) -- The issue lies here: roblox numbers should support up to 64 bits, but I can only preform operations up to 32 bits
bits_added += 22 -- enemy info size is 22 bits
if bits_added >= 64 then -- int size is 64 bits long
table.insert(array_of_int64, current_packer)
local remaining_bits = bits_added - 64
bits_added = remaining_bits
current_packer = remaining_bits > 0 and bit32.extract(enemy_bits, 22 - remaining_bits, remaining_bits) or 0
end
end
local array_of_vectors = {}
for vector_index = 1, math.ceil(#array_of_int64/3) do
local start_index = (vector_index - 1) * 3
local vector_args = {}
for i = 1, 3 do
local int64_index = start_index + i
table.insert(vector_args, array_of_int64[int64_index] or 0)
end
table.insert(array_of_vectors, Vector3.new(table.unpack(vector_args)))
end
return array_of_vectors
end
Another issue I’m having is that I’m trying to pack the packed bits inside a table of vector3’s in order to optimize the data being sent, however I’ve noticed that when the numbers start to become larger, there seems to be some floating point errors occurring:
If anyone has ever worked on optimizing large amounts of data being sent from server to client and knows of some hacky methods that might work for me, I’d love to know what any of y’all came up with!
The only thing that comes to mind is packing into bytes and converting those into UTF8 characters for a string using the utf8 global’s char and codepoint functions.
I’d love to write you an example, but I’m on my phone right now and someone considering bitpacking would probably be able to figure out what I mean. Best of luck!
Upon reading your solution I’ve done some investigation and learned that the utf8.char can actually store up to (2^20 + 2^16 - 1) bits, however, to make things simpler, I’ll most likely use 2^19 bits.
while true do
game.ReplicatedStorage.ReplicateEnemy:FireAllClients(utf8.char(table.unpack(table.create(1112, 1048575))))
wait(0.1)
end
I decided to do some stress testing to represent the actual data being sent (assuming there’s 960 enemies with 22 bits of information packed). My theoretical calculation is saying that the outgoing should be about 211 kb/s. However, from testing I noticed that my incoming is reading at 44 kb/s which is a SIGNIFICANT reduction from the theoretical calculation. Thank you for your advice on utf8 characters!
A new RFC was merged into the Luau repository and it adds a new buffer type, with a library still to come. This may be of interest to you as it could possibly speed the encoding and decoding of data up and maybe squeeze more throughput depending on Roblox’s network bandwidth limits.
Luau’s versions correspond with Roblox releases, so you should probably expect to see this and the companion library in the coming month (at least I hope).
I was reading the rfc that you sent me and noticed that it said something about relying on the “bit32 library semantics”. Do you know if this means the size of the buffers are going to be limited to only 32 bits in size? I only wonder this because the buffer object seems to have a method called tostring and I was concerned on how that’ll hold up considering a single utf8 character can only store around 20 bits of data
A buffer value, according to the RFC should at minimum, have a maximum length equal to that of the string type provided by the implementation of the Luau language specification.
So, if the official implementation used in Roblox limits strings to 200,000 bytes, you will have no less than 1,600,000 bits.
My bigger concern is how they plan to optimize this type of data when being sent through a remote event. If it’s anything similar to a string my guess is I shouldn’t have to worry. Hopefully the buffer class consumes less data when being passed through remotes
