For my upcoming experience named “Project Mineria,” I strugged for months on data compression issues due to the complexity and amount of data that Voxel Data takes. However, I believe I have made the impossible possible. I am sharing this a few months after making this great leap in hopes that I still may be the first one to achieve this.
The limit for loading a specified “chunk” of terrain is 100x100x100 studs (The Region3 limit)
The limit for saving/loading Voxel Data from a datastore key with minimal Vector mapping compression is somewhere in the ballpark of 400x400x400 studs (about 4000 kb of data)
With my experience, which can be viewed in the video below, I have managed to compress my data to only minimal key points. With this, I have expanded the theoretical limit (for my experience type) to somewhere in the ballpark of 20000x20000x20000 studs (no, that is not a typo). I compressed the above 400x400x400 200 fold, lowering the 400x400x400 4000 kb store to only about 5-30 kb.
(Data is in bytes)
However, one must take into account that with compression comes decompression: a data scientist’s nightmare.
A failed attempt of mine a while back which resulted in lossy compression looked something along the lines of this:
I just started looking into data compression so I can’t say something that will help descompress, But I have some questions, would really apreciate if u could answer them.
“What are you actually compressing? Position, CFrame, Rotation?” “What did u use to compress it? did u make your own compression system using luau based on algorithms like Huffman or LZ4?” Whats the actual process needed to compress and descompress? for an example how would I compress a vector3 (0,6,5)?
Sorry about all the questions, btw from what I understand it seems u’re storing all of this meshes positions and rotation after compressing it which sincerely looks really good.
I am compressing Voxel Data, which comes in a table format Ex:// VoxelNums[x][y][z] or VoxelMaterials[x][y][z].
I used BitBuffer, on top of my own algorithm that basically takes only the necessary information in which then I can rebuild it. For my case, it is pretty self explanatory: I only have two places… Inside the cave, and outside of the cave. Apart from that, all data is useless. However, the :WriteVoxel method REQUIRES you to fill all the 50x50x50 table, which means the only fragments I saved are the borders between the outside of the cave, and inside of the cave.
The process it pretty self explanatory, especially when using bit buffer, as it automatically does Vector compression for you. apart from that, it is about making a module that grabs all the table indexes and encodes them respectfully. I might in the future open source the methods I created for that, however a friend helped me create it (the BitBuffer handler), and I would have to get permission from him first.
For your last statement, I am actually just using roblox terrain! I am using that over mesh deformation, etc… because ROBLOX has some really cool features that makes terrain not lag as much as meshes do.
