I Love Compression pt 2: Huge Voxel Terrain Serialization

So yesterday and today I’ve created a script that serializes HUGE sections of voxel terrain and exports them as a string, and allows you to reconstruct the terrain using that string somewhere else (in my case, moving terrain from an offline save to a team create)

The serialization basically saves the occupancy and material values of each voxel as a single character. In this case, I’ve designated the 1’s place of string.byte as the (rounded) occupancy and the 10’s place as the material. This way of doing it does not support all terrain materials at once, however in my case I had just the right amount of materials that it worked fine with no loss.

My example and test case:
I generated a huge cone and an island, and then carved it up and painted on materials to make it a huge mountain. The entire space is ~800x800x800 studs which is about ~8,000,000 voxels of data. The code I used to read the terrain and convert it to a string took 330 seconds to finish, it encoded a string that was 7,749,809 characters long which I then compressed with @1waffle1’s LZW module down to just 115,999 characters (1.5% of the original length). For reference, this is ~44% of a datastore cell, although I am not using this with datastores.

The code I used to take the serialized data and write it to the terrain took 53 seconds to finish, and is nearly identical to the original terrain

Before:


After:

After[2] (In a new place in case you thought I was lying about the above screenshot):

Also to put this into perspective, this last screenshot was generated over the default baseplate. You can see the corner right there in the bottom right

20 Likes

This is super neat to be honest

3 Likes

I love this!! Great job!

1 Like

Tip for further optimization: Generate hash tables of the occupancy/material data as chunks of voxels. If you have many chunks that just have full occupancy, you only have to store them once. In theory you can even have a single byte indicator for whether a chunk is full or empty to compress the space further.

5 Likes

It feels like I should say “This makes no sense,” but quite honestly its pretty simple in theory. Would you end up creating a plugin out of this? This would be pretty handy.

@Crazyman32 already made a plugin that saves and loads terrain. If you are handy with a script editor you can grab the module inside of it and use it in a live game

3 Likes

What are its limitations?

Also @nurgenius I probably won’t make a plugin out of this, at least not in the near future. It might be worth integrating this + a bunch of other terrain functions I’ve scripted into an all in one tool but for the time being my system of pasting code in the command bar works fine for my workflow

1 Like

I don’t know… I’ve never stress tested it. It just uses the little known TerrainRegion object. The module just makes handling it a little bit easier and cleaner

Incredible work my friend.

1 Like