Optimizing Chunk Replication

Im working on a voxel engine that uses chunks, pretty much just minecraft. I store all the chunk data on the server and i replicate that data to the client. The issue is that the chunk data is really big, the chunk size is 16, 255, 16 with each block in the chunk having a {} attached to it for block data.

What the chunk data looks like:

local chunks = {
	[Chunk_Position] = { --> this is what im sending
		[Block Position] = {
			Name = "BlockName",
			Orientation = BlockOrientation,
			Tint = TintColor
		}
	}
}

Currently im converting the chunk data into a json and converting the json into a buffer and then compressing that buffer, but its still allot of data being sent.

Putting that whole table into JSON will waste a ton of bandwidth; it’s not exactly a good format for sending large amounts of binary data. Your approach should be to have an algorithm to generate the buffer yourself.

First, you should look to having a minimal data structure for representing a block. Name should probably be serialized as a number representing the block (which then gets converted back using a table), Orientation I’m guessing is a Vector3? That you can’t do much with, but if it’s just axis-aligned rotation, then just encode it 0-5: one for each direction it could face. Tint can be put into 24/32 bits, RGBA.

-- C-style struct pseudocode
Block {
	uint32 block_id;
	uint8 orientation; -- 6-256 are unused
	uint8 color[4];
}

From there, you can come up with a way to represent a chunk as an (sparse) array of blocks. This would be creating a data structure that defines the start and end index of a series of blocks, follow it with all of the blocks in that series. If Block Position is not a numerical type, that’s a bit more difficult. Ideally you’d find a way to encode the position into a one-dimensional array so you don’t have to worry about that.

-- C-style struct pseudocode
BlockArray {
	uint16 start;
	uint16 end;
	Block array[]; -- length = end - start
}

Some data pro will probably tell you this approach is extremely naive, but it’s better than nothing.

converting a JSON string into a buffer doesnt do anything because they are just the exact same uncompressed data

I think what minecraft does is that it only sends the blocks that are on the surface. For example, if you have a flat chunk with grass on top and stone underneath, you only have to send the stone blocks once the player mines down, not when they just want the grass blocks

just wait for next minecraft snapshot and look at their code, it will no longer be obfuscated

-- Pack a chunk into a buffer
function Buffers.PackChunk(ChunkId: number): buffer

	-- Get the chunk
	local chunk: {[Vector3]: {[number]: any}} = EnvironmentConfig.Chunks[ChunkId]
	if not chunk then return buffer.create(0) end
	
	-- Voxels module
	local Voxels = EnvironmentConfig.VoxelsModule
	
	-- Count surface voxels
	local surface_count: number = 0
	for voxel_position: Vector3 in pairs(chunk) do
		if Voxels.IsSurface(ChunkId, voxel_position) then
			surface_count += 1
		end
	end
	
	-- Incase a chunk if fully covered
	if surface_count == 0 then
		return buffer.create(0)
	end

	-- Buffer variables
	local chunk_buffer: buffer = buffer.create(surface_count * 5)
	local offset: number = 0

	-- Write data to the chunk buffer
	for voxel_position: Vector3, voxel_data: {[number]: any} in pairs(chunk) do
		if Voxels.IsSurface(ChunkId, voxel_position) then
			buffer.writeu8(chunk_buffer, offset, voxel_position.X)
			buffer.writeu8(chunk_buffer, offset + 1, voxel_position.Y)
			buffer.writeu8(chunk_buffer, offset + 2, voxel_position.Z)
			buffer.writeu8(chunk_buffer, offset + 3, voxel_data[1]) -- Voxel id
			buffer.writeu8(chunk_buffer, offset + 4, voxel_data[2]) -- Voxel health
			offset += 5
		end
	end
	return chunk_buffer
end

I used both idea’s and its allot better now. Went from ~50kb to ~4kb