Best way of storing a bunch of block values, that aren't always gonna be locked to a specific grid

[NinjoOnline]

I currently have a some what complex serialize system, that converts blocks from their CFrame to a grid based system, and stores their position via coordinates. With this, I’m able to save a ton of space and not have to worry about data limits. However, I am now delving into blocks and parts that aren’t as simple as a 3x3x3 block.

For example, a sign. When placed on a top or bottom surface its simple, and I can save using my current system as necessary. However, when it’s on a side surface, I obviously can’t just do my standard all pos values / 3 to get the grid position. What’s my best course of action?

o

Example of storing block info would be like so

coords[x][y][z] = BlockId

and so if a sign placed on its side occupies x y z of 0 0 0, then

coords[0][0][0] = signId

but I can’t just store the signs id, as when it loads, it’d just place the sign in the centre of that position.

Should I just scrap this whole serialize system, and just straight up save blocks full cframes?? Problem I fear with this is reaching data cap way too easily. Just a flat plate as show in my screenshots takes up 0.4% of the data store, and players can buy multiple slots, 15 total, so 15*0.4 is 6% in just flat grass.

[smellfish_360]

Instead of saving the ID, you can save the SignObject using OOP

local SignObject = {}
SignObject.__index = SignObject

function SignObject.new (CFrame,ID,coords,Text)
local this = {}
this.CFrame = CFrame
this.ID = ID
this.coords = coords
this.Text = Text
--Whatever parameter you wish to have. these are just examples
setmetatable (this,SignObject)
return this
end

And then you have everything neatly organised into one “SignObject” that you can create an acces at any time.

One other tip is that (if you’re recreating minecraft) to use a system that combines all blocks into bigger blocks. this way, a lot less polygons have to be rendered, thus saving yourselve some frames.

You also only have to save the blocks that have changed, not the entire terrain. this will save you a lot of storage on the datastores and simplies it a bit.

[ThanksRoBama (Emma)]

You need to encode additional information, not just the position. If your sign can only be in one of a few discrete states, you need log2(num_states) bits to store which state it’s in. E.g. if these are your states:

• On ground
  • Pointing N
  • Pointing NE
  • Pointing E
  • Pointing SE
  • Pointing S
  • Pointing SW
  • Pointing W
  • Pointing NW
• On side of block
  • Pointing N
  • Pointing NE
  • Pointing E
  • Pointing SE
  • Pointing S
  • Pointing SW
  • Pointing W
  • Pointing NW

… then you have 16 possible rotations, requiring log2(16)=4 bits in addition to the position information. If you always store the world as a sequence of blocks, and store blocks as a sequence of bits, and always store the block ID as the first part of each block sequence, you can make a look up table to keep track of how many bits are required for each block type.

[NinjoOnline]

I’m slightly confused what you mean?? The sign can only have 4 rotations to my knowledge

[ThanksRoBama (Emma)]

Depends how you want it to be. Do you want to allow only 90 degree rotations and being placed on top of a block? Then yeah that’s 4 states requiring 2 bits. Haven’t played MC in years but IIRC signs can be placed at 45 degree angles on the ground, and also placed on the sides of blocks not just on top. Although obviously 45 degree rotations doesn’t make sense on the sides of blocks xD

Whatever your game logic is, you need to figure out how many discrete states are available to determine the number of bits required to store that.

[NinjoOnline]

Oh right. Rotations are pretty much sorted, I index them. I didn’t think about 45 degree rotations, but I imagine they can easily be done

[1] = 0,
[2] = 90,
[3] = 180,
[4] = 270,
[5] = 45,
[6] = 135,
[7] = 225,
[8] = 315

coords[x][y][z] = blockId .. "~6" -- 6 being the rotation id and ~ just indicating that it's a rotation

It’s just the sign text that I can’t save easily

[ThanksRoBama (Emma)]

Yeah something like that for the rotation would work.

If there’s a limited number of chars on a sign you can just store each character and leave empty ones as spaces. If you need to allow any length of text you could use a null terminated string.

How are you encoding and storing things? I kinda assumed you’d be using BitBuffer to base64 encode everything to store it efficiently as a string in a data store, but I’m not so sure any more

[NinjoOnline]

Honestly, what I am doing is probably not the most ‘efficient’ way, but it’s worked up until now I’m not entirely sure what a BitBuffer is, or even what base64 means, but basically:

Coordinates are stored in a table. The table is arranged as [x][y][z], so example, coordinates of 0, 0, 0, would look as so in the table

Coords = {
    [0] = {
        [0] = {
            [0] = ""
        }
    }
}

And then the string signifies the blocks id + any meta data (I guess you could call it) regarding that blocks info. This is all serialized, as I didn’t wanna do like “Rotation=90”, as figured with thousnads of blocks that’d just waste characters, so something like “~2” seemed a lot better.

So block id 1 (Stone) is at 0, 0, 0

Coords = {
    [0] = {
        [0] = {
            [0] = "1"
        }
    }
}

Block is 1111 (Sign) at 0, 0, 0, with a rotation of ‘2’, 2 being 90 in my indexed table

Coords = {
    [0] = {
        [0] = {
            [0] = "1111~2"
        }
    }
}

and so on

[ThanksRoBama (Emma)]

Oh I see, yeah that’d probably work! Could definitely be more efficient storage wise, but if that’s not a requirement for you then don’t worry about that or the base64/bitbuffer stuff I mentioned.

[NinjoOnline]

Are base64/bitbuffer worth looking into though? Unsure what they are or do, but if they are ways to compact data even further then I am definitely intrigued, as I do fear players reaching caps quite easily

[ThanksRoBama (Emma)]

If you want to store things optimally you need to use a binary format. The simplest example is trying to store a table like this:

local data = { 1, 2, 3 }

If you just store this in a DataStore, IIRC it’ll automatically be turned into a JSON string (JSON “encoded”), resulting in something like the string [1,2,3]

Since this is a string, it needs to use one character per character, which is usually one byte = 8 bits. The length of that JSON encoded table is 8 bytes.

If you make up your own binary format you can do much better. If you know the numbers will never exceed 2^8 - 1 = 256, then you can store each number as a single byte for a total of 3 bytes, a pretty big decrease. It’s a lot more complicated to come up with a format that lets you store everything you want, but it’s definitely worth it if you’re running out of space in your DataStore.

But you can’t actually write individual bits or bytes to a DataStore, again it automatically JSON encodes whatever you try to store, so it eventually gets turned into a string somehow. The workaround is to actually use a string as a data structure for holding binary data, and just give that string to DataStore because then it gets stored as-is. This is the purpose of base64.

You’ve probably heard of number systems like decimal, binary or hexadecimal. They’re the same, except they have different bases (10, 2 and 16 respectively). To store a single digit in base X, you need X different symbols. In decimal we use the digits 0-9, in binary just the symbols 0 and 1, in hex it’s 0-9 and A-F. Base64 is a number system like these that needs 64 different symbols to represent a single digit. The advantage is that each digit can hold 64 different states, equivalent to log2(64) = 6 bits with a single character, which when turned into a string takes up 8 bits per character. Not 100% efficient but it’s decent.

How do we store a number in a base64 string then? Well, if we keep track of how many bits each number takes, we just store a number by appending the binary version of each number to the end of the string. Keeping track of all this is a chore though, so it’s better to use a library for it. BitBuffer is that library. There are tutorials on how to do that here on the dev forums, they’re pretty good and well worth a read!