Saving entire maps to datastore

[Mind_Bind]

I’m currently planning to make a way for users to create custom maps for an FPS game I’m working on.
My FPS game is very similar to CS:GO, so this map editor should be able to:

• Support the ability to adjust position, size, color, material, etc. of any BaseParts (obviously)
• Support custom classes/objects: I want players to be able to easily create bombsite regions, parts that indicate where water generates, etc. I will refer to these parts as “special parts”. They can’t create their own custom classes
• Support custom properties: Players should be able to adjust whether an object should unload from a set distance, unload/load at a specific graphics setting, etc. They also can’t create their own custom properties.
• Groups (optional) : Be able to group parts together.
• Support undo and redo: This doesn’t need to be saved to the datastore.

Right now, I have an idea how I might pull this off. I can make separate folders, where the groups, parts, and special parts are located, and then I can create a custom script that converts that all into one string, but I don’t know whether such method can work.
Are there any limits to this method? Like, is there a max length on how long a string can get? Should I use a different method altogether or is it possible to save objects as they are?
If you know anything, please let me know.

Edit:
also im not asking for any code and instead for ideas/suggestions, but you can of-course write some if you have the time and effort. Who knows, maybe it will be useful?
I will also leave this thread open for other suggestions until I finish my map editor.

Edit 2: I decided to close this post and to make the post that is the most important for me as the solution, but this doesn’t mean that everything else is useless. Thanks to xGOA7x and Hazania for the help!

[Hazania]

I will get back to you on this in a short while. I’ll edit this comment for my response.

[Mind_Bind]

Make sure to reply to my reply so I receive a notification and don’t forget to check in tomorrow. Thanks for your help!

[DasKairo (Cairo)]

This is a pretty difficult task, considering how much data each property will fill up, and how much space you have.

In total, each key can hold roughly 4 megabytes, with each piece of data you want to save, they contain a character in them, a letter if you will, which will take up 1 byte, here is the data of units:

Unit (Decimal / Binary)	Amount (Decimal)	Amount (Binary)
Bit	1	1
Byte (B)	8 bits	8 bits
Kilo- / Kibi- (KB / KiB)	1000 bytes	1024 bytes
Mega- / Mebi- (MB / MiB)	1000 kilobytes (1000000 bytes)	1024 kibibytes (1048576 bytes)

Looking at this, 1 character would be very small compared to 4 million, but once you add all the properties, names, colors, sizes, positions, and specific attributes, it can add up fast.

In DataStores, you are not allowed to save Instances, it isnt possible with the API provided by roblox, so instead, we have to save all the data it contains, which can be clamped with a table, which is much better and will help use separate each piece of Instance data from another.
There are specific stuff that you can cant save in a DataStore Iirc, so for the example ill show you, Vectors will be in tables.
But in this Example, would be something similar to what you should do

-- Here we are storing important Data from our Parts and putting them
-- in a DataStore to save
-- This is a Regular Lua Table containing vital Data for the game to work
-- This is an example, handle your own Data how you want to.
{ -- Table Beginning
    Name = "SpecialPart", -- useful if you are trying to specify something
    Color = "ff0000" -- means White in hexcode (255,0,0)
    Size = {X = 1, Y = 1, Z = 1}, -- Size
    Position = {X = 25, Y = 25, Z = 25}, -- Position
    Attributes = {Time = 5, Contested = false}, -- Special Part Data
    Material = "Brick" -- Part Material
} -- Table End

When you save stuff to DataStores, it will be convertted from Lua Code to JSON Code, which will look like this:

// This will be what it looks like when convertted to JSON
// this script block is also formatted using json
[
    "Name": "SpecialPart", // This can be useful if you are trying to specify something
    "Color": "ff0000", // this means 255, 0, 0 in hexcode (Base-16)
    "Size": ["X": 1, "Y": 1, "Z": 1], // there is likely a better way to do this
    "Position": ["X": 25, "Y": 25, "Z": 25],
    
    "Attributes": ["Time": 5, "Contested": false],
    "Material": "Brick" // Part Material
]

The total amount of characters (Excluding Comments), will be around 183 for this example, but its depending on how many characters are used, if we have 2000 parts in the map, and we are trying to have them saved using our Key, multiply 183 by 2000 (2000 parts will each have their own Data, so this would be nessacary), we would have 366000 bytes used (0.366 of a megabyte) , a pretty big amount, and if calulate this to a percentage, we are using 9% of what the Key can hold.
Adding more Data can significantly increase the amount of characters used, which can get you very close to reaching that 4MB limit, so every character is nessacary here in order to save thousands of objects at a time, if you plan to set a limit on how much stuff can be there, all power to you, you will likely never reach that 4MB limit, but this is still a big thing when it comes to saving data.

There is a video that covers How to Optimize / Compress your Data, which I would recommend you look into

[Mind_Bind]

Alright! I will probably create my own converter which basically shortens everything. This will be probably a very “I would rather only eat raw eggs for the next month”-type of painful process, but who cares when you get a few bits less, right?
Thanks a big ton, I appreciate your help! I’m guessing this took a long time to write!

[Mind_Bind]

Follow up question, since I’m making my own compiler without jsonEncode, can datastores save strings containing emojis? I just want to make sure that once a create a map and save it so it won’t turn to a corrupted file. By the way, do emojis or any other special characters take up more space than a regular character?

[Hazania]

jsonEncode happens automatically when you save anything to a Roblox datastore. You can’t bypass that regardless of if you do your own compiling unless you use your own database on an external server.

I believe emojis generally count as 2-4+ characters so just keep that in mind. Since they are part of strings I don’t think it’s an issue to save them, however you ought to test it before you allow users to risk losing their data.

I am the scripter for a game called Castaway, and our building system supports each player to place close to 20k objects on their island. That is more objects than we plan to ever make available to them, and unless you are allowing players to place tiny parts without a cap, it should never be a problem. If you want some info about how I did it, I’d be happy to share.

[Mind_Bind]

I made a compiler system that basically converts every part into the map into one VEEERRRY long string. So do I still have to worry about the json part?

Such string can look somewhat like this:
“p; ~W1, 8`a, a#9; ff0a6dc; 1; 0; 0:”… [other parts]
basically everything is shortend as short as possible. “;” separates the properties, “:” separates the parts. The first property the className, the 2nd is a vector3 so it can be the part’s position, 3rd is the part color, the rest are bools concerted to numbers.
All numbers are converted into some sort of base, for colors I decided to concert the numbers into Base16, but for all other numbers I created a custom base has about 90 different characters.
I also, for the sake of readability, added some spaces after every “;” that aren’t actually in the fully compiled map string.

This is only an example and it misses some of the other important properties, but I want you to get an example on how I decided to convert a map into a string. I also added 3 categories but they are almost exactly the same, so I won’t describe them in high detail:
Basically the regular parts, bombsites and map settings are split so decompiling will be a lot easier. These 3 categories are split by “/” instead of “;” or “:”

Also, is there a list of characters that aren’t supported by strings and take up more bits in size then a regular character?

[Hazania]

@Mind_Bind

So do I still have to worry about the json part?

I don’t know what you mean by this question.

@Mind_Bind

This is only an example and it misses some of the other important properties, but I want you to get an example on how I decided to convert a map into a string. I also added 3 categories but they are almost exactly the same, so I won’t describe them in high detail:
Basically the regular parts, bombsites and map settings are split so decompiling will be a lot easier. These 3 categories are split by “/” instead of “;” or “:”

This sounds like a good approach, you will need to do some stress testing and consider putting hard caps in place for part count. How many characters does the average part come out to?

@Mind_Bind

Also, is there a list of characters that aren’t supported by strings and take up more bits in size then a regular character?

I think just emojis, all standard characters including punctuation should count as one character each. You can test any character’s length by printing string.len(“insertcharacterhere”) to see how many characters it counts. You will notice that emojis vary in length.

[Mind_Bind]

Thanks!

I should rewrite my first question in greater detail probably.
What I meant is:
Should I still convert to JSON, and if so, will this break some stuff? Like, will some special characters dissappear, or will the structure change because of the encoding?

[Hazania]

@Mind_Bind

Should I still convert to JSON

You don’t have to, Roblox does it automatically when the data is saved using DataStoreService. When it is loaded, it decompiles. This happens regardless of what you do, unless you use an external service to save your data.