Hello developers, I am currently having an issue with the 4 MB limit of DataStore keys, I am trying to serialize a very large amount of data that exceeds 4 MB. (128 MB to be exact.)
However, I’m not sure if this will work due to the 30 second shutdown restriction of Roblox servers.
Surely trying to serialize that much data in 30 seconds would be near impossible?
All compression methods that I have applied such as Base90 encoding for large values have helped tremendously but still haven’t solved the issue. (I managed to get it so that I can store 25% more data with my current compression methods.)
Any help with this is massively appreciated,
Thanks, MBroNetwork.
What are you trying to serialize excatly, even with my infinite scp 3008 with saving clone still barely takes 0.2 percent per key, and that amount of mb surpasses even images, saving them would not be practical, even loading it ,if you are saving big terrain map you could create algorithm to generate map by a seed or smt
As for your question it should work in theory, because you can call different keys thus evading the 4 second wait time.
Trying to save manually modified values from the player, They can modify anything in the area.
My main concern is that there’s going to be a certain point where this data can’t even save correctly anymore due to the 30 second shutdown timer.
It just serializes the values for each modified position and then saves them to the key as a string under a dictionary/array. (I’m not very experienced with DataStores so I have no idea if this is a good way of doing this or not.)
Hello, I am not an expert in this field but dealing with large amounts of data and serialization can indeed be tricky, especially with the 4 MB limit in DataStore and the 30-second shutdown restriction in Roblox. However, you’re on the right track considering a chunk system, similar to the one in the FreedumbStore wrapper, and compression. Here are a few more considerations to optimize your approach:
Chunking: Continue with the idea of chunking the data into smaller parts. Even though the FreedumbStore can be seen as a somewhat extreme example, the basic idea of spreading data over multiple keys is sound. You can divide your data into manageable chunks, each well below the 4 MB limit.
Background Saving: To deal with the 30-second shutdown restriction, you might want to serialize and save data in the background continuously, not just when shutting down. This way, your data is incrementally updated over time, rather than trying to handle the whole dataset at once during shutdown.
Delta Updates: Instead of saving the entire data each time, consider saving only changes (deltas) to the data. This would drastically reduce the amount of data that needs to be stored at each operation, although it may require more complex management.
Binary Serialization: Although you have mentioned Base90 encoding, consider using a more efficient binary serialization format, if you aren’t already. Roblox’s built-in HttpService:JSONEncode method can be inefficient for large datasets. Instead, consider other binary serialization libraries that convert your data into binary format. This usually results in a smaller size than JSON or other text-based formats.
Further Compression: There might be other compression techniques you can apply, depending on the nature of your data. For example, if your data is numerical, it might benefit from a numeric encoding scheme. Also, consider if there are repeated data structures that can be simplified or eliminated.
Remember that handling large datasets is a balance between efficiency, complexity, and Roblox’s inherent limitations. It might be worth reevaluating if you really need to store 128 MB of data in the DataStore, and whether some of it can be pruned, summarized, or stored elsewhere.
Lastly, keep in mind Roblox’s request rate limitations (e.g., 60 requests per minute per game, 100,000 requests per day per game, etc.) as these could also pose a challenge to chunking and saving data regularly. Monitor your usage to avoid hitting these limits.
For my game Ive also saved the position, currently I round them a few digits and in the form of {a,b,c}
For rotation I am using the values as quaters(90°) instead of a vector 3(for ex 90° would be 1) and also how much object do you have, and do you save them only when modified by adding them to an array that you use a get request on?or you add it only when it does not exist in the table and if it does exist, you replace it
I’m currently just overwriting the current data with the new data when needed.
I have already implemented things such as storing the rotation data in numbers and the names in numerical IDs which are automatically Base90 encoded if they reach a certain size.
As far as I know, there isn’t a hard limit of 100,000 requests per day per game in Roblox’s official documentation.
However, there are other limits related to DataStore usage that you should be aware of:
You can have a maximum of 6 requests per second per game server.
You can have a maximum of 60 DataStore requests per minute in a single game server.
A data store has a maximum size limit of 8 megabytes.
These limitations are in place to ensure the scalability and reliability of the DataStore service. If you exceed these limits, additional requests may be throttled or fail.
It’s also worth mentioning that if you’re reaching these limits, it might be a sign that you need to optimize your data handling. This could involve batching requests, using data compression, caching data locally, or rethinking how and when you save and load data.
I understand the limit but I can’t figure out how to store so much data properly, So far I can store 200,000 objects in 1 key but I would need 64 keys to store everything which is a big problem.
First, consider if there’s a way to streamline your data. Do you truly need to store all 200,000 objects per key? If you’re storing properties for each object, are all of these properties necessary? Trimming unnecessary data would be the first step in managing large amounts of data.
Secondly, consider alternative approaches to storing and managing your data:
Use multiple DataStores: Each game can have up to 256 unique DataStores. You can use more than one DataStore to store data if one isn’t enough.
Data compression: If your data is text, consider compressing it using a method like Huffman coding or LZ77 before saving, then decompress it after loading.
Spread out your data: Spread out your data to different keys on a per-user or per-session basis, rather than trying to fit everything into a single key.
Use of external databases: You could consider using an external database for storage. Although this involves more complexity (you would need to set up an HTTP server that interfaces with your database, and use HttpService to communicate between your game and your server), it gives you much greater flexibility and control over your data.
Finally, remember that you’re trying to fit a large amount of data into a system that has limitations. There might be architectural changes needed in how you’re managing or using data in your game. It’s sometimes necessary to adjust the game design to fit technical constraints.
After some time, I have decided that I will look into compression methods such as zlib and then use encoding such as Base64 (to allow for datastore serialization functionality) in-order to cut down on the amount of data stored per object along with streamlining repeated values. (ex. Object1 is at 1,5,2 and Object2 is at 1,5,1 so we just need a value that represents 1*2 instead of storing several 1’s)
What do you think? (Also, Thank you so much for your help so far!)
You’re welcome! That sounds like a very sensible approach. Compression and encoding can indeed help you to reduce the amount of data you need to store and make your storage usage more efficient.
Compression libraries like Zlib work by finding and eliminating redundancy in the data. They are especially effective when you have large amounts of repetitive data. Using Zlib to compress your data before storing it can drastically reduce the amount of space it takes up.
Base64 encoding is a way of taking binary data and turning it into text so that it can be more easily transmitted in things like email and HTML form data. It’s important to note that Base64 encoding increases the size of the data by about 33%, but this may be outweighed by the compression you’ve done beforehand. So, even though it increases the size a bit, it’s necessary for making the compressed data safe to store in the DataStore.
Furthermore, you could look into data serialization formats that support compression, such as Google’s Protocol Buffers, BSON, MessagePack, etc. They might provide better space efficiency along with fast serialization and deserialization.
The method you mentioned, about storing repeated values only once, is known as data deduplication. It can help you to save a lot of space when you have repeated data.
However, all these operations (especially compression and deduplication) have a cost in terms of CPU and memory usage. You’ll need to balance the savings in data storage against the increase in computation time and complexity.
Remember, Roblox has a 6MB memory limit for each instance of a script, so you’ll need to take this into account when decompressing and decoding your data.
It’s great that you’re thinking about these kinds of optimizations! Just be sure to thoroughly test any changes you make to your data storage system to ensure you’re actually seeing the benefits you expect, and that there are no negative side effects.
I’ll be sure to test everything thoroughly, I wasn’t aware of the 6 MB limit for each script instance so I’ll be sure to watch out for hitting that limit.
Do you know of any other encoding that could be used that accomplishes the same thing as Base64 but won’t inflate the size as much?
I can only think of using something like Base85 or Base90 currently.
Never heard of Protocol Buffers, BSON or MessagePack, I’ll be sure to look into it!
