DataStores Access and Storage Updates

Updates Announcements
#190

I guess this will affect a bit games using berezaa’s method of saving data, which has been infamously known for using a lot of data on roblox’s end, but being completely error proof

fortunately, data versioning is now directly supported by roblox so i guess it can be an easy migration, as normal saving consumes far less

as for the limits. i think people are overreacting. i got a game that could be considered “dead” or “zombie”, 30k keys of around 50 strings each are just ~500kb, and the game got a 46.3gb limit. So i can assume i am pretty safe, now imagine what the limit is going to be for big games

#191

You wouldn’t even need to store the position if you structured your data correctly. you could just derive the block’s position in the chunk based on its index in the block list of that chunk, meaning you only spend probably 4~ bytes representing an air block for any blank spaces in-between blocks in the list.

1 Like
#192

This could have been amazing but 1MB is too little for games that saves player’s build, especially if they are voxel builds.

1 Like
#193

no need cuz we know exact amount of blocks in chunk. Just use index to retrieve it.
About block type - it uses N amoung of data. But blocks can have their datas too - stairs orientation, slabs - upper, down, or mb full. Chests - their content, wool - it’s color.

#194

This is only somewhat true, you can “store” buffers in a DataStore, but everything stored in a DataStore is a string, no matter what. Roblox simply converts the buffer to a string automatically when you call SetAsync(), there’s a huge amount of overhead from this, so this actually ends up making buffers more inefficient on average than just using your own optimized data format.

You can actually see this pretty well with the new DataStore manager:
image
This was generated with this code:

local ds = game:GetService("DataStoreService") local ds = ds:GetDataStore("datatest") local buf = buffer.create(8) buffer.writeu8(buf,0,255) ds:SetAsync("TestKey", {["TestValue"] = buf})

If there is some change to this in the future, like allowing developers to write custom binary file formats (which I really doubt, because of security reasons.) then buffers would definitely be by far the best way to go about storing a mass amount of data.

#195

It’s confirmed that overhead is 25% of buffer size.

#196

I don’t think even 50% of the people in this thread are making UGC creation experiences despite citing it as their problem scenario. It’s not good to wax hypotheticals and fearmonger; you should actually provide exact data from your observability dashboards, though if you actually try and do that to prove a negative point, you’ll realise that even for a building game you aren’t getting near 1MB even without your own compression strategies applied before Roblox’s and that there’s nothing to be scared of.

I am the programmer of a data-heavy dungeon-crawling RPG that is still running on a mix of both modern optimisations and outdated practices from 2017 due to the difficulty of converting over and lack of attention to it yet. Our strategies are not great yet I wouldn’t even sneeze at 4MB. One of my top players who has a save file from 2020 doesn’t even push 100KiB of data.

image
image1920×1040 159 KB

Just trying to put things into perspective by opening their entire data profile in JSON format in Visual Studio Code. There are also tons of (better) resources for DataStores than the “berezaa” method which, for its time, was helpful, but Roblox has made so many strides since then that it doesn’t take much effort anymore to design good data infrastructure for your experiences anymore.

Suggesting that the only possible game to make now are “cash grabs” and “simulators” is crazy, unfounded and entirely untrue. Don’t spread misinformation if all you have to go off of is hypotheticals. You need to show real data and real problems. Go to your dashboards now and see that you barely crack the limit, and if you do, then review your data systems. Only if you’ve tried everything and still are crunching should you actually start to be worried; and the people that need to be worried have already been contacted for expedited support.

4 Likes
#197

Anyone else has thousands of number datastores that seems to be coming from nowhere, all created exactly on the same date?

image
image1851×985 104 KB

image
image1511×364 38.7 KB

#198

A DataStore should be considered “deleted” when it has no keys left. Try running a job to clear out the keys that you don’t need more (and migrate where necessary) and see if that impacts your DataStore count. DataStore2 really did a number here huh.

2 Likes
#199

I mean if Roblox isn’t gonna store the data why not just allow the client to play with the data.
Seriously though 1 MB isn’t a lot of data points. If an integer is 4 bytes (a common default int size): 1,048,576 / 4 = 262,144 integers

Seriously a disappointingly low number. Sure everyone who decides to only make a standard player inventory or a standard player housing or a standard leaderboard… great. Was kinda hoping developers would use Data to start doing interesting things. But I guess that’s out the window.

1 Like
#200

If i’d be honest, in my opinion this is one of the riskiest updates to the datastores. This update will add vulnerabilities and will destroy some games. What if some exploiters will decide to flood the requests and the game will hit the limit? Also if CCU counter will not be always up to date and update immediately when someone leaves/joins it will make so much troubles with server migration when developers will update the game (imagine a game with a lot of leaderboards and the server migration, when servers will be migrated leaderboards will start to load and if CCU counter and the request limits will not be updated immediately it will most likely cause the rate limit and broken leaderboards along with broken players’ data).
I may be wrong somewhere, but it’s my current opinion.

2 Likes
#201

Roblox uses 8 byte numbers, not 4 byte. 4 byte only for Vector3int32 and Vector2int32, and Region3int32

#202

Yes of course LUA uses 8 byte numbers but with a 1 MB limit are you seriously going to be using native 8 byte numbers? That alone will blow 50% of your budget for precision that isn’t used. Guess I haven’t really had a need to explore how you’re supposed to compress data in Roblox because previously the data store was without limits… there’s no way they just introduced limits without a way to store values with lower precision right? I assume something like pack?

#203

its the end for games like the graveyard

2 Likes
#204

No ofc, I’ll do as much compression as I’ll be able.

#205

I completely agree with your perspective—this update does introduce significant risks to the datastores, particularly in terms of vulnerabilities that could impact the stability of certain games. The issue of exploiters flooding requests is a serious concern; once the limit is hit, it could cause massive disruptions, essentially breaking critical features like leaderboards and potentially players’ data. This would be a devastating consequence for experiences reliant on seamless data flow.

Moreover, the CCU (Concurrent Users) counter not updating immediately adds another layer of uncertainty. During server migrations or rapid player transitions, delays in updating the CCU counter could lead to inaccurate request limits being applied, triggering rate limits unnecessarily. As you’ve pointed out, the scenario where leaderboards start loading during server migration is a perfect example of how this oversight could escalate problems, resulting in broken leaderboards and compromised gameplay data.

While Roblox’s intentions to provide tools for optimization are commendable, the potential for unexpected issues arising from these limits feels like an unnecessary burden on developers who already have to juggle various challenges. Introducing such a change without ensuring absolute precision in metrics like CCU or robust safeguards against exploitation risks leaves a lot of room for error. I hope Roblox revisits these changes and considers feedback like yours to prevent negative outcomes for developers and players alike.

In a nutshell, the dashboard is a good idea, but the limits are not frfr

1 Like
#206

While it may be true that Roblox uses 8 byte numbers in Luau, you can get 4 byte numbers through buffers

#207

I take issue with the new storage limits.

This change benefits only Roblox while harming developers. There are no positives whatsoever for the developer. The future storage limits for anyone that doesn’t know, is a base 100MB for the game with an extra 1MB per unique player that has played the game. Personally, if these changes are final, I’d end up losing a project I’ve worked on for 2+ whole years! More on that at the end.

The real problem here.

Roblox needs to look into how a game increases or gets its limits. Going from unlimited storage to a literal megabyte per player is too sudden and large a difference. Technically the difference is infinite but we don’t talk about that.

There are millions of games where probably nothing even happens, the base 100MB allocation not even being required. Then there’s the few thousand that actually do things! The way allocation is being handled right now is simply terrible.

This should be public information.

During development of my own project, I have not saved any world data into Roblox’s servers during testing because the values I am working with will change as I work on it. Now I’m being punished further, because Roblox’s automation wouldn’t even be able to figure out that I will be going to be over the limit since I did them the favor of not saving anything yet! This means I will never even receive said inbox message in question until I finish my game! Aka, far too late. This shouldn’t even be an inbox message for a few users. It should be visible to absolutely everyone in the off chance it actually has something useful!

There’s no way the acclaimed “0.0000054%” of experieneces that go over the limit by a good margin aren’t going to have some sort of special treatment. We should all be made privy to what will happen in the off chance another developer happens or will happen to fall into that group. There are already developers cancelling their projects due to the uncertainty raised by Roblox and that is absolutely terrible. Could be me too!

To the developers who are not hitting this limit.

I see a lot of developers claiming things like:

  • “The storage allocation is enough”
  • “You’re not the 30 in 55 million”
  • “It’s unrealistic to go over”

I don’t agree with these stances, while for most games this may be true. This stops creators both now and future from potentially making something truly great. To see their ambitions realized. Or to you know, not have their efforts wasted. There’s a limit to how much you can crunch your data and it’s not fair for the people that are affected.

Bonus

How this affects me.

I have spent nearly 2 years slowly working on a project, dealing with all of Roblox’s issues and new features coming out. Countless hours of rewrites, planning and learning. Now all of that time spent would come to waste immediately with these new limits. It was annoying enough having Roblox bugs stop me at every step, new features superceding custom implementations and more. This change would be the equivalent to slapping me and pointing the middle finger at me for all my hard, and unpaid work!

I’ve been working on a 2D grid based sandbox game where a world is generated within a given size. But every cell in the world can be changed by the player. Of course, due to gameplay reasons there may be extra data required per cell. And I don’t think I can even allocate 1 world for a player anymore.

I need to plan for the absolute worse case scenario, and the worst case scenario is the player has changed every single cell in the grid with unique values. My game requires 54 bits (7 bytes rounded up) per cell. My game requires all 54 bits, this is not up for discussion, you do not know what my game entails. I am not here to be told how to optimize my game, I’ve had plenty of time in those two years to learn lots of things and decide on how I should structure my game based on it’s needs. I am not so ignorant as to ignore further optimization, but I know what my game needs and really, do not wish to have to go there again.

Roblox does some compression on their end, but I am not 100% sure as to exactly how it works so I can only assume the worst case scenario is worse than it really is.

How little 1MB can be.

Let’s create a hypothetical where miraculously I only required 24 bits (3 bytes), the same amount of bits required to store a color value (I am not doing this).

I would only be able to fit a grand total of 333k cells with this! Now, 333k sounds like a lot. But remember, not only are we playing for the worst case scenario, but 24 bits isn’t much. It wouldn’t even be enough for a 600x600 image! If a minecraft chunk in the worst case scenario took up 3 bytes per block, you wouldn’t even have 2 chunks!

The worst case scenario for your information, is completely random values. Absolutely zero repeating patterns that can be compressed. It is impossibly likely that would ever happen, but the importance of planning for the worst case cannot be underestimated. Imagine your game blew up, and then loses all momentum because you hit a limit!

5 Likes
#208

It is very rare that any game that isn’t just a baseplate is gonna have a cap of 1 MB, if you have any amount of visits at all you’re not gonna have any problems

#209

Honestly this is the only reasonable take ive seen here for why this update is bad… however, I do have some tips to help you out

Stop assuming for the worst case scenario, start assuming for the current case scenario.

Let’s say the player only edits like 5000 grid cells, that means you only need to store 5000 grid cells, and the rest can all be stored in a single number; the seed.

It is highly, HIGHLY unlikely a player is gonna edit every single cell, so your best bet is to do this
Also, if a player, say, sets a cell back to the value which the seed determined, do not store it any longer, and just consider it a part of the seed again.
7 bytes is already pretty good for 1 cell atleast imo, you just need to make more assumptions