Developers need reliable DataStores. What we have now is not acceptable

Feature Requests Engine Features
#1

A recent issue that is believed to have caused DataStore calls to return nil and that led games to inadvertently wipe their players’ data has re-ignited the developer community’s attention and scrutiny towards DataStores as a service.

This isn’t really new. Poor reliability with DataStores is an issue we’ve been fighting for a long time. A lot of developers don’t realize that DataStore issues are far more common than they might think. DataStores quietly fail whenever there is a large spike of traffic to Roblox.

During these outages, players can lose much of the time and sometimes even money that they’ve put into our games.

Here’s a chart that I made while I was an Incubator at Roblox:

Using GameAnalytics, I tracked the amount of users affected by DataStore loading issues (how many players were delayed or could not get into the game at all due to DataStore issues. Errors saving data not included) and then divided it by the total player counts for each day to get:

image

Around 9/15, almost 12% of players had difficulty entering my game due to DataStore issues throughout the entire day. This outage only lasted a couple hours, so that rate was certainty much higher during the outage.

A lot of top developers are starting to notice these increasingly massive DataStore outages, usually on weekends or holidays. @callmehbob was forced to implement an external web server for her game Royale High to hook up to using HttpService whenever DataStores start failing:

This is unacceptable. With the revenue split that Roblox has in place, these vital services that are offered to developers HAVE to work. Something as crucial as player data saving can’t fail this frequently.

As developers, we need to be able to reliably save and access our player’s data without having to purchase and maintain an external database.

Please address these long-standing reliability issues.

Developers: please comment your experiences below. Have you received reports from players about data wipes or data loss? Do you track DataStore errors?

199 Likes
Script Injection Vulnerability
Data loss starting two days ago
GDPR - A Toolbox Talk
#3

Just going to point out it is not confirmed DataStore calls return nil. As I mentioned in the thread you linked, I used my personal knowledge from programming on Roblox to attempt to diagnose an issue. If anyone can reproduce a DataStore returning nil, please share here or DM me.

We are actively working on resolving these issues. Would love to hear what other developers are going through so thanks for making this thread.

14 Likes
#4

Appreciate your feedback, but I’m going to have to disagree that it could be anything else. Unlike normal DataStore outages, this was a temporary and widespread incident of total data loss, not partial loss or data rollback.

The only way, logically, that this would have occurred is if nil or empty data was passed to game servers. It is common practice to not save data unless valid (nil is valid) data is obtained from DataStores, so the total data loss would not be so widespread and so isolated if it was just poor practices from developers.

Given the information we have now, the most likely explanation is that for a period of a couple of days, some GetAsync calls did indeed return nil. This will be impossible to reproduce as reports of total data loss have ended.

It is also important that this individual incident does not take away from the main point of the post, which is that overall DataStore reliability is unacceptable.

22 Likes
#5

I haven’t gotten any reports for data loss in any of my games interestingly. There might be more factors to this.

–

This is another problem, most likely not what your are running into because Vestaria has a load save slot page (and how you explained the nil thing), but this might be the problem for other devs,

A problem we ran into a while back is that players would get kills before their datastore for kills even loaded, causing it to get overwritten. Just an oversight that we had back then.

6 Likes
#6

I personally have not received any reports of total data loss for my games. I’m going off of the countless reports for many top games including Vehicle Sinulator, Island Royale and others.

The way I utilize datastores has been used against me in the past to shut down reports of data issues, but this is not a valid defense as games that do not use this method are hit just as hard whenever an incident occurs.

8 Likes
#7

There’s two issues here.

  • Widespread datastore outages preventing gameplay
  • Recently reported datalos.

I’m almost certainly GetAsync() calls did indeed return nil in my game.

  • I have specific code to detect
    • Datastore throwing an error while loading
    • Datastore not loading at all
  • GetAsync() calls occurs very lowly in my game
  • I only make UpdateAsync() calls which do validation before saving (shouldn’t have wiped anything unless :GetAsync() returned nil and thus, assumed it was a new player)

My game has never experience datastore loss for the last 4 years. I think it’s suspicious that this week I got reports.

29 Likes
#8

My experience hasn’t been terrible with datastores until now. I rarely had complaints of someone’s stuff not saving, and never ever had someone’s data completely wiped. Until it happened numerous times starting a week ago. I don’t have a backup datastore system in place, so some of these players lost lots of stuff that I have no way of verifying they ever owned. And I’m still getting people who just bought something with products(a big reason why I prefer gamepasses over products where possible) having their data wiped not long afterwards.

The issue certainly was my server incorrectly being told that some player’s data never existed. I ran into this myself, my own server recommended I do a tutorial which is only for brand new players who never had datastores.

6 Likes
#9

I’ve been sitting at my computer trying to work on my game while also fixing peoples data
I’d hate for people to stop playing my game because of an overall roblox data issue

But profits have dropped sharply and my rating went from an 87% to 80% because of this, and the playerbase was already fairly small
Would be nice to get some banner or something on the site so people don’t assume it was just my game and downvote it

11 Likes
#10

Many players have reached out to me on Twitter saying they have lost their data, and the tweets are not stopping. The worst part is their data is gone forever, some players have been playing for years.

Hurts me to see dedicated players leave Vehicle Simulator because they lost their data to something completely outside of my control. Not a very good reason to use Roblox as a platform (oof).

Priorities at Roblox must be double checked, this is just too urgent.

22 Likes
#11

To be honest, I’m slightly disappointed that there isn’t more urgency on roblox’s part. I understand it’s very difficult to find errors in your code when you don’t have evidence and you can’t hypothesize what the problem may be, but with this many experienced developers complaining, something is seriously wrong and out of our control.

It isn’t all of us making mistakes, it’s pretty easy not to wipe your own player’s data, it shouldn’t be up to us to prove it isn’t our fault after all these developers come forward at the same time.

I don’t know if I’m out of place here, just voicing my feelings.

15 Likes
#12

Ever since last Friday I’ve been hearing about the “data wipe bug”. Our community discord has players who are still afraid to join the game a week later because they think their data will be wiped. The worst part is I can’t tell them they’re wrong :confused:

The plan is to migrate Swordburst 2 to a mongodb database so that our players feel safe playing again. TeleportService + DataStores have never worked well together, but full data loss is too severe to overlook

16 Likes
#13

Been having quite a lot of issues with DataStore errors and requests taking unusually long to process. This has been happening from time to time for most of last year, usually during peak hours on weekends or holidays.

These widespread reliability issues have a large impact on the game as a whole:

  • Loading times increase
  • Developer product purchases can’t be processed/granted
  • Player data doesn’t save properly, causing progress to sometime be lost

The large amount of messages from concerned players also put a lot of strain on me as a developer.

Thankfully haven’t received any reports of complete data loss in Welcome to Bloxburg recently.

8 Likes
#14

I had this issue a week ago in Redshift Arena.

Luckily, it was only the System / Settings datastore, so people got their stats wiped, as well as their settings.
This wasn’t too much of an issue since XP / Levels weren’t implemented yet however.

It’s still a very scary thought that this can happen at any given moment.

4 Likes
#15

I’ve had a number of reports from my players that their data was completely wiped. Normally when we have data store failures (which aren’t common to begin with) we see data from only one play session getting lost (we actually have a system that queues requests until they can be completed so for short outages we can save data later on), but this was the entirety of the players data, gone and reset as though they had never played before. I’ll be checking the analytics reports later to see what errors were produced during this time.

Anyone have any suggestions for how to reimburse players? I can’t manually edit the data stores of hundreds of players… I might just give an “I’m sorry bonus” to everyone for damage control. This isn’t ideal as it gives free things to players who never purchased anything, and can’t cover the loss of really committed long time players.

One of the players who lost data had 207 1st place wins (and lost them all), that takes over 60 hours of game play to achieve… its a real loss for them.

Edit: One of my players reported this as having occurred around the same time top games were set to nil. Is it at all possible that the mass exit of thousands of players (and servers trying to save their data on exit all at once) strained the data stores?

10 Likes
#17

I’m also seeing huge “request added to queue” spam recently and it feels like a bug because I’m almost certain that I’m not saving the same key within 7 seconds, ever. I’m convinced this is an issue but its very difficult to reproduce since it’s inconsistent :confused:

4 Likes
#18

A post was merged into an existing topic: Off-topic and bump posts

#19

Our engineering team is actively investigating this issue by directly contacting the developers impacted.

Thank you to everyone who actively brought up this issue. It has helped us get a step closer to figuring out the root cause of this issue.

We will continue keeping you up to date as we find out more information.

19 Likes
#20

Maybe, but the first issues started for me January the 3rd(as in, I haven’t found any instances earlier than that), and apparently still happened several days afterward, and in between.

1 Like
#21

Everyone is affected by poor reliability with DataStores. It wasn’t just this incident. Thousands of players lose some progress in every game every other weekend when DataStores start returning Internal Server Errors for no reason.

This is the reason that many top devs are considering switching over to their own databases. DataStores are just not reliable for anyone.

6 Likes
#22

Please to not spread misinformation. Prior to this incident (starting mid December), all data loss reports we received ended up being due to developers misusing data stores.

Examples:

  • Returning ProductPurchaseDecision.PurchaseGranted before record of the purchase was successfully saved to data stores
  • Overwriting player data without checking that data had properly loaded

That is not to say that data stores are fine. We know there is a ton of room for improvement. I just want to be clear that data stores data loss is not something we have previously had problems with. We are still actively investigating this latest issue and trying to determine a root cause. If you have specific information that can help us narrow down the issue, please let us know.

3 Likes