Developers need reliable DataStores. What we have now is not acceptable

I personally have not received any reports of total data loss for my games. I’m going off of the countless reports for many top games including Vehicle Sinulator, Island Royale and others.

The way I utilize datastores has been used against me in the past to shut down reports of data issues, but this is not a valid defense as games that do not use this method are hit just as hard whenever an incident occurs.

8 Likes

There’s two issues here.

  • Widespread datastore outages preventing gameplay
  • Recently reported datalos.

I’m almost certainly GetAsync() calls did indeed return nil in my game.

  • I have specific code to detect
    • Datastore throwing an error while loading
    • Datastore not loading at all
  • GetAsync() calls occurs very lowly in my game
  • I only make UpdateAsync() calls which do validation before saving (shouldn’t have wiped anything unless :GetAsync() returned nil and thus, assumed it was a new player)

My game has never experience datastore loss for the last 4 years. I think it’s suspicious that this week I got reports.

29 Likes

My experience hasn’t been terrible with datastores until now. I rarely had complaints of someone’s stuff not saving, and never ever had someone’s data completely wiped. Until it happened numerous times starting a week ago. I don’t have a backup datastore system in place, so some of these players lost lots of stuff that I have no way of verifying they ever owned. And I’m still getting people who just bought something with products(a big reason why I prefer gamepasses over products where possible) having their data wiped not long afterwards.

The issue certainly was my server incorrectly being told that some player’s data never existed. I ran into this myself, my own server recommended I do a tutorial which is only for brand new players who never had datastores.

6 Likes

I’ve been sitting at my computer trying to work on my game while also fixing peoples data
I’d hate for people to stop playing my game because of an overall roblox data issue

But profits have dropped sharply and my rating went from an 87% to 80% because of this, and the playerbase was already fairly small
Would be nice to get some banner or something on the site so people don’t assume it was just my game and downvote it

11 Likes

Many players have reached out to me on Twitter saying they have lost their data, and the tweets are not stopping. The worst part is their data is gone forever, some players have been playing for years.

Hurts me to see dedicated players leave Vehicle Simulator because they lost their data to something completely outside of my control. Not a very good reason to use Roblox as a platform (oof).

Priorities at Roblox must be double checked, this is just too urgent.

22 Likes

To be honest, I’m slightly disappointed that there isn’t more urgency on roblox’s part. I understand it’s very difficult to find errors in your code when you don’t have evidence and you can’t hypothesize what the problem may be, but with this many experienced developers complaining, something is seriously wrong and out of our control.

It isn’t all of us making mistakes, it’s pretty easy not to wipe your own player’s data, it shouldn’t be up to us to prove it isn’t our fault after all these developers come forward at the same time.

I don’t know if I’m out of place here, just voicing my feelings.

15 Likes

Ever since last Friday I’ve been hearing about the “data wipe bug”. Our community discord has players who are still afraid to join the game a week later because they think their data will be wiped. The worst part is I can’t tell them they’re wrong :confused:

The plan is to migrate Swordburst 2 to a mongodb database so that our players feel safe playing again. TeleportService + DataStores have never worked well together, but full data loss is too severe to overlook

16 Likes

Been having quite a lot of issues with DataStore errors and requests taking unusually long to process. This has been happening from time to time for most of last year, usually during peak hours on weekends or holidays.

These widespread reliability issues have a large impact on the game as a whole:

  • Loading times increase
  • Developer product purchases can’t be processed/granted
  • Player data doesn’t save properly, causing progress to sometime be lost

The large amount of messages from concerned players also put a lot of strain on me as a developer.


Thankfully haven’t received any reports of complete data loss in Welcome to Bloxburg recently.

8 Likes

I had this issue a week ago in Redshift Arena.

Luckily, it was only the System / Settings datastore, so people got their stats wiped, as well as their settings.
This wasn’t too much of an issue since XP / Levels weren’t implemented yet however.

It’s still a very scary thought that this can happen at any given moment.

4 Likes

I’ve had a number of reports from my players that their data was completely wiped. Normally when we have data store failures (which aren’t common to begin with) we see data from only one play session getting lost (we actually have a system that queues requests until they can be completed so for short outages we can save data later on), but this was the entirety of the players data, gone and reset as though they had never played before. I’ll be checking the analytics reports later to see what errors were produced during this time.

Anyone have any suggestions for how to reimburse players? I can’t manually edit the data stores of hundreds of players… I might just give an “I’m sorry bonus” to everyone for damage control. This isn’t ideal as it gives free things to players who never purchased anything, and can’t cover the loss of really committed long time players.

One of the players who lost data had 207 1st place wins (and lost them all), that takes over 60 hours of game play to achieve… its a real loss for them.

Edit: One of my players reported this as having occurred around the same time top games were set to nil. Is it at all possible that the mass exit of thousands of players (and servers trying to save their data on exit all at once) strained the data stores?

10 Likes

I’m also seeing huge “request added to queue” spam recently and it feels like a bug because I’m almost certain that I’m not saving the same key within 7 seconds, ever. I’m convinced this is an issue but its very difficult to reproduce since it’s inconsistent :confused:

4 Likes

A post was merged into an existing topic: Off-topic and bump posts

Our engineering team is actively investigating this issue by directly contacting the developers impacted.

Thank you to everyone who actively brought up this issue. It has helped us get a step closer to figuring out the root cause of this issue.

We will continue keeping you up to date as we find out more information.

19 Likes

Maybe, but the first issues started for me January the 3rd(as in, I haven’t found any instances earlier than that), and apparently still happened several days afterward, and in between.

1 Like

Everyone is affected by poor reliability with DataStores. It wasn’t just this incident. Thousands of players lose some progress in every game every other weekend when DataStores start returning Internal Server Errors for no reason.

This is the reason that many top devs are considering switching over to their own databases. DataStores are just not reliable for anyone.

6 Likes

Please to not spread misinformation. Prior to this incident (starting mid December), all data loss reports we received ended up being due to developers misusing data stores.

Examples:

  • Returning ProductPurchaseDecision.PurchaseGranted before record of the purchase was successfully saved to data stores
  • Overwriting player data without checking that data had properly loaded

That is not to say that data stores are fine. We know there is a ton of room for improvement. I just want to be clear that data stores data loss is not something we have previously had problems with. We are still actively investigating this latest issue and trying to determine a root cause. If you have specific information that can help us narrow down the issue, please let us know.

3 Likes

When a player leaves the game and multiple attempts to save their file to DataStore fail, they can and very frequently do lose the last few minutes (or more) of their save file.

Spikes in DataStore failures and partial outages have been a problem for a long time. This incident was particularly nasty because it was total data loss, but partial loss occurs very frequently, and there is only so much developers can do to prevent it when requests start failing left right and center.

13 Likes

I get that is is hard to track down what is causing these issues. But when you have so many devs recording fail rates from DataStores, and you have devs of top games like Welcome to Bloxburg and Royale High voicing their concerns about overall reliability, it is incredibly frustrating that this longstanding issue is being met with resistance and deflection.

14 Likes

Didn’t change anything about my code and sometime last night when I was asleep players started losing all their data in mass. This is rather unfortunate.

At this point it’s looking more necessary to use a third party database. With bugs like these, given the current DataStore system it’s too difficult to keep backups (with a bug like this it’s not reliable) or issue rollbacks. It’s great that Roblox offers a service like this for free and it’s mostly been reliable in the past for me, but I’ve always felt a lack of control with DataStores.

17 Likes

I feel like with the growth of Roblox’s playerbase, it’s time they look into upgrading DataStores & increasing the DataStore limits, rather than keep us on something that may or may not have been around back in 2012 / 2013, when the playerbase was arguably much smaller. Force-stopping a few kilobytes of data seems like artificial hindrance to me.

3 Likes