DataStores Incident Report

Summary

On October 20, Roblox released a game server update which caused issues when requesting GlobalDataStore through GetDataStores. The version has been rolled back and restored, and now current data is being read/written correctly. However, this update may have affected some of your players’ data.

Please reference this post on steps you can take to determine if your players’ data was impacted and what to do if it was: [Update] DataStores Incident: Data Consistency Checker API

Timeline (PST)

10:02AM: Started release of new version.

10:25AM-11:00AM: Increasing reports of issue with DataStores.

11:00AM: Rollback started.

11:15AM: Rollback and player transition to different server complete.

Impact

Who was affected?

Games which attempted to save data to a DataStore in new Roblox servers (v. 453).

How were they affected?

Any data saved between 10:02AM - 11:15AM PST on an affected server, to a GlobalDataStore, has been lost or reverted. Existing data before 10:02AM was not overwritten. OrderedDataStores were not affected.

However, if you were using an OrderedDataStore and DataStore together:

  • The OrderedDataStore has not lost any information, and therefore may point to data which was lost from the DataStore.

  • Modules such as DataStore2 use this approach to store backups of player data. In this case, the OrderedDataStore is likely to be out of sync and point to missing data.

This table gives you an idea of the exact issue:

Key OrderedDataStore Value DataStore Value EmptyScope DataStore Value
54 54 <NEW_PLAYER_DATA> nil
53 53 nil <NEW_PLAYER_DATA>
52 52 <PLAYER_DATA> nil
51 51 <PLAYER_DATA> nil
50 50 <PLAYER_DATA> nil
49 49 <PLAYER_DATA> nil
48 48 <PLAYER_DATA> nil
47 47 <PLAYER_DATA> nil

What can you do if you were affected?

We have released a temporary DataStore API which lets you query any data that was redirected to an empty scope during the incident window (Oct 20, 10:02AM - 11:30AM PST). The API will help you solve inconsistencies with your players’ data. You have to restart your studio to access the new API. Please reference this post for more information.

FAQ

Q: Is it safe to re-open my game?
A. Yes, if you were using DataStores directly, and not versioning them.

If you were versioning them, then be aware that you may be saving new-data to players who have an old save in your game. The old save can be recovered but will become more difficult to do-so over time. Use your best judgement.

Q: Can Roblox do a manual rollback of our game?
A: We are looking into this, but are unable to offer rollbacks at this time.

Q: What about transactions or purchases which occurred during this time?
A: We have provided reimbursements to all affected players. Please direct them to check their Roblox Inbox for a message on a refund.

October 21 - 11:00AM PST Update

We are compiling a list of all games that were impacted by the incident. Once that is complete, we will reach out and provide tooling for you to restore or rollback player data if you were among the affected group. There will be references and examples on how to use the tooling if you decide to proceed with data rollbacks.

We will also be providing refunds to all players that made purchases during the incident. The refund will include messaging to clarify what occurred. If you are receiving messages from your players, please direct them to Roblox support so we can assist them.

Please standby for another update later this afternoon.

October 21 - 5:00PM PST Update

As of this evening, reimbursements to all affected players are being processed. If you receive messages from players regarding inconsistent Robux purchases in-game due to the incident, please direct them to watch their Roblox Inbox for a message on a refund.

Our team is actively working on a patch that contains a new API which will check for keys which were affected during the incident. We are aiming to release it tomorrow morning (Oct 22, 11AM PST).

We also want to further clarify the timeline of events and provide more information on what occurred.

Technical Details

When we began rolling out version 453 to game servers, the default scope of GetDataStore was changed from “global” to “(Empty String)”. This meant that all requests made through a GlobalDataStore on the affected servers were redirected to the empty scope, but still succeeded. Between 10:02AM - 11:30AM PST, data was written or read from the empty scope which resulted in players appearing to have “lost data.” Since rolling back to version 452, the default scope was corrected and players appeared to have their “lost data” restored.

However, since OrderedDataStore was unaffected by this issue, reads and writes went to the correct scope. If you use OrderedDataStores to point to data in a GlobalDataStore, this data could have been written to the wrong scope resulting in data mismatches. We will follow up with more details on this use case shortly.

Next Steps

We anticipate to have the API live tomorrow around 11AM PST and will be providing documentation on how to use it to resolve data inconsistencies.

We are doing a thorough investigation of the problem and will provide updates on how Roblox will ensure issues like this will be prevented in the future.

October 22 - 5:15PM PST Update

We have released a temporary DataStore API which lets you query any data that was redirected to an empty scope during the incident window. The API will help you solve inconsistencies with your players’ data. You have to restart your studio to access the new API. Please reference this post for more information.

224 Likes

This topic was automatically opened after 6 minutes.

Are there new steps being put into place to make sure that incidents like of today do not occur anymore? As well as obviously seeing the issues that came about from this wilI new data restoration features be put into place? I am sure many developers do take into account a possible loss of revenue so I think it is important to find out.

64 Likes

Appreciate the update. Rollback via a tool or something would be nice. Manual rollback via the developer sounds painful. I guess, then again, it’s based on how much data was lost.

6 Likes

I mean, I respect the fact that you guys at least fixed the issue very quickly.

6 Likes

For any developers working on a new project using DataStore2–I recommend switching the saving method to “Standard”. This will make them use traditional data stores rather than ordered backups:

DataStore2.PatchGlobalSettings({
    SavingMethod = "Standard",
})

You can learn more here: https://kampfkarren.github.io/Roblox/advanced/saving_methods/#standard

The only reason this is not the default is because of backwards compatibility concerns.

37 Likes

I’m glad this was fixed quickly. Tons of respect.

12 Likes

Well, it happens, but in either case, thank you for fixing this extremely quickly!

4 Likes

I 100% think ROBLOX should rollback our games if we ask them to before this happened. People who had Worker Points in my game who joined during that time period lost them, and it isn’t fair.

8 Likes

Hundreds of thousands of players may have lost all of their progress. This will no doubt cause many long-time players of certain games to quit if there is no way to get all of their progress back. There definitely needs to be a way to rollback datastores in the future.

8 Likes

Out of curiosity, why are there no tests in place to prevent this? It seems exceedingly obvious and easy to test.

51 Likes

So are we going to use the same method provided here:


To restore the users data?
Seranok’s method was neat and seemed to give an option to roll back the datastores. Is the data automatically going to go back to the previous version this time?

3 Likes

From the standpoint of a developer, I can say that this happened to be one of the failed updates of ROBLOX.

As @Dekkonot said in his thread, there should have been a few more tests before launching this update.

A way to fix this would usually be manual restores or a second method.
Anyways, I hope the rollback is coming soon. I know a few developers whos games and player base got hit hard from this.

3 Likes

Hey folks,

I recognize that there is a lot of concern regarding how we at Roblox will be treating this matter internally. Let me assure you that we will be doing a thorough post-mortem. This specific incident was a complex and complicated issue, and foremost, we want to make sure that we are doing things right for you all. We’re actively working on improvements, both in code and process, and we’ll update you when we have some tangible, tested solutions.

25 Likes

Seems like Adopt Me haters got what they wanted. A lot of the popular games were getting shut down because of this.

Hopefully, we can get our data back for any games that we played during this time.

6 Likes

Why would you recommend this for new projects, out of curiosity? I happen to use the ordered backup method (not using DataStore2, but a custom system), and I have failsafes for API errors like this where, if there is ever an error in retrieving a DataStore, or if there’s ever an error in retrieving data at a key, the player’s data will be marked as “unsafe to save”, and no purchases will go through; this ensures that the empty data retrieved in an incident like this won’t cause the data from that play session to override existing data (meaning you lose your data for that play session, but not for everything before), and purchases will be processed the next time the player rejoins iff the error has disappeared.

So far I haven’t had any complaints about data loss from this incident using the ordered backup store method.

8 Likes

My frontpage game had 18k+ concurrent players (Super Power Fighting Simulator) during the time of the Data Store issues. We closed our game as soon as the problem occurred.

We are using DataStore2 method of saving our game data and found a way to restore back to a previous ordered backup with the help of @vCaffy who also experienced issues with his game. Even still, after restoring most player data which was lost and re-opening our game back up, there are still players who have lost data.

This was extremely distressing for us as we had to close for 8 hours in order to resolve these major data loss issues. We suffered a large loss of revenue, caused us a lot of mental stress and has damaged our player base all due to this critical Data Store outage.

Hoping for a solution to this issue that can restore player data back fully or receive better support for developers when issues like these arise.

16 Likes

If this happens again I recommend you use a Database for your roblox game, (firebase, etc) to prevent these problems for your game.

1 Like

As someone who maintains databases outside of Roblox, I would never recommend using external data storage to anyone unless you have overly complex data requirements or want to do something realtime between servers. Roblox data stores are easily scalable, offer less points of failure (if wrapped properly), and are free to use. Chances are, if you are hosting your database on an external service, you are going to have issues if your game experiences a large influx of players. I’m surprised how many people don’t realize how much easier (and cheaper) data storage is when using Roblox’s services.

7 Likes

It’s used to prevent data loss or Data Outage.

1 Like