[Update] DataStores Incident: Data Consistency Checker API

On October 20 at 10:02AM PST, we released a game server update (v 453) which caused issues when requesting GlobalDataStore through GetDataStores. All scopes of GetDataStore, regardless of value, were changed to “(Empty String)” which meant that all requests made through a GlobalDataStore on the affected servers completed but were redirected to the datastore with no scope.

The version has been rolled back and restored, and now current data is being read/written correctly. If you were only using standard DataStores (not linked with OrderedDataStores) for storing player data, you are most likely not affected.

However, If you use OrderedDataStores to point to data in a GlobalDataStore (such as the default save function from DataStore2), this data could have been written to an empty scope resulting in data mismatches. Your players’ data will be in one of the following states:

  • Unaffected, if their data was not touched
  • The current OrderedDataStores pointer is referring to the incorrect DataStores key. There are two potential scenarios:
    • Scenario 1: The player has an previous save and may wish to restore the previous save
    • Scenario 2: The player started a new save and may wish to continue the new save

The diagram below shows an illustration of the potential states of the regular Datastores (of all player data keys pointed to by the OrderedDataStore):




New API & Next Steps

We have released a temporary DataStore API which lets you query any data that was redirected to an empty scope during the incident window (Oct 20, 10:02AM - 11:15AM PST). The API will help you solve inconsistencies with your players’ data. If you have an existing, active TeamCreate session, you will need to restart Studio before the changes are reflected.

DataStoreService API Documentation

DataStoreService:GetDataFromEmptyScopeDataStoreAsyncTemporary(string dataStoreName, string dataStoreKey)

This is a temporary function allowing developers to read data from the empty scope GlobalDataStore which had writes directed to it between 10:02AM - 11:15AM PST on October 20th 2020. To be explicit, this is a temporary function which will be removed in the future. It has been released as Deprecated and thus does not have documentation on the DevHub.

This function is in practice, equivalent to calling GetAsync(key):


DataStoreService:GetDataStore(dataStoreName, “”):GetAsync(key);

-- note: this is an example and is not currently run-able as-is due to the empty scope input parameter. The new, temporary function that we are providing will allow Developers to get around this validation check.

If data is returned, then the developer can use this information to determine what is the best course of action for this data. When this function is called and it returns nil, the developer can be certain that the key in the specified datastore was not affected during the incident.

If this function throws an error, the error message will describe the problem. Note that existing GetAsync limits will also apply to this function (more on this below).

Reminder: Do keep in mind that this is a temporary function which will cease to operate three months from today. After three months the function will throw an error thus it should be wrapped in a pcall, if called directly. If you use the provided modules you can ignore this, as the module has already wrapped the function in a pcall.

Introducing the DataConsistencyChecker Lua Module

DataConsistencyChecker.rbxm (6.5 KB)

What does this module do?

The provided module is an example we’ve created that a developer could use to programmatically determine if their data was affected by the incident. If it is determined that the data was affected, there are utility functions also included which can be used to correct the existing data.

When to use this module?

The provided code module is intended to be called lazily, e.g. when a player joins a game or a player interacts with the data store. It is not meant to be called as a batch job within Studio, doing so could be further problematic as it is an “expensive” call, and will quickly use up a game’s DataStore allowance.

How does the module work?

The provided module makes some assumptions you should take into account:

  • One OrderedDataStore is used per player.
  • One DataStore is used per player.
  • The key used in the DataStore should match the value from the OrderedDataStore.

The module provides three utility functions:

  1. DataConsistencyChecker::IsDatastoreFixed

    • This function will determine if the data has already been corrected for this key.
  2. DataConsistencyChecker::CheckPlayerData

    • This function will check the OrderedDataStore to find the lost data. If nil, then no data has been lost for the provided key.
    • This is an “expensive” call, as it is by default making 20 requests to DataStores. Be aware of your budgets.
  3. DataConsistencyChecker::FixData

    • This function will correct the data and update the OrderedDataStore as fixed.

Documentation / Example

DataConsistencyChecker DataStores2 Example.lua (3.8 KB)

If you use the module as-is, then the data will go back to the last “good” version for the player. Here is an example of what your data could look like after the script has run:





To make sure that affected developers can sufficiently execute operations to correct their data, we have temporarily increased the limits for the DataStore budget. Please note this increase is temporary and will be reset to the previous limits sometime later next week.

Moving Forward

Ensuring our services, especially critical features such as DataStores, are reliable is our top priority. We are reevaluating and improving current internal workflows around change release management, quality control, and additional automated verification.

On top of that, we are planning to add features and make additional improvements to DataStores which include: versioning, backups, and other tools to manage and protect your data. All of these will have a focus on reliability and quality ensuring you have control of your data. To be clear, this incident is driving internal change and reprioritization of our roadmaps. You will see improvements.

138 Likes

This topic was automatically opened after 17 minutes.

does this mean my system of detecting minutes played is (was) fine?

9 Likes

Thanks for making an update and nice procedures/protocols for us developers on the DataStore incident that happened earlier this week that somewhat destroyed and didn’t save a whole ton of Data among users, it’s nice we got a simple and very easy-to-understand response :slight_smile:

Very nice that new API documentation has been temporarily made for developers to use for reading and quick-scope data for users who play the game, is this just going to be temporary as it suggests or permanently into the DataStore document and Lua functions?

To ensure security and safety precautions for the DataStores’ being created and altered by developers on games that range from big and popular to experimental and well-made, how will we make sure that the Module provided and new API will ensure that this will never happen again in months or weeks?

In the future when more of these incidents and errors happen, will we get a similar update and great response from the engineers on a DataStore related bug in the future as this one? I feel like if any incident happens, it would be nice to have an update on the specific incident.

8 Likes

Seems to be answered in the post:

11 Likes

Keep getting this error everytime I try to use the Module. GetDataFromEmptyScopeDataStoreAsyncTemporary is not enabled getting data from EmptyScope for key 5158

4 Likes

Really cool to see Roblox recognising that a lot of people use a third party module and going out of their way to figure out what issues it would cause in this case, sort of adopting it as one of their own features. Awesome!

10 Likes

Hi ZephsyJ,

Could you try again? We needed to update some additional settings…it should work for you now.

5 Likes

Seems to work now, however is there anything that can be done for players who somehow got over 100 new saves? I’m using DataStore2, and restoring data only seems to work for some players (assuming it’s because it only checks for 20-100 saves?).

1 Like
  • If you happen to have explicitly stored a timestamp in your player data for some reason (such as recording when they last joined), you could binary search back for the most recent entry before the date where the problem occurred using the timestamp (check 1000 back, then check 500 back, then 250 back, etc looking at the date each time). Unfortunately DataStore2 does not store the date by default.

  • You could temporarily offer players an option to explicitly choose how many versions to go back and let them decide.

  • You could manually roll back the individual users if few enough people have too many saves already.

8 Likes

Is there a way in the module I am able to set how far back the roll back starts at? I have a player that already had 100 more saves and I am wondering if I can tweak the code to make it start further back instead of by recent saves.

You can go back multiple pages with OrderedDataStores. The module seems to only do the first page, but you can modify that to search back over multiple. Look at the DevHub for code samples.

Alternatively you could install some plugin for viewing datastores and manually find and restore the save for that user. Just check the diagrams in the first post and do that to the player’s save datastore.

1 Like