On October 20 at 10:02AM PST, we released a game server update (v 453) which caused issues when requesting GlobalDataStore through GetDataStores. All scopes of GetDataStore, regardless of value, were changed to “(Empty String)” which meant that all requests made through a GlobalDataStore on the affected servers completed but were redirected to the datastore with no scope.
The version has been rolled back and restored, and now current data is being read/written correctly. If you were only using standard DataStores (not linked with OrderedDataStores) for storing player data, you are most likely not affected.
However, If you use OrderedDataStores to point to data in a GlobalDataStore (such as the default save function from DataStore2), this data could have been written to an empty scope resulting in data mismatches. Your players’ data will be in one of the following states:
- Unaffected, if their data was not touched
- The current OrderedDataStores pointer is referring to the incorrect DataStores key. There are two potential scenarios:
- Scenario 1: The player has an previous save and may wish to restore the previous save
- Scenario 2: The player started a new save and may wish to continue the new save
The diagram below shows an illustration of the potential states of the regular Datastores (of all player data keys pointed to by the OrderedDataStore):
New API & Next Steps
We have released a temporary DataStore API which lets you query any data that was redirected to an empty scope during the incident window (Oct 20, 10:02AM - 11:15AM PST). The API will help you solve inconsistencies with your players’ data. If you have an existing, active TeamCreate session, you will need to restart Studio before the changes are reflected.
DataStoreService API Documentation
DataStoreService:GetDataFromEmptyScopeDataStoreAsyncTemporary(string dataStoreName, string dataStoreKey)
This is a temporary function allowing developers to read data from the empty scope GlobalDataStore which had writes directed to it between 10:02AM - 11:15AM PST on October 20th 2020. To be explicit, this is a temporary function which will be removed in the future. It has been released as Deprecated and thus does not have documentation on the DevHub.
This function is in practice, equivalent to calling GetAsync(key):
DataStoreService:GetDataStore(dataStoreName, “”):GetAsync(key);
-- note: this is an example and is not currently run-able as-is due to the empty scope input parameter. The new, temporary function that we are providing will allow Developers to get around this validation check.
If data is returned, then the developer can use this information to determine what is the best course of action for this data. When this function is called and it returns nil, the developer can be certain that the key in the specified datastore was not affected during the incident.
If this function throws an error, the error message will describe the problem. Note that existing GetAsync limits will also apply to this function (more on this below).
Reminder: Do keep in mind that this is a temporary function which will cease to operate three months from today. After three months the function will throw an error thus it should be wrapped in a pcall, if called directly. If you use the provided modules you can ignore this, as the module has already wrapped the function in a pcall.
Introducing the DataConsistencyChecker Lua Module
DataConsistencyChecker.rbxm (6.5 KB)
What does this module do?
The provided module is an example we’ve created that a developer could use to programmatically determine if their data was affected by the incident. If it is determined that the data was affected, there are utility functions also included which can be used to correct the existing data.
When to use this module?
The provided code module is intended to be called lazily, e.g. when a player joins a game or a player interacts with the data store. It is not meant to be called as a batch job within Studio, doing so could be further problematic as it is an “expensive” call, and will quickly use up a game’s DataStore allowance.
How does the module work?
The provided module makes some assumptions you should take into account:
- One
OrderedDataStoreis used per player. - One
DataStoreis used per player. - The key used in the
DataStoreshould match the value from theOrderedDataStore.
The module provides three utility functions:
-
DataConsistencyChecker::IsDatastoreFixed- This function will determine if the data has already been corrected for this key.
-
DataConsistencyChecker::CheckPlayerData- This function will check the OrderedDataStore to find the lost data. If nil, then no data has been lost for the provided key.
- This is an “expensive” call, as it is by default making 20 requests to DataStores. Be aware of your budgets.
-
DataConsistencyChecker::FixData- This function will correct the data and update the OrderedDataStore as fixed.
Documentation / Example
DataConsistencyChecker DataStores2 Example.lua (3.8 KB)
If you use the module as-is, then the data will go back to the last “good” version for the player. Here is an example of what your data could look like after the script has run:
To make sure that affected developers can sufficiently execute operations to correct their data, we have temporarily increased the limits for the DataStore budget. Please note this increase is temporary and will be reset to the previous limits sometime later next week.
Moving Forward
Ensuring our services, especially critical features such as DataStores, are reliable is our top priority. We are reevaluating and improving current internal workflows around change release management, quality control, and additional automated verification.
On top of that, we are planning to add features and make additional improvements to DataStores which include: versioning, backups, and other tools to manage and protect your data. All of these will have a focus on reliability and quality ensuring you have control of your data. To be clear, this incident is driving internal change and reprioritization of our roadmaps. You will see improvements.
