Data Stores Data Loss Issue

[ChilledCW]

Loving the fast response from the team

[railworks2]

Thank you for very much for this update. I can’t wait for the incident report and hope this will never occur again, although may remain possible.

[Extuls]

@Seranok

Now that the immediate issue is resolved, we are working on a remediation plan in order to help developers restore lost data. We will provide more information in the next day or two once the plan is finalized.

This is great. I highly agree with @Rocky28447, this response was much needed and I’m happy to hear there are plans to restore lost data.

[Om4rr17 (Om4r)]

I have an idea in mind that can solve this problem (detecting data loss) that I will be using… Since the game is paid access, thus not millions play/buy it so in your case it is about 14K yes I check Vesteria daily, saving each player name upon purchasing the access and joining (as getting the purchasers from the revenue tab would not work) and saving them external (maybe google sheets) will keep you with a list of every player that has his data stored… Adding a function that gets triggered if the player is on the list if his data store is nil would let you be able to inform the players to sign out immediately (or just kick them )
The reason why not to get the players from the revenue tab is that ones who bought it but didn’t play it yet will receive the error due to nil data store.
I hope I helped you with this idea! Sorry if there was any typos due to thr fact that I am on phone…

[SteadyOn]

I appreciate the transparency! Is there any information on how widespread/ serious this was and how we can tell if our games have been affected? I’m assuming this will be in the incident report, but any basic information you could give would be helpful to prepare us for attempting to fix the data loss issues

[xelph_rs (xelph)]

Thank you for letting us know, I love it when you are transparent about issues.

[Seranok]

@SteadyOn

I appreciate the transparency! Is there any information on how widespread/ serious this was and how we can tell if our games have been affected? I’m assuming this will be in the incident report, but any basic information you could give would be helpful to prepare us for attempting to fix the data loss issues

We do not have precise information on which games were impacted. To determine if your game was affected, check for reports of data loss in any communication channels you have with players (e.g. private messages, Discord Channels, group walls). If you keep logs of Data Store requests, you may also be able to determine how many players in your game were impacted by seeing how many Data Store entries were wiped.

[JustSeanC (Sean)]

No incident report yet?

[Seranok]

Hi everybody,

In order to help you restore lost player data, we are going to make a full backup of data stores from prior to any data loss. Because players joined the game after data loss occurred, there is no way for us to automatically resolve conflicts so we are putting full control in your hands.

We will let you know as soon as this backup is available. The earliest possible date for data loss is December 19th, so our backup is from the 18th.

This backup will be exposed through the same Data Stores API that you are accustomed to. In order to access backup data, you will need to prefix your data store name with “roblox_restore_181218_” as shown below:

local dataStoreName = "ExampleDataStore"
local dataStore = game:GetService("DataStoreService"):GetDataStore(dataStoreName)
local backupDataStore = game:GetService("DataStoreService"):GetDataStore("roblox_restore_181218_" .. dataStoreName)

Attempting to write data to backup data stores will throw a 403 error. The backup data stores will be available until July 1st, at which point they will start returning nil values.

You have a few general strategies to restore data:

• Give players the option to restore from backup
• Manually restore backup data for players who have reported data loss
• Write code to lazily restore backup data when players join the game

We recommend you go with whichever strategy makes the most sense to you. Please let us know if you have any concerns. Next week we will be posting a full incident report explaining how the data loss occurred and what we are doing to prevent future incidents.

[colbert2677 (ImagineerColbert)]

I feel like the best course of action personally would be to add up both data from current player sessions and those who lost data by making a data viewer (e.g. players can access a tab that shows all the data that they previously had). This way, players can send in screenshots or select “merge data” options in order to retrieve lost stats. I’d handle automatic restorations with utmost caution, in the instance that things don’t work out and complications extend.

The support and effort that the engineers have gone through to assist us developers with this detrimental issue is graciously appreciated and I’m sure that the players as well, not just developers, are thankful that an appropriate solution has been reached. I’m looking forward to seeing the Incident Report regarding the data store failure and the push to prevent future issues like this.

All in all, I can’t remember when exactly the Incident Reports category opened but I am grateful that Roblox is providing as much transparency as they can regarding major issues. It’s helpful to be in the know-how of certain situations and how we can adapt or contribute (through Platform Feedback) in the future.

[c0mmandhat (Commandhat)]

I’ve signal-boosted this onto the roblox subreddit; just so people are aware what’s happening with data files, and why they may see new, unfamiliar prompts and such about data.

I might also code a quick module to allow users to choose individual save files (as defined by the developers) to restore, I’m not sure.

[hoontee]

How do we restore from backups of GlobalDataStore?

[hoontee]

I still can’t find any way to restore from GlobalDataStore backups. Is this even currently possible?

[Seranok]

We have not made the backups available yet – we will keep you posted.

[Polymorphic]

@Seranok

Next week we will be posting a full incident report explaining how the data loss occurred and what we are doing to prevent future incidents.

Is this still happening?

[Seranok]

Hi developers,

The backups are finally live! See my previous post for detailed instructions on how to access them. Keep in mind that we will be removing the backups on July 1st.

Edit: There is an issue where the first few times a key is retrieved, it may throw an error instead of returning the value. We are investigating what can be done about this.

We are still planning to make a publicly available incident report to give you more insight into what happened and what we are doing about it.

[NeloBlivion (Nelo)]

@Seranok

The earliest possible date for data loss is December 19th, so our backup is from the 18th.

Sorry if i’m reading too much into this/asking an obvious question, but does this mean backups are only available for the 18th of December?

[Seranok]

That is accurate. We considered providing more frequent backups but there is a large financial cost per backup and we did not think it was worthwhile.

[Seranok]

Hi Developers,

We have decided to remove the Data Store backups on Monday April 29th. After that date, any attempts to retrieve backup data will return nil values. If you have not already done so, please remove code that reads from the backups from your game as it will no longer be useful.