Private Games & Data Stores Incident Report

Summary

Between December 2018 and January 2019, the Roblox platform experienced multiple issues with data consistency:

  • On December 19th and January 8th, many games unexpectedly became private, had their playable device settings modified, or had seemingly random places added to their game without developer action
  • Between January 7th and January 15th, some games experienced data loss due to Data Stores incorrectly returning nil values for existing entries

Both issues were due to an update to an internal caching library that returned incorrect results from the cache under heavy load. The issue has since been patched.

Resolution

For the January 8th incident, games that were unintentionally made private were restored to their correct state.

In response to Data Stores data loss incident, we have provided backups from before any possible data loss occurred.

Private Games Incident Timeline

  • December 19th – Several top games became private suddenly and without developer action. We immediately rolled back the change to prevent further damage. We identified and patched a suspected root cause but were not able to confirm whether the issue was resolved because it was not reproducible.

  • January 8th – Many games unexpectedly became private or have their playable devices changed. At this point we suspected that our earlier patch had not addressed the underlying issue and resumed our investigation.

  • January 14th – Games that were incorrectly made private on January 8th were restored to their correct state.

  • January 15th – The root cause is diagnosed and a patch is released. We begin investigating our options for restoring lost data.

Data Store Incident Timeline

  • January 7th – We started investigating after a developer reported suspected data loss. It was unclear whether this was a game-specific issue.
  • January 10th – We received several more reports of data loss, confirming that this is a Roblox-wide issue.
  • January 14th – Developers are informed of the data loss issue and what Roblox is doing to address it.
  • January 15th – The root cause is diagnosed and a patch is released. We begin investigating our options for restoring lost data.
  • January 24th – We publicize our plan to provide backups prior to any data loss to developers so they can restore lost data.
  • February 12th – Data Store backups become available to all developers.

Key Takeaways

  • We will provide more timely updates about ongoing issues. There were multiple times in the investigation where we failed to publicize relevant information until days later. Going forward, we will strive to close this gap in communication.
  • The process of creating and exposing Data Store backups took significantly more time than we anticipated. We are investigating what we can do to streamline this process should future incidents arise.

Conclusion

We are striving to provide more transparency when it comes to disruptive incidents so developers have more insight into the issues and what we are doing to address them. Thank you for your patience as we deal with issues like this.

127 Likes

Glad to see that staff always got our backs :grin:

11 Likes

I don’t see any mention of places being inexplicably getting added to developers’ games, only that games were being made private unintentionally. What happened on that end?

https://devforum.roblox.com/t/../222724

This incident happened within the same time frame that the private games incident happened.

3 Likes

Regarding those who were affected with Data Loss, how much longer would we need to use “roblox_restore” datastore back ups?

3 Likes

It says it right below.

How long you as a developer need to access these stores between now and July 1st to finish your reverting process is up to you (manual reversion, period of time to merge data, etc). The restoration DataStores will be gone after July 1st.

2 Likes

That issue had the same root cause but I forgot to mention it. I have updated the original post to reflect this.

3 Likes

Phew glad it wasnt a result of something verry bad . However i wasnt affected.

5 Likes

The information being provided is very helpful. Great to see it being made a focus to prevent future issues.

3 Likes

What kind of places were being added to games? Were they just default base plates?

2 Likes

About the games being set to private issue, there is this one game called Boxing Simulator 2 that was affected by the glitch. You can tell it was not the creator due to the group being banned. Can anything be done about this?
game link: https://www.roblox.com/games/987105910/Boxing-Simulator-2#!/about

5 Likes

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.