November 14th Incident Report

Updates Developer Incident Reports
#1

Summary

Between 5:22 pm and 8:23 pm Wednesday PST, opening a game in Studio unintentionally renamed the references to all assets uploaded through the Game Explorer. As a result, games which relied on those image or script assets became non functional. By 7:54 pm Thursday PST, all games were restored to a working state.

Timeline

  • At 5:22 pm Wednesday PST, we enabled a feature flag intended to fix an issue where renaming resources in the Game Explorer did not work. Instead, it inadvertently caused Studio to take every named asset in the game, strip the prefix, and update the name in the web database. At no point was any reference in the place file modified.

  • At 6:54 pm, we received the first developer report about this issue.

  • At 8:05 pm, we received another report, at which point we understood the severity of this issue.

  • At 8:23 pm, we disabled the feature flag to prevent further games from being affected. Then we began work on a solution to correct the affected data in the database.

  • By 7:54 pm Thursday PST, all games were restored to a functional state.

Next Steps

We are adding thorough test coverage to the Game Explorer and other parts of Studio to prevent similar regressions.

We are also going to ensure that posts tagged as ROBLOXCRITICAL (such as the first report we received) are treated as emergencies until we determine the impact.

We were unable to restore asset references which were created between the last database backup and when the feature flag was disabled. This was due to a bug where assets uploaded through the Bulk Import feature were not prefixed, and adding prefixes would have some content nonfunctional. We are still determining what to do about these unprefixed asset references and will notify developers once we have a plan.

Conclusion

We take incidents like this seriously and want you to know we are taking action to ensure they do not happen again.

125 Likes
November 2018 Recap: Keeping Up With the Developer Community
#2

Will there be a similar report as to why the site has been going down for the past 3 Fridays?

(And thank you for the transparency!)

31 Likes
#3

Can’t say this affected me or any of my games, but I do love the transparency of reporting these large scale issues, in great detail.

15 Likes
#4

They lightly addressed that here.

8 Likes
#5

We heavily appreciate the response that was granted to this problem. Its good to see that we get to know what exactly happened, and the process is transparent. Thank you guys, and I really hope you keep this up.

11 Likes
#6

2 posts were merged into an existing topic: (11/16) Joining Games Issue (Resolved)

#7

Thank you, the transparency is very much appreciated.

In regards to this, perhaps it would be prudent to update the guidelines on when to use ROBLOXCRITICAL, and massively clarify what is and what is not a critical issue.
It’s easy to get jaded when many things flagged as critical are decidedly not, which benefits nobody.

8 Likes
#8

The transparency and effort from staff to reach out to us and address our issues within the past week is outstanding - I mean, look, a whole new category for incident reports has been created.

Thank you for the update.

5 Likes
#9

It’s great to see this from Roblox. Thanks for this

6 Likes
#10

They really need too, everything has been down hill lately.

8 Likes
#11

Thanks for the transparency, hope to see more of these incident reports in the future.

6 Likes
#13

Thank you for the full rundown of information. My team nearly had a heart attack.

3 Likes
#14

A post was merged into an existing topic: Off-topic and bump posts

#15

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.