Summary
On 8 December 2019, at 15:20 UTC, SM Games was notified by several Keyon Air Simulator Community Members that game-pass owners were not receiving their purchased Mod Commands.
After an investigation, we uncovered that Adonis was removing the game-pass from the Mod list on server startup. Our engineers, who were not connected to the live database, could not reproduce this issue.
Root Cause
It emerged that the game’s Adonis database, which was two years old, was corrupted. We believe that the settings were modified to remove the game-pass; resulting in Adonis automatically revoking any permissions attached to the game-pass.
Timeline
15:22 We became aware of ongoing issues with some game-pass holders being unable to use their commands
15:34 Engineers were alerted to the issue and began to investigate. We were able to rule out issues on Roblox’s end immediately.
16:03 We started testing a possible temporary fix to this issue. Mod game-pass owners were given Admin instead of Mod temporarily.
16:09 The above fix was rolled out.
16:17 We identified the issue and narrowed our investigation.
16:23 We determined the root cause of the issue and began testing a fix.
16:44 We believed that we had resolved the issue and began to roll out a patch. We were actively monitoring for any further issues.
17:00 We closed the incident as resolved.
Next Steps
We are taking extra steps to protect Adonis’ database from unauthorized or unintended modifications.
Additionally, we will treat issues like this as being emergencies until we can measure the impact.
Thank you for your patience and understanding while we addressed this issue.
Happy Holidays,
SM Games