As a Roblox developer, it is currently too hard to track and manage which feature flags and experiments my accounts, groups, and games are enrolled into.
Roblox sometimes conducts experiments which affect games in non-obvious ways. For example, there was recently an experiment where some servers were capped running at 10 FPS as a way to save cost. However, the experiment broke a number of games and created a bad day for a lot of developers.
When users are added to “risky” experiments like the one above, they should be notified and have a way to opt out if things break. This is especially important if engineers are offline when things go wrong and we need to turn something off quickly.
If Roblox is able to address this issue, it would improve my development experience because I would have more visibility over what experiments Roblox is running and the ability to opt-out if things break. Developers want to know why service could be inconsistent. Experimenting with our games and breaking things without notice does not respect the community.
It would be really nice to have some sort of menu on the configure page for experiences where we can see what ‘experiments’ are currently being run, with the option to opt out of them if they’re breaking our games, perhaps with a way of sending feedback directly back to the team about why we’re disabling the feature.
It would be nice if developers impacted by AB tests could have some way to see if they need to fix anything or if Roblox is messing around again. Otherwise Roblox is making developers waste a lot of time and effort (money) hunting for bugs and optimization in a panic because of unexpected breakage. Some games update regularly, so they cannot just use analytics to determine if a change was linked to an update, or Roblox changing something. Even if there’s no details, a nudge on analytics that an AB test was enabled for the experience on a specific date would go a long way. You’d get people upset about being enrolled in a test, but that’s the cost of trust and transparency. This is a far more respectful direction to take than what Roblox does currently.
+1 to this. It’s so common to see so many bugs arise from engineers messing with experimental flags that have unintended consequences on some games. The worst part is it can take up to a few days for these flags to get reverted so I think having some transparency on what’s actually going on and the ability to opt out is super important.
, since it may improve update testing, engineers will be able to know bugs in a fast way, and fix them.