When new engine flags or experiments get enabled or disabled on our experiences, please timestamp them and make them available to us via our crash analytics, so we can correlate crashes and incidents for ourselves.
ALL serious developers have war stories of flags being changed internally, as part of A-B groups, and then spending time and money trying to work out if WE broke our games, or something changed on the platform itself.
Right now there isn’t enough transparency on these flag changes.
If roblox would implement this feature, it would build trust and a sense of respect between developers and roblox, save us actual real money and engineering time, and help improve transparency so we have leads when there are ongoing incidents in our businesses.
Request for other devs
Please post your war stories here of when you’ve had flags break your game and you’ve had to spend your own resources trying to track down what would have been visible via this improvement…
I can say for the fishing sim update a timestamped analytics dashboard would have helped a ton!
Scenario A: What Happened
We hit publish on a massive update that went through rounds of flawless testing hours before, went to go test in a live game, saw a game breaking lag bug, spent hours panicked and stressed trying to solve the issue, only to find out later through other studios who had to report it on the forums for engineer to say “we flipped a flag”.
Scenario B: Categorized Flag Dashboard
We hit publish on a massive update that went through rounds of flawless testing hours before, went to go test in a live game, saw a game breaking lag bug, check a flag was flipped via analytics, sees a category of flag such as Physics, Meshes, etc. Therefore 90% chance it is a Roblox flag that we can deal with in a MUCH better way then panik.
We forked the PlayerModule because we need to make changes to many different parts of it. We were not significantly altering the systems, but just making small alterations to allow for enabling/disabling particular features, adding extra logic to polish various aspects of the game (such as things related to the camera subject, user input, movement directions, and VR), and also fixing the occasional bug in the PlayerModule.
Some time after we published this update, one of the flags was removed from the engine which caused an error to be thrown whenever it was queried. This entirely broke our forked PlayerModule and eventually I came up with a system to control flag changes to prevent them from affecting our game in an uncontrolled manner (see Better handling of FFlags in PlayerModule).
The latter bug was pretty easy to diagnose and fix, but it is indicative of a general characteristic of the PlayerModule in particular which is that it is particularly vulnerable to causing issues in games due to it existing in this weird place between the actual engine and the developer’s code. It is possible for a PlayerModule flag to change to cause a subtle systemic bug which cannot be easily attributed, and the same goes for the engine in general as the OP said.
After updating my game early last month, we got tons and tons of reports of players getting randomly flung in various situations. I had no idea what caused this for days because I never touched any of the code responsible for physics and it wasn’t happening before the update.
Turns out Adaptive Physics Step was changed to be opt-out by default without any notification post or release note (that I could find), and therefore publishing the game meant Adaptive physics step was turned on, and broke Humanoid physics. Changing it back to fixed step fixed all of these issues.
Also almost a year ago there was a change to Roblox’s navmesh generation that caused a bug that’d generate walkable areas inside solid walls, causing all zombies in my game to bang their heads against walls in maps that were never changed. I think there was a release note for whatever change caused that, but I’m not sure it was directly related.
I don’t think these are related to the A/B flags thing, but whatever.
In 2023 I worked on a platforming game and ran into ‘war’ scenarios a few too many times.
Case 1: Players falling over
This one affected all games for a brief 10 minute window on September 7th 2023. This was right after I updated my game, causing me to panic. I don’t think this was properly bug-reported, but it still caused a lot of damage.
This one happened during development of my game, so on the bright side it didn’t cause any disruptions to my players, but it did cause issues for many players in other games at the time. About a dozen other developers replied to the related release notes that they experienced problems. If your character jumped onto a moving platform, the platform would slip away from underneath your player, not dragging the player along with it like usual. I still spent a lot of time figuring out why my game suddenly acted differently.
I think this topic highlights another problem that should be worked on. Roblox, as a platform, is very unstable. I understand that you get more and better feedback if you release a feature to everyone at once rather than a small portion. And understandably it can be very difficult to release a bug-free feature - even if you approach it carefully. The whole debacle with the new docking behavior in studio from a year ago is a good example of that.
However, on a platform where the median Roblox developer is very young, you would expect stability to be a priority. On top of that, Roblox developers do not have the option to fix most engine-related problems ourselves, nor do we get paid to solve those issues. We need to spend our own time writing up bug reports which can be very expensive / time consuming.
There was a change last week that stopped weapons from doing combo moves in my live game. It was related to possibly new behavior to Tool.Enabled affecting the firing of Tool.Activated event, preventing swords from doing any combo moves. I spent a few hours digging through our code, with a focus on what code we’ve recently changed. From what I can tell, it was actually an engine change that changed how our existing code functioned.
If there was a flag or engine change timeline, I could have assessed this issue with our experience much quicker.
SST encountered this just last week. We started getting bug reports from our players that avatars were uncontrollably sliding off of slopes, making some levels nearly impossible to beat. I scrambled a bit trying to figure out what I broke, because we had not touched our avatar controller at all in the recent patches we released.
Turns out, Roblox flipped a physics fastflag that broke Roblox’s physics controllers, which in turn broke our slopes. I had to DM an engineer to get the issue raised & fixed. And allegedly the issue returned again, last night…
+1 to this feature request, would really like a feature like this.
