New servers loading up old place versions

[Kensai666]

Yesterday around 5/22/2023 8PM EST, multiple places under the Experience World//Zero were loading up servers that seemed to be many version histories behind.

From what we could tell, these places were set back to a version somewhere in ~2021, while no changes were visible in the Version History page.

When opening these places in Studio, they seemed to be the latest version. Publishing the places and shutting down the servers seems to resolve the issue.

In the screenshot below, you can see that no changes were made to the affected place until we republished today. (5/23/2023)

I also want to clarify that we do not use packages for our games, if that helps narrow down the issue.

[SuperPat77]

Interesting. Thanks for the report - we’ll look into it

[Tomi1231 (Chilly28)]

I had a similar issue with my game, yesterday between probably 6 pm and 8 pm (est time), the starting place of my game reverted to an early 2021 version. (People were on the game at the time and the hub/starting place for them was fine, I was the first to notice it at 8 pm)
The game in question: https://www.roblox.com/games/6152945654/BloxBox-Experimental-0-05-M

I know the starting place was affected. The test world place was not, I did not check if the template place was affected (the two other places)
The game also has a bunch of places made with CreatePlaceAsync(), they don’t seem affected

[Hexadecagons (Hex)]

The exact same behavior has affected places within two of the games that I’m a developer for. The games in question followed by PlaceIds that were affected:

Notoriety
1231303427
1257393236

Wings of Glory
2074734339

In Notoriety’s case, we have a dozen different places, but only two places in particular seems to have been affected at seemingly random.

Notoriety does use packages while Wings of Glory does not, which could indicate that their usage is irrelevant to the issue.

[SuperPat77]

Thanks for the reports everyone. We’ve identified the issue and new game starts shouldn’t use old versions anymore. We’re working to find and close all game instances that were started on the old versions

[Brick_man]

Thank you! So no action is required on our ends to resolve this? (Place Ids are mentioned in @Hexadecagons 's post noted for Notoriety)

We have a pending update that’s not ready yet, so hopefully this won’t require a publish from us.

[SuperPat77]

It will take us some time to find and close the impacted game instances. If you’d like to recover faster, please use Migrate to Latest Update to shut down the instances running old versions yourself. You should be able to use it without publishing a new version to restore the “good” latest version

[AbstractAlex]

Can you share any details on what triggered the bug?

What would cause a 2021 version of places be sometimes loaded? Is that like a datacenter that was behaving weirdly?

Mainly just curious because of how obscure of a bug this is!

[GoodNameOnly (Hashtags)]

Hello, I don’t know if this is related or just a coincidence, but about an hour ago all servers hosted in Los Angeles got killed, I was in one myself and got disconnection error 277. The version of the game is correct

is this problem related to killing of glitched servers or just a coincidence with time, or is it intentional

[SuperPat77]

We’ve found and manually closed the last remaining game instances that were running old versions. If you still see any instances running old versions of your games please let us know the PlaceId. Thanks!

[SuperPat77]

I’ve asked the team if they want to share a mini post-mortem but it requires quite a few internal details to understand so I’m not sure they’ll be able to

[sweeteggtart]

Hi there!

Status:

• 1PM: We rolled out a fix for this issue (so no new game launches should have loaded old versions).
• 4PM: We identified all servers running old versions and shut them down. @GoodNameOnly since we ran this at 4PM I don’t think those disconnections were caused by our efforts to mitigate this issue .