Ghost Servers Causing Issues

[focasds]

My game is losing players every time this happens. It goes from 150 to 60, and for sure if I wait longer before taking action it reaches the 0.

Well, what’s the issue?
I have no idea on how this happens, but sometimes I go to the game page and look at my game dropping in players, then I look at my game’s lobby which is in charge of teleporting everyone to different places and I see only one server.

This server doesn’t have any players and stays alive for long periods of time, I have no idea if is a server that crashed or what, but the point is that Roblox doesn’t create a new server for the players who are requesting to join. They seem to teleport them to this server which leaves them on the Loading Screen of Roblox saying “Joining server” and done. It remains stuck there.

This issue will mainly affect small games since there are not hundreds of servers, mainly one active in my game lobby.

Expected behavior

I expect that if a server has any issues like crashing or that if a server becomes empty, it should become unavailable.

[SuperPat77]

Interesting. Can you tell me the PlaceId and the last timestamp when you noticed this happen?

[focasds]

PlaceId: 3144588566
Game Link: The Lost Land [Survival] - Roblox

The two times I have saw it, it was on: (These are the times I noticed the issue and texted about it in my discord server)

• 05/04/2023 16:18 (EST)
• 12/04/2023 13:18 (EST)

[SuperPat77]

Thanks! I was able to track down one of the ghost server instances in our logs - the one on April 5th stayed alive for almost an hour with no players, which is unexpected (empty servers should be shut down immediately). I’ll raise this issue within my team and hopefully we can find the cause

[SuperPat77]

For the April 5th case - did you close the “ghost server” yourself or did it close on its own?

[focasds]

I closed the server myself after I noticed the issue both times.

[KovarMachane]

Maybe I can add information.
The same happens occassionally to our game. The lobby hangs on with 0 players. Nobody can join the lobby, players are getting error "Failed to connect to the Game. (ID = 17: Connection attempt failed.) (Error code: 279).
It lasts until we manually shut down the server.

• Place ID: 1087852616
• Last case: 12 Apr 2023 01:08 UTC until approx 5:45 when I shut it manually
• Previous case: 8 Apr 2023 at about the same time

[Maelstorm_1973]

I play this game (The Lost Land) on my regular account and I’ve seen this happen to me. It will just sit there and do nothing. It’s almost like the hypervisor gets stuck and won’t spin down the VM and leaves the VM in a crashed state.

[SuperPat77]

Thanks for the reports! We found two game servers that were in a bad state and repaired them. If you see this happen again, please let us know - PlaceId and rough timestamp of when you noticed the issue / shut down the game instance

[KovarMachane]

I have withdrawn this comment about a new occurance as my data was probably wrong. I did not expect that a new player can join a place after the last of the players was removed (#game.Players:GetPlayers() = 0), which seems to happen frequently to my place.

[KovarMachane]

Now I have a confirmed case. Same as before, players were unable to join the lobby with error "Failed to connect to the Game. (ID = 17: Connection attempt failed.) (Error code: 279).
According to my in-built http reporting, the place was empty with no player ever have joined. Seems the place started without a player, and did not close. To other players, it appeared live but errored when they tried to join the game.

• Place ID: 1087852616
• Duration: UTC 26 Apr 2023 3:10 am until approx 4:30 am

[SuperPat77]

FYI, if a game server is empty but another player is in the process of joining we won’t shut it down.

I’ll take a look at your other report once the data is available on my end. We’ve been keeping a closer eye on server health over the past week

[SuperPat77]

A couple of new servers had gotten into a bad state and were repaired yesterday. Thanks and please let me know if you see this issue again! Also, if you have it available, posting the server’s jobId will help us find the issue faster in the future

[KovarMachane]

Reporting another 2 cases when my game was inaccessible due to this problem yesterday. Again, it appears that no player had actually joined the place while it reported running empty. It stopped by itself, I did not close servers.

• Place ID: 1087852616

1. duration: UTC 10 May 2023 22:47 pm for at least 2 hours
2. duration: UTC 11 May 2023 1:48 am for approx 10 minuts

Sorry, I did not record the JobId yet. Is there anything I can do in a script to prevent the place hanging empty? Is there a hope that this will be solved permanently? Thank you.

[SuperPat77]

Thanks for the reports. We detected and resolved this issue last night when it occurred.

Unfortunately, I don’t think there’s anything you can do to proactively solve this issue. Given how frequently this has been reoccurring though, I’m planning on adding some short-term mitigations for this in the next couple weeks which will close unjoinable instances automatically. It won’t stop the problem, but it will help us detect and recover faster

[KovarMachane]

We had another 2 cases yesterday, it stopped by itself in the end. Curiously it is the same time of the day again, around 0-4 am UTC, ie. afternoon peak time in USA. I wonder if a malicious attack can be ruled out?

• Place ID: 1087852616

1. duration: UTC 20 May 2023 21:04 pm for 4 minutes
   job id: a6c35128-3797-40ef-aa22-db5f9f181e57
2. duration: UTC 20 May 2023 23:49 pm until 21 May 2:14 am (146 minutes…)
   job id: b19cf37e-91fb-434c-b8fa-739e1fd4d491

[SuperPat77]

Thanks for the report - looking into it

Automatic detection will be here in the next week if all goes well on my end

[SuperPat77]

Instances will now automatically close if they’re empty for too long, even if players are trying to join. “Too long” is currently 5 minutes but that’s subject to change without notice as we monitor the impact of this