Servers crashing for unknown reason at inconsistent times

Bug Reports Engine Bugs
#1

Experience Link: [⭐NEW RANKS⭐] Police Roleplay Simulator 2 - Roblox
Description: Our game servers are experiencing random server crashes. These crashes occur at random times (sometimes frequent & sometimes infrequent).

These crashes also don’t seem to be happening as a result of hitting the server memory cap. I know this because the crashes occur even when the server is at only 2,000 Mb of memory usage. From my knowledge, crashes due to lack of memory should start to happen when memory is above 6,100 Mb. There are also no errors in the error reports section of our analytics relating to a server-side lack of memory.

When it happens: As mentioned above, these crashes happen completely at random. This problem came to my attention on 20/08/2023 16/08/2023. I believe the crashes started happening since then.

Where it happens: The crashes do not happen in the start place of the experience. The start place is simply a server hub which allows users to connect to the main place; which is where the crashes happen. A link is provided just below.

Place Link of Issue: [⭐NEW RANKS⭐] Police Roleplay Simulator 2 - Roblox

Reproduction Steps: Due to the fact that these crashes happen completely at random. I don’t currently know of a way to reproduce them. The best way to reproduce this would be to join the experience, pick a server & wait until a server crash occurs.

This is the error message players seem to be getting when the server crashes.

772×454 139 KB

I’ve managed to collect additional information about the crashes.

What happens: Every other player begins to walk in place, some players have also reported that the game does get slightly laggy before the crash happens. Here is a screenshot of server stats as the crash happens. On the client end, things seem to run fine.

996×1232 736 KB

Here is the client log file which was generated after being kicked from the server (about 5 minutes after the server crashed / froze. Basically when all players started walking in place). A screenshot of the error message which appears is also linked to my previous reply.

Log/dump files were attached to the internal ticket by a Roblox Staff (@Focia19), due to the sensitive information they may contain, will not be shared publicly!

15 Likes
Abrupt Server Crash Bug
Roblox silently loses connection without notice or rejoin option
Remove AvatarUnificationModel [UPDATED 07.09.2023]
#2

We’ve filed a ticket to our internal database, and we’ll follow up when we have an update!

Thanks for the report!

8 Likes
#3

Just wanted to let you know we are also having this issue with our game as well.

Game - in Question: 🌵 Perris, California - Roblox

Our Solutions:

Reverted to a date where servers lasted (8 hours+)
Result: Crashed at 2 hour 30 minutes

Disabled majority of our scripts
Result: Crashed at 2 hours and roughly 40 minutes

Enabled everything and went back on our latest updated
Result: Some servers crashed at 30 minutes, others crashed at 3 hours.

Our ping in our game also has been recently very high (300+) constantly.
We have used Micro - profiler and script - profiler to see which scripts were most in use and we also disabled those as well with the same result of crashing around the 3 hour mark but are again at very random intervals.

Is there anything that you would need from me to help me out with this issue as these crashing issues are very frustrating with us and our community as well and after reverting the game to the point were servers were fine we realize it could be a Roblox Engine type issue.

Thank you
~ Ash

4 Likes
#4

We are also having this in our game Airship Assault. Players will lag out, everything will appear frozen while the players are still able to move. Here is a video of it happening to one of our players: https://streamable.com/6l5369
This has also been happening to a few other developers who I have contacted. This also started happening around the same time. This matches up with around the time when Roblox seemed to remove Virginia servers and started adding a lot more Singapore servers. He is another bug report that popped up around the same time about a DDoS attack which might’ve actually been this issue: Experience likely targeted by DDOS attacks

5 Likes
#5

Can confirm this is occuring in SCP: Roleplay as well. It appears that UntrackedMemory on the server is unusually high (2GB~) when this occurs.

6 Likes
#6

Hey! We are also having this issue on our game Papers Please. Players will do the same as reported above, still be able to move yet will not see other players moving as if their internet connection has dropped. Are there any solutions to this issue? This has only been a recurring issue in the last >1 month. Thanks

2 Likes
#7

This issue has been occurring for my experience as well.

4 Likes
#8

This is happening in my experience Galaxy as well.

5 Likes
#9

Can you explain what is going on, Does it feel like you almost got bad wifi (but your wifi is fine) ?

2 Likes
#10

Typically within the first 30 minutes of a server being online, all players are kicked out and told to check their internet connection. HTTP requests that are meant to come from the server immediately stop. Typically, if all players left at once the server would still have a chance to make its HTTP requests but the server instantly dies, resulting in all clients losing their connections.

5 Likes
#11

This seems to be true in my experience. I moved the game from a place that wasn’t the start place to the start place of the experience and the servers. Previously, servers lasted 5-10 minutes with the highest anyone saw being 18 minutes. I moved the game 30 minutes ago and it hasn’t crashed since. It could be either related to start place/not or to reserved server/not.

6 Likes
#12

Have you noticed any change in these problems since yesterday 12pm GMT-7?

1 Like
#13

My game, Trench Warfare, started having this exact same issue you describe starting 16/08/2023 too, and I was made aware of it a couple days later. I have spent the past week trying to find a solution without any success (disabling demanding scripts, trying to reduce memory usage, disabling 3D clothing, etc).

There are no warning signs that the server is about to crash, and my players report that it can happen on any gamemode and on any map. A server usually crashes within an hour and sometimes as soon as 20 minutes. I have not been able to intentionally replicate the crashes, but I have been ingame a few times when they randomly happened.

Here is a video one of my players captured showing the crash: https://www.youtube.com/watch?v=4W2PZ_Ue-8E

All other players freeze in place, and their animations continue to play. Any scripts on the client continue to function, but the server is completely non-responsive.

The most puzzling part was that I hadn’t saved or published any changes to my game since 03/08/2023, and everything was perfectly stable for those first two weeks. This made me suspicious that this was a general issue on the platform, not specific to my game.

This is an extremely significant problem affecting player engagement, and I hope it can be fixed as soon as possible.

3 Likes
#14

Its odd how not all games are getting this issue, but both small and quite large ones are however getting hit bad.

3 Likes
#15

Has moving directly to the start place fixed the issue?

#16

Still happening as far as I know

#17

My players are reporting teleporting like behaviour with the most unstable server performance ive seen in some time.
Dont help Brookhaven has nearly 1mill players, and bloxfruit around 400K…

1 Like
#18

I made the change and the crashes stopped for a few hours, but now they’re happening again. My CCU is currently at 4 instead of my typical ~50.

1 Like
#19

Im re attempting a game launch in 10 minutes or so, il feedback server results like ping, etc from players.

1 Like
#20

For me the issues grow as more players load into the game.

4 Likes