Servers crashing extremely fast with no changes to the experience

[milk74l8O]

Reproduction Steps
An increase in server crashing (Around 9-20 minutes)

Direct links to games that this occurs in:

Greenville

Greenville Beta Final Release

This has been happening since November 10th, at around 3 PM CST.

We have not updated anything in the past 3 days (November 8th was the last time) to the experience and this just popped up on the 10th around that that time.

System Information: Windows 11 Gaming PC with around 40 MBPS download ethernet connection stable.

Expected Behavior
When a full server occurs/server fill happens, server memory goes up way more than usual for us versus before.
This causes a crash to occur.

(We have a higher memory start-out count than most games which can cause crashes earlier/work on code is needed, but we have never had crashes in 9-20 minutes ever, rather it usually occurs around the 2 hour+ mark and not 9-20 minutes.)

Actual Behavior
Possibly untracked memory could be causing this? I’m not sure exactly on what category/what exactly is causing this and if there is a way to see that would be helpful as to what’s causing the crashes as we have not changed anything in the past few days and this popped up randomly to the extremity of this server crashing.

(This is when the server crashed)

Issue Area: Engine
Issue Type: Connectivity
Impact: Very High
Frequency: Constantly
Date First Experienced: 2021-11-10 15:11:00 (-06:00)
Date Last Experienced: 2021-11-11 17:11:00 (-06:00)
A private message is associated with this bug report

[asdfiji123]

Hi there - I’m so sorry to hear that this is happening. I just turned off a memory-heavy performance optimization that was introduced at around the same time for the two places you mentioned. Please let me know if you continue to see crashes with the same frequency.

[milk74l8O]

Will do so if this occurs again, thank you!

[Heromoreo (nil)]

I’ve been having an issue that has been happening around the same time as this bug but on Roblox studio and it may have some kind of relationship to it. It’s been really frustrating me, and I don’t have anywhere else to put this.

I can’t open any of my games in studio without getting this error:

Sorry, this published place could not be loaded. Details: “HttpError: ConnectFail, curlCode:7, curlMsg:Failed to connect to c5.rbxcdn.com port 443: Timed out”

Incident ID: 7845886104361281715

If you continue to encounter this error, please report it on the Developer Forum

and even if it does load, it’s terribly slow and none of the Roblox materials load, most of the online assets wont load either. after about 30 seconds to a minute Roblox studio freezes.

when trying to open a game offline, no internet connection. Roblox studio doesn’t even open and gives me an error.

I can’t create any experiences either. none of the template or my game icons will load, appears with the N/A symbol.

I have three computers that I’ve tried using studio on. Same result on all of them, even connected to a hotspot on my iPhone on only cellular or another wireless or wired network.

Desktop:
Windows 10
28 gigs of ram
core i5 (2017)

Laptop:
Surface book 3
windows 11
8 gigs of ram
core i5 (2020)

Laptop 2:
windows 10
16 gigs of ram
core i5 (2015)

Here are some images:

The green baseplate is supposed to have a Roblox grass material.

Roblox Studio trying to start offline

Roblox Templates with missing icons

[asdfiji123]

Hi - the change that I was talking about is a change that was made on the Game Engine, and an optimization that impacts Server/Client efficiency. I don’t think this should have any impact on being able to load places in Studio.

I will post your concern internally and try to have the right person take a look. Thank you for reporting, and sorry to hear about this

[alphavoyager]

Hi asdfiji123,

Suspect there may be connection issue to c5.rbxcdn.com. Appreciate if you can send ping output to c5.rbxcdn.com.

on command prompt, please run “ping c5.rbxcdn.com”

Also, do let me know if you are using VPN to connect.

Thanks

[Heromoreo (nil)]

Thanks, asdfigi123, you’re the best!

hey alphavoyager, I’ve got the results here, image below. No, I don’t use a VPN.

[BetterNotDie]

This issue has also been plaguing our two two main experiences since the 10th, world conquest and world conquest group. Before the 10th, our servers would typically last a good few hours at least, or up to a day. Now they’re barely lasting an hour, and typically less before they crash.

World Conquest [READ DESC]

NEW GROUP

For us the cause seems to be a section of core memory labelled as network/replicationCoalescing - I assume this is related to the memory optimization mentioned above. It gradually rises to around 3000 MB of memory usage over the life of the server, then shortly causes the server to crash once it reaches those levels. Like the one who reported this bug, our experiences start out with relatively high memory usage because of the amount of parts in the experience.

Although the experiences haven’t been updated since October, I’d originally asked about this in help and feedback because I wasn’t sure whether it was something I’d caused, or an exploit (and also because I don’t have permission to report bugs), but I guess it’s clearly a bug.

Below: Memory usage of network/replicationCoalescing shortly before server crash

Below: Total server memory usage shortly before server crash

[asdfiji123]

Sorry to hear that we’ve now disabled the change globally, which we think should resolve this issue for your game. Please let us know if it persists.

[alphavoyager]

Hi WellsElectronic,

It seems you have packet drops, that could be an issue. If you can provide us your IP address, we can look into more.

you can find your IP address by going to whatismyip.com

Thanks!

[Ryrasil (Sem)]

i have the same issue
my ip: 192.168.1.1

[iceman_137]

That is your private, internal address. You would need to provide the address from whatismyip.com as stated above for your public, internet facing address.

[Heromoreo (nil)]

Edit: My issue has gone away as of 8:40 pm CST, 11/12/2021. Everything is loading, including icons to games, Roblox materials, and the game itself with no crashing.
All right alphavoyager, I’ll send my address as a private message. I don’t want to accidently leak anything private. If necessary, should my issue be made into a new topic as I’m somewhat off this topic?

[alphavoyager]

Thanks Camera_2 for providing additional information. Good to hear the problem has been resolved.
We checked and didn’t have any issue on our side that can explain slow loading that you were observing.

[thirdtakeonit]

Closing per above comment. If this issue starts recurring in the future, please submit a new bug report.