Game Ping Reaches 3k~80k after a Roblox engine Update

[TimTsuki]

After a roblox update the 28/3/2023 my game: Steven Universe Future: Era 3 RP
Started having an issue i’ve had before (12/12/2022) that ROBLOX fixed randomly after 1 month (12/01/2023) . This issue consist in getting the whole game EXTREMELY LAGGY out of thin air, no update was done to maybe trigger this “EXCESIVE LAG” the 28/3, the game was 100% functional before this started happening.
We get excessive random pop-up crashes messages, is not consistent within the message, every player that is showing me that they can’t enter the game, gets this variety of errors: (Included some of mine as well.)

Those errors are all for trying to join the game, crashing your character making it so if you rejoin of the never-ending loading screen you get the message “Same account launched from different device” which is not true, is just your last attemp of trying to join the game.

these high ping occur randomly, nothing triggers it apparently since nothing pop up, i’ve checked if it’s a memory leak but nothing lends to that it may be.

As i repeat, this started happening after a roblox update in the engine around the 28/3
I’ve tried my game in a very old update, (which was functional indeed)
And this issue continued happening even tho NO change was made to that version.
(This is a very old version from september 28th 2022)
I’ve gathered my players in an old version, yet still happens (This version is no connected whatsoever to the main game, it was published in a place aside)

Proof of server freezing in an old version of the Game.

Proof 2 of server freezing in old version of the game.

The issue still persist in the main game and latest version, recalling that started happening after a roblox update, i haven’t updated anything on my end to cause this.

Reproduction Steps are INCONSISTENT

-Join the game, you might get the infinite loading, you may not.
It only happens if you get to join the game while a delay is going on (incredibly huge ping)

-Play for a while, everyone may frezze, the best way to know that this is going on is that the ping is super high, normal is 200~300 but it may reach 700~80k, while on this you clearly cannot interact with anything, all other players would be walking in place and everything is buggeable since the server is not responding.

proof of game crashing, loading & ping recorded in 5 minutes

This video represents the reproduction steps. take a good look in the ping, loading time & that i went afk leaving this recording so this is in real time. the game ends crashing after 7 minutes (real time, i had to crop)

A Workaround
A workaround was to lower the servers from 50 to 35. this KIND OF smoothered the intervals beetwen delays (high pings)

And proof that the game was capable of having 50 players in one same server without major issues.

Proof that the game were able to take 50 players in one server Without any major issues

And here are two microfile dumps!

microprofile-20230422-234410.html

microprofile-20230423-222415.html

Thanks for you beforehand.

[Focia19]

We’ve filed a ticket into our internal database for this issue, and we will update you when we have further information!

Thanks for the report!

[TimTsuki]

Thank you so much!
Dealing with this issue have been a big problem.

[TimTsuki]

And something to add is to try reproducing the bug in the “Homeworld Servers” to reach there you must enter the game > Pick a character > scroll in the teleport interface to “Homeworld” and then it takes you to other place of the game where the ping is highly unstable, in both Earth and Homeworld they are unstable, but for some reason, they are more unstable in Homeworld. And another thing to add is that in vip servers with not so much people, it wont lag (2-6 people)

[SuplexTheTrain]

The microprofiles you’ve provided appear to be from game clients. Is it possible for you to generate a server microprofile?

This can be done via the developer console.

[fishrotR]

Hi
When you say “i’ve had before (12/12/2022) that ROBLOX fixed randomly after 1 month (12/01/2023)”, do you still have the link to devforum thread where the previous issue was reported?
We think this could be a issue with underperforming server script or maybe in combination with some engine issues, since we don’t see the same issue with any other games with a lot of players. We are trying to investigate what the root cause is, but are you also able to try to find if there is any code that could be extremely slow as number of players increase?

FishrotR

[TimTsuki]

Sure thing, i will get in-game and get a correct server microprofile, is my first time doing this sorry for that!!

Here i took 3 microfiler dumps just in case, pings of 12k - 3k -5k happened at the time.
log_671C6_microprofile_20230425-190812.html (292.3 KB)
log_671C6_microprofile_20230425-190732.html (295.7 KB)
log_671C6_microprofile_20230425-190643.html (290.4 KB)

[fishrotR]

Hi @TimTsuki, could you try changing “Seconds to Record” to 30 seconds so that we could see multiple frames in one dump.
And could you try one dump with Frames Per Second being 30 and Seconds to Record being 10 as well?

[TimTsuki]

Hello! i am not extremely sure of which the real problem is, i just have been into thousands of different issues that had to do with lagging the game itself.
I am not 100% sure of which of the nexts have been my issue.
Here you go:

On player load PhysicsParts memory increases permanently, resulting in crashes and severe lag

Reproduction Steps How to reproduce: Create a fresh baseplate. Look at PhysicsParts and Instances Server memory usage. Reset a couple times. Look at memory usage again. Issue has been occurring for about a year or so now. We did not isolate the problem until today after extensive testing. Our game is called SCP: Site Roleplay. It is a combat heavy game with lots of death. This bug causes servers to grow extremely unstable as they age and more players load/respawn. This eventually leads to…

Game doesnt lag in studio, but lag spikes occur in game

Occurring randomly, the game gets lag spikes to around 50K ping per person for a few seconds, and then comes back down. I used the script profiler to get some information, [image] [image] I fixed these leaks and the game still had lag spikes. Here’s a video of the game’s script performance tab. Game link. And a 60-second micro profile. log_BA5FD_microprofile_20221227-125833.html (3.1 MB)

Post from when the first one started happening:

The Game Freezes after a Roblox Update

What do you want to achieve? I want to fix this high ping issue, game is unplayable. What is the issue? The game gets 50k+ ping variating from 2k to 50k, The game’s common ping is of 200 but is still playable. Its common ping is like that because this is a morph game, which all characters have more parts and trices than a robloxian. (Need to clarify that the high ping comes for like 5 minutes and then it leaves for a very short period and then comes back, this is a loop.) [image] [im…

This is when it started happening again:

Experience is Unplayable after a ROBLOX Update

You can write your topic however you want, but you need to answer these questions: What do you want to achieve? I want my players to be able to play me game in peace as they used to. What is the issue? After a roblox update the 28/3/2023 my game: Steven Universe Future: Era 3 RP Started having an issue i’ve had before (12/12/2022) that ROBLOX fixed randomly after 1 month (12/01/2023) . This issue consist in getting the whole game EXTREMELY LAGGY out of thin air, no update was done to ma…

Sadly i haven’t found when the past issue was fixed or how it was fixed, these are the only post where i’ve been into to check on this bug, but we didn’t know what to do.

[TimTsuki]

sure! let me do it right now, i have to wait until a high ping is going on so i might take a bit.

[TimTsuki]

I’ve tried changing it to what you’ve requested me but it doesn’t want to change.
@fishrotR

[fishrotR]

OK. What about 1 fps with 30 seconds?
10 fps with 4 seconds might also be OK.

[TimTsuki]

Here you go:
30 Frames / 4 sec
log_671C6_microprofile_20230425-193236.html (1.7 MB)

10 Frames / 4 sec
log_671C6_microprofile_20230425-193336.html (741.9 KB)

[fishrotR]

Hmm, server frame rate looks pretty good from the dump. Were you still seeing high pings around the time you did microprofiler captures?

[TimTsuki]

Yep, it was around 3k, 5k but they lasted very shortly. i will go and check for a long one with most high ping that occurs even when testing in an empty baseplate only with morphs & their features and 50 players servers.

[fishrotR]

Yea, a microprofiler dump with constant long pings will be very helpful.

[TimTsuki]

I am waiting for a long ping but this server with 50 players haven’t triggered one yet and im surprised, have you guys changed something?

[TimTsuki]

Nevermind. as long as i said that it started happening.
log_671C6_microprofile_20230425-194123.html (233.7 KB)

This is the only microprofile that took longer since it froze and took some time saving it-
log_671C6_microprofile_20230425-194254.html (2.7 MB)

log_671C6_microprofile_20230425-194430.html (233.8 KB)

EDIT: i added one more just in case.

EDIT 2:

log_671C6_microprofile_20230425-195103.html (903.0 KB)
IT took so long to record because even the engine frooze and didn’t record because it took a lot of time to stabilize for just a milisecond-

[fishrotR]

Hi @TimTsuki ,
Our investigation shows that the garbage collections take really long time in some server frames, which slowed down the server frame rate, resulting in long client pings.
As this issue is specific to your game, we think it might have something to do with how this game works. If a lot of instances are created and destroyed/dereferenced, and if this happens a lot, there will be a constant high cost on garbage collection.
Could you do some initial investigation on your lua script and see whether there is any code could create excessive number of instances on server.
The developer console could also show the number of instances right now in memory.

fishrotR

[WheretIB]

The engine issue that caused this has been fixed.
It affected experiences with a large number of server script instances.