Large, unrecoverable increase in network ping

Video is compressing to 10mb:

But here is a Full Quality Video:

Or:

1 Like

I believe the problems are happening server side, so we will need microprofiles from the server rather than the client.

Another option would be to look at the ServerJobs panel and see if you see anything unusual happening there when the problem occurs.

1 Like

Also do you see anything unusual in the server logs when this happens?

1 Like

Nothing at all, again this version of the game from 2 months ago were we didn’t have this issue. (We reverted), ill grab a server side for microprofiler in a second and reupload to YT.

1 Like

This info is unclear to me.

Does this issue still happen if you don’t use the FPS Unlocker?

For other games yes, for our game no.

If I disable the FFflag it goes away (but weirdly comes back after like 20 hours later randomly), but our community members use FPS unlockers so just me disabling it doesn’t have much affect.

Here is the video:

2 Likes

Could you provide a link to the two places where you’re also seeing this issue? Also, can you let us know when you first noticed the issue?

1 Like

The two places we’ve noticed the issue are as follows - unfortunately both are group locked via script so you will likely be unable to join:

We first started noticing the issue within the past couple of weeks - though this might not be exactly when it started as the game is only open for a few days a week at most.

2 Likes

Is this issue about streaming enabled ON and when you keep moving to places that require to be loaded/streamed in eventually it stops to load and your ping increases and your vehicle stops moving/working? Or is this a different issue?

Updated the title here to be slightly more descriptive:

“Weird Streaming Issue?” => “Large, unrecoverable increase in network ping”

Thanks!

1 Like

Update: we are still investigating this issue, which seems to affect multiple experiences. We believe that the use of an FPS unlocker may cause the issue to occur more frequently, but it seems the issue can occur even without the use of an FPS unlocker.

If you are seeing this issue in your experience, please provide:

  • a link to the experience
  • the date when you first noticed the issue
  • any actions or gameplay steps that tend to occur before the issue happens

More details and a narrower time range may help us identify the root cause.

1 Like

If you are talking about FPS unlocker causing issues, are the experiences who reported issues have any code that fires remote on Heartbeat or RenderStepped? Few days ago I had discussion with few people about crash issue, which someone determined that it was caused by them firing remotes every frame more than 60 times because of FPS unlocker, which resulted in Roblox server going up in ping by massive amounts until it crashed. I am not sure how credible that it is, but I thought I would share that it could be a possibility.

I am seeing this issue on a game called “Granny”. Here is the link of the game: Granny - Roblox

I believe this started happening since the 2nd of December or so and the lag is much worse when using an FPS unlocker like the bloxstrap FPS unlocker. The FPS would just keep dropping and the ping gets so high. This happens when doing any interactions in the game for example opening/closing doors, picking or throwing items, putting traps, etc. The ping spikes doesn’t happen when I am idle or in a game menu.

While all of this happens I’ve noticed that the CPU latency is also higher than the usual. It mostly stays at 15-20 ms while the GPU latency is at 5-9ms. The graphics quality is at 1. The CPU latency used to be similar to the GPU latency before. This latency issue happens on all games. Here’s a screenshot:-

PC system information:-
CPU: Intel core i7-1065G7
GPU: Nvidia Geforce MX330
RAM: 8GB DDR4 2667MHZ
OS: Windows 10 64-bit 22H2 build 19045.3803

In my Android mobile phone (Samsung Galaxy M12) the latency issue is much worse. It affects both the CPU and the GPU and lag spikes happens at random times. This also happens in all games and in graphics quality 1. Here’s a screenshot:-

2 Likes

This is also happening in Prior Extinction. Playing on 50ms ping for ~10-20 minutes, then within a second, it’s at 10000ms & counting without explanation / any special pattern.
image

It is 99% related to streaming, as it mostly happens when loading in new terrain & it’s easily testable when teleporting around, thus loading in TONNES of new terrain.

This is a HUGE problem, as we’re going into one of the most profitable / important seasons of the year & our game is largely unplayable due to this issue. I’m sure it’s annoying for the Roblox Staff wanting to go on holidays too. It feels like there’s a new issue with Streaming every week.

It’s a bit annoying that we’re forced to be the guinea pigs & suffer for it every time a new broken update is released. Are these unfinished Beta’s being pushed upon us or somethng? Is there a way to opt-out of being forced into such?

Thank you for your time & hope it gets resolved soon!

  • Jaco
4 Likes

Hi Developers,

We identified the root cause of the ping spikes. Our DDOS/DoS protection mechanisms take the ‘max player count’ in consideration to efficiently allocate server resources among all players. However, based on this post, it seems like this place is configured to support 700 players, whereas it actually is designed for a maximum of 40 players. This discrepancy results in a malfunction of our DOS prevention mechanism, which causes an ‘increased ping time’.

We are implementing targeted temporary measures to address this issue in the impacted places by fine-tuning our mechanisms. We will be implementing a more permanent solution and it will be rolled out soon. If there are any other places experiencing similar high ping issues, please contact us.

7 Likes

Did the temporary fix got rolled out because I am still having ping spike issues with the granny game.

Hiya, could you please confirm whether the two places I listed earlier in the post have had this fix applied to them? We’re currently losing revenue as we’re not able to open the game while this issue is impacting us.

There might be a bigger issue here, first report came in this morning after it didn’t happen at all last night.

image

Yes, the temporary measure was rolled out to that experience. Does the ping spike issue occur when FPS is NOT unlocked? If so, please DM me a copy of the client logs from a client experiencing the ping spike. (see https://en.help.roblox.com/hc/en-us/articles/360016022492-How-to-Retrieve-Client-Logs for instructions on how to retrieve client logs)

If your experience is still seeing high pings, it may be generating more network traffic per second than the server can process, especially if it is sending many remote events. This leads to data ping times increasing continuously, and would need to be resolved by modifying the experience to send fewer remote events per second.

Yes, the temporary measure was rolled out to both of those experience. Does the ping spike issue occur when FPS is NOT unlocked? If so, please DM me a copy of the client logs from a client experiencing the ping spike. (see https://en.help.roblox.com/hc/en-us/articles/360016022492-How-to-Retrieve-Client-Logs for instructions on how to retrieve client logs)

If your experience is still seeing high pings, it may be generating more network traffic per second than the server can process, especially if it is sending many remote events. This leads to data ping times increasing continuously, and would need to be resolved by modifying the experience to send fewer remote events per second.