Reproduction Steps
I do not have a reproduction for this issue as I am not sure what’s causing it. A few hours ago I started getting reports of massive network lag spikes in all servers of my game. After investigating, I cannot determine the cause. I know that it started happening sometime between 1 PM and 3 PM EST today. Vehicles are broken, some players do not replicate their positions properly, and overall network performance is extremely degraded. I have not made any updates since this issue started occurring. My game has been subjected to DoS (Denial of Service) attacks before, and this is nothing like that.
Here’s an example of what it looks like. Notice the player floating nearby and the multiple broken vehicles.
Expected Behavior
I would expect there not to be extreme network related issues that make it annoying for players to enjoy my game.
Actual Behavior
There seems to be quite serious network performance issues.
Issue Area: Engine Issue Type: Performance Impact: Critical Frequency: Constantly Date First Experienced: 2021-07-15 13:07:00 (-04:00)
Is this still occurring? I tried enabling a physics replication change for a subset of users and reverted it after about 45 minutes due to issues I was seeing. Do you have a link to your place?
Even though the timestamps don’t line up with the original report, I believe my change caused the problem. It’s already been reverted but will only take effect once people restart their clients. Would you mind restarting your servers?
We received a wide influx of reports on July 12, 2021 5:50PM EST lasting until around 6-7ish. After that point, it seemed to have settled down but we received a large number of reports again today at 5:57PM EST until present.
This seems to be a large scale issue around most Roblox servers, I have an application that tracks latency, I started receiving reports of latency issues a couple minutes ago:
It could have been later, I started getting most of the reports between 5:45 PM and 6:15 PM I believe. I will restart all servers to see if it alleviates the issue.
Thanks @vikingboy and @Cuyler, I enabled it at 2:18pm and reverted at 3:11pm PST, but because it requires clients to restart it takes a while for the revert to fully take effect.
@FxllenCode That should be completely separate, but thank you for chiming in!
Got it. Can you try restarting the servers one more time? I tried something else.
Edit: Coming back to this to offer a little more explanation. For this type of networking change, clients need to restart for the change to be reverted, but mobile clients don’t restart when joining a new game (only closing and opening the app counts as a restart). So I had to make a change on the server-side before the revert would fully propagate to all the clients.
I only became aware of what was happening about an hour ago and it looks like the issue is already mostly resolved, but I’ve been having serious issues with this on my game in older servers. Players and NPCs(which are server-owned) are constantly hanging for 10 seconds or more, but attacks still register. I would like to point out that I am extremely supportive of changes to physics replication—there are some replication issues that have been crippling certain aspects of my game since I launched it, for example creating explosions near players often leaves tons of sleeping physics parts stuck in the air, so it would be amazing to see replication issues like that get fixed. However, my game has a concurrent player count around 40% lower than what I would expect it to be at this time of day and old servers are completely unplayable. I have not shut down my servers, but new servers don’t seem to be having this issue as far as I can tell. Thank you for responding to this issue so quickly!
At this point restarting should fix any issues with physics replication, I’m incredibly sorry for the interruption to your places! This change was tested and QA’d, but we obviously missed something. I might reach out to you to get some of your models and scripts if I can’t reproduce the issues on my own.
