It does appear that the ping in a server will eventually return to normal, but likely not for long.
DataSendKbps doesn’t appear to be at any high number (<1 mb/s), though our monitoring is also affected by this issue so I’m unsure if what I said is true.
There haven’t been any updates that would create this issue. The last few weeks have been minor patches to scripts.
I diffed version 500 (from the 2nd of October) and 518 (latest), which confirms this.
@catinpsyop Thanks for reporting this issue. This might be due to a recent change that we rolled out on server-side. Do you happen to have any repro for this issue?
The rollback for the change for your experience ([MTC] Multicrew Tank Combat - Roblox) is in progress. Please watch out for new servers, if this change was the issue, I would expect that the new servers will not have the regression.
@catinpsyop Great! Thanks for confirming. We will work on identifying the issue and resolve it before rolling it out again. In the meanwhile, if you happened to create a repro for this issue, please share it with us either here or via a private thread. We will notify you once we wanted to rollout this change again in the future once we addressed the issue.
This high frame time can last for minutes, I find it unlikely to be on-join replication.
This has come to my attention due to the fact that remote events are being dropped (or delayed to the point it’s interpreted as dropped)
Recent updates to our game essentially made the above visible to the users…
We have a system that exports server data (heartbeat ms… average latency) every 15s to a timeseries database (think of Clickhouse, InfluxDB). Sometimes these exports are e.g. minutes apart which is our only external indicator of this extremely high server frame time. These gaps between exports have been going on for as long as our data is retained (which is ~30 days)
I’ve looked into extended services for CPU, but that’d cost us >2,000 USD a month so it’s just a non-option.
