Moments of extreme latency and Networking Data being sent

I would like to start by saying, I do believe this is a repost of the following post. Due to the nature of the DevForums, I am unable to attach private profiling data in comments of DevForum posts.

It is entirely possible these two issues are not coorelated, however they do seem very similiar. I was hoping maybe further insight could be given via my post, alongside the what the profiler is showing (Once again, I cannot send my profiling in the other post)

For the past week or two, I have noticed an extreme increase in server bandwidth, seemingly at random times, so much so that users will ā€˜spike’ to over 1000/ms, I have provided numerous screenshots below.



I have provided additional profiles in the private section attached to this message.

A private message is associated with this bug report

112 Likes

No response is seemingly very demotivating. This is effectively ruining a ton of communities, and any game which is making an attempt at maintaining over a hundred players.

This has nothing to do with Streaming Enabled, numerous tests have proven as much. This issue seemingly occurs periodically.

Additionally I’ve done profiling of my own, we have monitored our networking. We’re seemingly only sending around 300KiBs over the span of a minute. This isn’t the greatest, but it’s far from the spikes of nearly 10,000KiBs we have seen at times.

This seemingly has begun being an issue towards two weeks ago, the middle to end of last month. Unsure if any update or changes were pushed then.

We’ve noticed that players seemingly all freeze, and then after a moment they catch back up. As though they were spiking in ping. Which matches what we’re seeing in the networking being sent from the server. Except no such networking is occuring, we are not utilizing any form of mass-events, etc.. As I mentioned above, our average KiBs/per minute is around 300KiBS; we see spikes up to 10,000.

56 Likes

Bump.

This happened to our game as well

We ran 300 player servers fine 2-3 weeks ago.

Now they are almost impossible, 60k ping server took 20 minutes to recover.

This was 200 players + tons of AI + thousands of projectiles running between 300-800 server data ping MAX. It was PLAYABLE

Now it’s not. The above video is roughly 4 weeks ago. So something has clearly changed.

1 Like

@SuperPat77 Again, this issue is not only related to our game but many other games with a large community, could you please take a deeper look into this issue?

This issue seemingly is split in to two parts. One where players begin to become unresponsive, and choppy. In the sense of they freeze in place, with no rhyme or reason behind it.

The second part is what’s described elsewhere. Which is that players begin to experience extreme ping-spikes. Where server kb/s goes upwards of 10,000KiBs

You can see in the video provided, the Recv is seemingly low, with 24/ms Network Ping; yet the physics replication as a whole seems to be grinded to a halt.


The alarming part, is this latency seems to differ from everyone? I may see players teleporrting, and others don’t.

1 Like

Yeah I noticed the exact same thing at my event yesterday. It definitely performed better than the initial onset of the lag a few weeks ago but it was extremely variable. I would have 60ms and another could have 300ms or up to 2000ms in some cases.

I have gone ahead and added additional profiling.

Yet continued lack of comment from Roblox; hopefully they see this issue & have a solution in store.

3 Likes

Bump, the lag is so bad this need to be resolved.

It spirals into 20k+ ping for no reason.

2 Likes

cc’ing @regdorf and @asdfiji123; they seemed to of been some assistance for others, I was hoping they could take a look here, as this issue is still one in which is prominent and growing. Much appreciate for all you guys do :slight_smile:

Hey guys - could you include links to games that are experiencing this problem so we can try to join and repro ourselves as well?

Our game has been experiencing this since mid-february with a slight 2-week pause after a roblox staff member responded to me and ā€œapplied a flagā€ which seemed to fix the problem. However, it has now re-appeared as of last Wednesday.

We are a combat community who regularly hosted 250+ player battles in one server, but now the server chokes when approaching 180+.

Here is a link to my game. Thank you! If you need additional repro steps please let me know.

Thanks - is my understanding correct then that you would need a server with a high number of players to repro?

Hey Node! I actually don’t think I have access to your game, I tried clicking the link and got https://www.roblox.com/request-error?code=404. Could you give me a working link? And any repro steps would be awesome!

Yes, it appears to reliably break down when surpassing about 180~ in-game very quickly as a hard limit. Prior to that it runs completely fine but post that on the approach to 200+ server performance instantly degrades.

Oof, I see that’s a little difficult to repro. We’ve been seeing similar problems in some other games too, so that’s good to know.

If anything like MicroProfiler logs would help i’ll happily benchmark them and send it over. Like you said i have been contacted by other games that do similar large battles in a single server that ran historically fine but now choke.

I have tried to profile it myself with ScriptProfiler but have been unable to find any culprits beside an ā€œUnknownā€ script which i cannot identify (which i believe is some sort of corescript). All i have been able to detect is massive spikes in Total Data KB/s and of course ping.

Yep, that’s sadly Roblox’s new policy for groups you don’t have access too. The game itself is closed and behind a group right now; for our testing phases.

If you’re able to pend to the group, I can accept you and you should then be able to look in to it.
Alternatively here is the Creator Place link if that helps any.

I’ve made an additional post here which goes over a physics related issue, and sending data, which seems to be prevelant in specific servers or regions, but not others. My sneaking suspicion is that this issue (seperate one); also falls victim to that case. Where it’s more prominent in some regions or servers.

I’ve included a lot of additional information in my private message associated with that specific post. If you’d be able to take a look and draw some conclusions yourself that’d be amazing. You can find the private message here.

I’m unsure what the exact Repro steps would be for you, but if you’d like I could attempt to schedule a time and do the repro steps I have been doing to enflict this issue.

My experience is unfortunately experiencing this as well which happened around January into February with no good results. I did post in the other thread, and seems like that thread is mostly inactive (and I’m not sure which is the main thread for this issue).

I did some ScriptProfiling (in the development console) and here’s my results:

On the client:

On the server:

I’m happy to provide more information and if I need to for y’all to diagnose the issue. Let me know what I need to provide.

We had an event today, performed substantially better than before but still with degraded server performance to a degree where it affected gameplay.