StreamingSolver on occasion causes large memory use and large frequent lag spikes

Huge lag spikes occur in my experience sometimes lasting up to a second. I’ve tracked the issue down to streaming but there isn’t any documentation on what causes ‘Set Movement’ lag spikes.

These lag spikes happen repeatedly sometimes once every 5 seconds, making the experience unplayable.
When these lag spikes occur, the memory used on ‘network/streamingSolver’ spikes to full GBs of memory.

If this is my own doing, could somebody help me figure out what’s going on?

Expected behavior

Streaming shouldn’t cause huge frequent lag spikes or huge memory usage.

8 Likes

Also saw the same (or similar) thing in my experience: ✈️ [SHIP] Star Wars: Roleplay - Roblox

The server became extremely laggy and was using 5GB of RAM.
900mb was going to replicationCoalescing, another 520 to megaReplicationData.

I restarted my game’s servers and I’m now monitoring and waiting to see if it happens again. Could be a memory leak on my end I suppose but don’t think that would explain why those replication-related categories got so high.

Place instance this specifically happened in: 14425807237, aaca35aa-bbb0-4780-9de4-515e2cb9cfba

2 Likes

Yeah, if I notice it happening again, I’ll turn streaming off. Though, I’m sure this will make things difficult for mobile & low-end devices so not really ideal.

1 Like

Thanks rain, should make gar less laggy :+1:

@RainyLofi
How big are the maps in your games?
The bigger the map, the more memory that will be consumed, I’ve had this very issue in one of my games that generates a massive map, without Content Streaming the memory issue is no more but at the cost that it will cause lag to clients.

It’s not that big, I’d say it was medium sized. Only around 1918x1381 studs with 57142 baseparts without players + terrain. Servers are 75 players.

Regardless, I’d say that 900mb going to replicationCoalescing and another 520mb to megaReplicationData is way too much memory for the server to be using for streaming in any reasonably sized map.

Streaming settings:
Defaults, min radius 64, target radius 768, stream out low memory.

Maybe try out other stream out modes?

Personally using opportunistic, and haven’t noticed this.

@GeneralGory thanks for reporting this issue. From the microprofile you shared it appears to be due to “Set Movement”, which most likely is due to moving an object or atomic model that has a large number of parts, or a very large extents (or both). Does your place happen to have any of these?

  1. An atomic model with a large number of descendant parts. For example, making a single atomic model and putting all NPCs or players into the same model.
  2. An atomic model with descendant parts that are widely distributed (spatially) around the place.
  3. A mechanism with a very large number of parts and/or extents

What place id are you experiencing these issues on?

@GeneralGory are you able to provide a repo?

And is this new behavior you started experiencing recently?

Place id 12068411700

I can’t reliably reproduce the issue but once it starts it’s ongoing till certain/nearly all players leave so I agree it’s probably something to do with characters.

We do set characters to atomic streaming behaviour.

Would adding parts on client which increase the extents cause this problem, or is it only the server extents that would matter? I can see a case in our game where we accidently give the characters huge extents on client.

Are you seeing these lag spikes on the client, the server, or both? Are you able to share any microprofiles from when this occurs?

How often does this occur?

The high memory usage for Content Streaming is a problem for me too.

Nearly 900mb of memory was going to /megaReplication data, and causing my servers to reach nearly 5-6GB of RAM. (Even worse if players join)



2 Likes

I receive the same thing as well.

2 Likes

I have also received the same thing. :thinking: