LAG SPIKES With Replicated Part Garbage Collection

(Reporting on behalf of a friend who is experiencing this issue)

Their game: [V3.9.3 Patch] T ang County, Hebei - Roblox

has been experiencing noticeable frame drops recently, as reported by their players and confirmed by their development team. Upon investigating, they determined that the issue lies on Roblox’s end. In the Micrprofiler, they observed spikes in the ‘Replicator GC job’ related to ‘Part/Model Removals’, which is associated with streaming. These spikes exacerbated and became more frequent in longer server sessions (server sessions that were up for 1 hour or longer).

Microprofiler Replicator GC Job:

They made adjustments to the streaming settings by decreasing the StreamingTargetRadius from 1040 to 400 and changing StreamOutBehavior to Default, which has resolved the problem for now. However, this is just a temporary solution as they aim to increase the target radius once this issue has been fully resolved.

Streaming Settings prior to change:
image

Streaming Settings after change:
image

This issue started occurring on the 23rd of May. They have confirmed that their game is not the only one affected, there has been an additional report here: Game Lag Spike - Microprofiler shows high Replicator GC Job - Part/Model Removals - Part Removal

6 Likes

Hi @CDDevelopment

We would require a microprofiler from your friend in order to find the root cause of the issue!

Thanks!

Hey @Focia19, here is the microprofiler:

Microprofiler Dump:
microprofile-20240527-180237.html (4.3 MB)

Thank you for providing the microprofile. Is the length of the Replicator GC Job steps in that profile typical of what is encountered when this issue occurs?

Yes, that is the typical length for each Replicator GC job spike in the microprofiler when this issue occurs

I’ve made a configuration change for the indicated place id: 4618049391

If they could change the settings back to the previous settings and retest we would appreciate it.

Thanks for bringing this to our attention.

1 Like

microprofile-20240528-151112.html (8.7 MB)

Seems to be improved now, here’s the Replicator GC job now:

I do want to mention that there are other games that are experiencing this issue, is this configuration going to remain specific to 4618049391 and 17371491481, or will it be universally applied?

We have disabled the implicated change globally while we investigate.

Do you happen to know which other games experienced the issue? Any additional reports could be useful for us during the investigation.

Here are the place ids:
4637668954
8265622251

For the first places where the issue was reported, 4618049391 and 17371491481, I can confirm that the issue appeared due to the use of JointsService, and due to a very large number of joints being stored in JointsService.

While JointsService has been deprecated since 2019, the performance impact that resulted in the long frames was not intended, and we will develop a fix.

I noticed very large joint counts in JointsService for both 4618049391 and 17371491481, over 160,000 in one of them and over 40,000 joints in the other. This, along with the fact that the issues were reported to be longer with longer server sessions, leads me to believe that there are likely joints being left in JointsService and not completely removed in those experiences. Perhaps joints are being added to JointsService but never removed? Having over 160k joints in an experience seems high.

I recommend checking that joints are properly being cleaned up, and if possible not storing the joints in JointsService. This is usually a simple matter of changing the parent of the joint to be one of the parts in the joint rather than JointsService. This can also help ensure the joint is destroyed when the part is destroyed. If you are using A-Chassis then I think it is a one line change to stop using JointsService.

I did not see usage of JointsService in 4637668954 or 8265622251, but I was also not able to reproduce the performance issue with those experiences. They were updated recently so I’m wondering if they were using JointsService but are no longer.

This was resolved by a configuration change.

We do recommend avoiding the usage of JointsService as that has been deprecated since 2019.