Layered Clothing Has SERIOUS Performance Issues

Reproduction Steps

Simply start up any game where users are using layered clothing, then watch the microprofiler, plenty of frame skips will be visible.

Here’s a profile dump generated on my Windows 10 desktop, AMD Ryzen 9 3950X, RTX 2070 Super:


Expected Behavior

I expect Layered Clothing to run smoothly, or in a fashion that makes my games playable to some degree.

Actual Behavior

Zooming into one of the frame drops on the profiler shows that it’s caused by the clothing replication registering as “Invalidated Fast Clusters”. These clusters take 6ms on my BEEFY machine just to render the clothing for a single frame. For other users on mobile, console, or not so great PCs, the game is unplayable. Framerate drops extremely low at seemingly random points.

Workaround

Not really a workaround, just removing layered clothing fixes the problem.

Issue Area: Engine
Issue Type: Performance
Impact: Very High
Frequency: Constantly
Date First Experienced: 2022-06-01 00:06:00 (-04:00)
Date Last Experienced: 2022-06-18 00:06:00 (-04:00)

11 Likes

We’re taking a look, we’ll let you know what we find.

7 Likes

Some more information I’ve found. It’s only on first render. So once that character appears on screen you’ll get a large lag spike due to invalidated clusters, but never again for that one character. If they respawn however, it’ll happen again once you see them.

1 Like

There is an expected delay when an avatar joins wearing layered clothing, as the clothing is being fit to the avatar. We are planning to optimize this further, but as-is it should not make the game unplayable or otherwise significantly impact game play. Can you send a specific example of a place you think is badly impacted, along with instructions on how we can reproduce the playability issues you are seeing? We’ve looked at the test scene you already sent, but we haven’t been able to reproduce significant issues. Thanks for your help.

1 Like

Here is a place impacted: Rolling Thunder - Roblox

It’s an FPS game of 16v16 and everytime a single player is first rendered, it takes 6ms out of the frame time. Leading the game to be choppy often. This is a problem because it means users will lag as soon as they spot an enemy, which is particularly bad for combat gameplay.

In addition we’d like to increase the player count per sever, but can’t until this issue is addressed.

2 Likes

For further help, I’ve collected some microprofiler dumps from that game. These were collected on my windows machine with about 28 players in the game.

This profile is at around 115 FPS, at this framerate you can better see when this bug rears it’s head because it will produce abnormally long frames quite frequently: microprofile-20220702-140705.html - Google Drive

Due to the frequency of players respawning this bug occurs EVERY frame at normal 60 FPS gameplay and it makes the game feel really choppy. This profile is captured at the normal 60 FPS where you can see almost every frame affected: microprofile-20220702-140738.html - Google Drive

I have narrowed down these “invalidated fast clusters” as 100% coming from the layered clothing alone. No one specific clothing object creates the problem, just clothing in general causes it. Oddly enough the frequency and render time it takes is always the same, regardless of the complexity of the geometry or number of clothing items.

1 Like

While playing your game, I see 1-3 FastCluster updates per second. This is not once per frame, but it still seems like more FastCluster updates than can be explained by players dying/spawning alone. We are continuing to investigate.

lag as soon as they spot an enemy

Are you saying there is a lag that happens when a player first sees another player? Or is this coincidental, ie sometime the lags happen when an enemy is spotted, but sometimes not, and sometimes they happen at other points in the game?

We’ve looked at your game and your issue appears to be a combination of two things:

  1. Updating/rebuilding a character’s cluster with layered clothing does take ~6ms like you said, causing a missed frame.
  2. The clusters are not only rebuilt once when a character spawns, but several times in rapid succession, sometimes with less than 20ms between updates on the same character.

There is nothing you can do about 1. We are planning further optimizations to improve the rebuild times, but I don’t know when that will be available.

But, you may be able to optimize 2 yourself by tweaking your game scripts. We don’t have permission to see your scripts, so we can’t tell you exactly what is triggering all the cluster updates, but our suspicion is they make several data model changes in rapid succession when a character dies or spawns, that each trigger their own rebuild. These changes could be things like setting up a rag doll, breaking a weld between a character and a weapon, changing a character’s transparency, among many other possible changes. If some or all of these changes can be avoided, or done simultaneously so they only trigger a single rebuild, you should see fewer lags.

Some ideas that you could try:

  1. When a character dies, don’t make them transparent or remove them from the workspace, instead move them to a hidden location on the map, e.g. far from the map or under the ground plane. This way, when they respawn, you can just place them back on the map without triggering a rebuild.
  2. Keep a pre-built rag doll version of each character hidden in the map as well, and swap its location for the animated character when they die.
  3. Keep a non-welded version of the weapons hidden in the map, and swap it in with the rag doll, if you want the character to drop its weapon.

These are just suggestions and depending on what is happening in your scripts, you may have to try other things. The idea is, do anything you can to avoid changing the character’s data model any more than necessary while the game is running. If you can do this, you should start seeing fewer lags.

Please reach out if you need more help with this.

The frequency of the updates seems identical both in Rolling Thunder and my empty test place. In my empty place I’m not making any changes to the data model and it occurs when they are rendered on screen. Which in an FPS game does actually happen really frequently.

I do plan on optimizing the ragdolls a bit, but your suggestions aren’t really feasible. Not creating new characters is AWFUL for anti-cheat. I can’t make a pre-built ragdoll because we’re adding character customization, so updating the ragdoll to include a player’s clothing would create more clusters.

Some additional information though, I fixed the invalidated clusters entirely back in February and the game performed perfectly fine until mid-may when they cropped up again out of nowhere. Then adding layered clothing made it a million times worse. I’m not 100% sure what roblox update around that time would’ve caused it, but that might narrow down the cause of issue 2.

1 Like

Sorry for bumping this post again, but I did some more investigating and found some worrying issues.

Even without Layered Clothing, on regular roblox avatars, the following things will cause Invalidated Fast Clusters and slow frames:

Animation Track play/stop
Humanoid:LoadAnimation()
Humanoid:MoveTo()

When you combine that with the avatars and clothing in my game you’ll get very frequent slow frames when users respawn, when they deploy (because that’s when it applies clothing), and whenever there’s an animation track change.

6 Likes

Thanks for the details, we’ll take a look.

1 Like

We cannot reproduce this behavior. None of the above trigger FastCluster invalidation in our tests. Can you please share a simple place file that demonstrates what you are seeing?

Have you read this?