Simply start up any game where users are using layered clothing, then watch the microprofiler, plenty of frame skips will be visible.
Here’s a profile dump generated on my Windows 10 desktop, AMD Ryzen 9 3950X, RTX 2070 Super:
Expected Behavior
I expect Layered Clothing to run smoothly, or in a fashion that makes my games playable to some degree.
Actual Behavior
Zooming into one of the frame drops on the profiler shows that it’s caused by the clothing replication registering as “Invalidated Fast Clusters”. These clusters take 6ms on my BEEFY machine just to render the clothing for a single frame. For other users on mobile, console, or not so great PCs, the game is unplayable. Framerate drops extremely low at seemingly random points.
Workaround
Not really a workaround, just removing layered clothing fixes the problem.
Issue Area: Engine Issue Type: Performance Impact: Very High Frequency: Constantly Date First Experienced: 2022-06-01 00:06:00 (-04:00) Date Last Experienced: 2022-06-18 00:06:00 (-04:00)
Some more information I’ve found. It’s only on first render. So once that character appears on screen you’ll get a large lag spike due to invalidated clusters, but never again for that one character. If they respawn however, it’ll happen again once you see them.
There is an expected delay when an avatar joins wearing layered clothing, as the clothing is being fit to the avatar. We are planning to optimize this further, but as-is it should not make the game unplayable or otherwise significantly impact game play. Can you send a specific example of a place you think is badly impacted, along with instructions on how we can reproduce the playability issues you are seeing? We’ve looked at the test scene you already sent, but we haven’t been able to reproduce significant issues. Thanks for your help.
It’s an FPS game of 16v16 and everytime a single player is first rendered, it takes 6ms out of the frame time. Leading the game to be choppy often. This is a problem because it means users will lag as soon as they spot an enemy, which is particularly bad for combat gameplay.
In addition we’d like to increase the player count per sever, but can’t until this issue is addressed.
For further help, I’ve collected some microprofiler dumps from that game. These were collected on my windows machine with about 28 players in the game.
This profile is at around 115 FPS, at this framerate you can better see when this bug rears it’s head because it will produce abnormally long frames quite frequently: microprofile-20220702-140705.html - Google Drive
Due to the frequency of players respawning this bug occurs EVERY frame at normal 60 FPS gameplay and it makes the game feel really choppy. This profile is captured at the normal 60 FPS where you can see almost every frame affected: microprofile-20220702-140738.html - Google Drive
I have narrowed down these “invalidated fast clusters” as 100% coming from the layered clothing alone. No one specific clothing object creates the problem, just clothing in general causes it. Oddly enough the frequency and render time it takes is always the same, regardless of the complexity of the geometry or number of clothing items.
While playing your game, I see 1-3 FastCluster updates per second. This is not once per frame, but it still seems like more FastCluster updates than can be explained by players dying/spawning alone. We are continuing to investigate.
lag as soon as they spot an enemy
Are you saying there is a lag that happens when a player first sees another player? Or is this coincidental, ie sometime the lags happen when an enemy is spotted, but sometimes not, and sometimes they happen at other points in the game?
We’ve looked at your game and your issue appears to be a combination of two things:
Updating/rebuilding a character’s cluster with layered clothing does take ~6ms like you said, causing a missed frame.
The clusters are not only rebuilt once when a character spawns, but several times in rapid succession, sometimes with less than 20ms between updates on the same character.
There is nothing you can do about 1. We are planning further optimizations to improve the rebuild times, but I don’t know when that will be available.
But, you may be able to optimize 2 yourself by tweaking your game scripts. We don’t have permission to see your scripts, so we can’t tell you exactly what is triggering all the cluster updates, but our suspicion is they make several data model changes in rapid succession when a character dies or spawns, that each trigger their own rebuild. These changes could be things like setting up a rag doll, breaking a weld between a character and a weapon, changing a character’s transparency, among many other possible changes. If some or all of these changes can be avoided, or done simultaneously so they only trigger a single rebuild, you should see fewer lags.
Some ideas that you could try:
When a character dies, don’t make them transparent or remove them from the workspace, instead move them to a hidden location on the map, e.g. far from the map or under the ground plane. This way, when they respawn, you can just place them back on the map without triggering a rebuild.
Keep a pre-built rag doll version of each character hidden in the map as well, and swap its location for the animated character when they die.
Keep a non-welded version of the weapons hidden in the map, and swap it in with the rag doll, if you want the character to drop its weapon.
These are just suggestions and depending on what is happening in your scripts, you may have to try other things. The idea is, do anything you can to avoid changing the character’s data model any more than necessary while the game is running. If you can do this, you should start seeing fewer lags.
The frequency of the updates seems identical both in Rolling Thunder and my empty test place. In my empty place I’m not making any changes to the data model and it occurs when they are rendered on screen. Which in an FPS game does actually happen really frequently.
I do plan on optimizing the ragdolls a bit, but your suggestions aren’t really feasible. Not creating new characters is AWFUL for anti-cheat. I can’t make a pre-built ragdoll because we’re adding character customization, so updating the ragdoll to include a player’s clothing would create more clusters.
Some additional information though, I fixed the invalidated clusters entirely back in February and the game performed perfectly fine until mid-may when they cropped up again out of nowhere. Then adding layered clothing made it a million times worse. I’m not 100% sure what roblox update around that time would’ve caused it, but that might narrow down the cause of issue 2.
When you combine that with the avatars and clothing in my game you’ll get very frequent slow frames when users respawn, when they deploy (because that’s when it applies clothing), and whenever there’s an animation track change.
We cannot reproduce this behavior. None of the above trigger FastCluster invalidation in our tests. Can you please share a simple place file that demonstrates what you are seeing?
Hate to re-open such an old thread but I’d like to point out that this is still an issue. I’ve done all of those optimizations, and cut down on the amount of cluster calculations, but I can’t optimize it any further.
Whenever there’s an invalid cluster calc it’s around 7 to 9ms (that’s on my beefy PC, I imagine it’s much worse for lesser machines). These clusters occur around once every couple of seconds for no discernable reason. There does seem to be a hard limit on how long it takes to run though, so it’s common to see several 9ms calcs in quick succession.
Not sure if Layered Clothing has just gotten less optimized over time, or if having DynamicFaces in my character models is making it worse.
I’ve also detected that 50% of our game’s PC users are crashing out of the game, and with the invalid clusters being the only performance impactor I believe it’s the most likely cause.
The longest calc is “Deform Vertices”, invalid clusters without that marker perform very quickly. I observed this with some friends in a private servers, with just them walking around, doing absolutely nothing else, I’ll get a 9ms cluster every couple seconds. If they stand still there will be no clusters.
Unless of course I’m looking around, if they’re standing still and I look around I’ll still get 7ms clusters frequently.
I’ll see if I can get a log from this user, but you can see in the image that the layered clothing on two of the characters was being calculated just as it crashed out.
Found some exact causes on my end that will trigger fast clusters, these are mostly unsurprising.
Adjusting the transparency of one of the character parts triggers a 9ms cluster
Setting a Highlight’s Adornee to the character model will trigger multiple 9ms fast clusters
Setting the parent of the character will trigger a 7ms fast cluster
However there is one issue that is most DEFINITELY on Roblox’s end.
If two characters touch each other (or at least come close to touching), and at least one of them is moving. It will trigger 2 9ms fast clusters. These clusters are always spaced evenly apart, somewhere around 0.2 seconds apart. Once these two clusters occur, the characters can move around and touch each other no problem, until they leave each other’s personal space. Then once they re-enter each other’s personal space bubble, it’ll trigger a new 2 clusters.
Since their sphere of influence seems very small, it’s possible that two players moving next to each other can trigger a bunch of fast clusters in quick succession as they enter and leave each other’s space.
This is why it seemed like animation tracks and humanoid:MoveTo() affected it in the past. On the test place when that was happening, character models were just bumping into each other.
There’s nothing I can do on a developer level to fix this, seems to be physics related in some way.
This is also a big deal because that means using SpawnLocations in a game with layered clothing or skinned meshes means that characters will constantly be colliding with each other when they respawn, and any characters nearby will get clusters.
