In our game, we do not use standard Roblox characters. In fact we do not even use the Character property on the Player instance at all. When players respawn in our game a server-sided part is created to represent the player. On the client, we create the actual visuals for the player. This is done with a cache that we pull from when we need to get a character model. We are noticing that when players respawn there is always an “updateInvalidatedFastClusters” job happening in the microprofile. The more players that respawn at once, the slower this is. After months of searching we are unable to diagnose the issue and need assistance figuring it out.
A few questions:
Is this a new issue that has started happening recently or was it happening for a while? Has it gotten worse?
We made some improvements to avatar rendering (and the FastCluster that is showing up on the micprofiler) a few months ago. The announcement and some tips on improvements are here: Making Avatar Rendering More Performant
On the client, we create the actual visuals for the player. This is done with a cache that we pull from when we need to get a character model. We are noticing that when players respawn there is always an “updateInvalidatedFastClusters” job happening in the microprofile.
How does the cache work exactly?
So updateInvalidatedFastClusters gets called whenever there is a change to the player avatar that requires a geometry rebuild. On respawns this makes sense. But its possible with your cache mechanism perhaps this is doing redundant work.
A couple of tips to help improve this:
reduce property changes as much as possible, or do them before bringing the player avatar model into the workspace / game world
any hierarchical changes to the player avatar rig will cause it to rebuild so reduce that as much as possible
break up the player avatar rigs into separate Models for accessories, and extra parts
if you are bringing in all the player characters at once that could cause a frame spike, so perhaps stretch this out over time
If you can provide more info on your cache system or provide an example, we might be able to help further. Let us know if any of the above help you.
Our model cache keeps the player models at a position outside of the game world. They perpetually exist in workspace, but are moved far away when not active. When a player respawns, we release their character back to the cache, and grab the next available one.
This is not a new issue, I’ve seen this happening for years with our game framework. When we use standard R15 rigs it is not an issue. It is only an issue when we use skinned mesh avatars with many bones and higher triangle counts.
It is difficult to diagnose because our project is huge. Some general themes we are seeing:
When a player dies we spawn their gun (physically simulated) at their death position, and it falls to the ground. This causes an updateInvalidatedFastClusters. The weapon model is a simple Model with ~ 10 parts/meshparts on average.
When a player respawns we attach a weapon model to their avatar (skinned mesh). This causes an updateInvalidatedFastClusters.
When a player dies they ragdoll (Skinned mesh). Enabling and disabling the ragdoll causes an updateInvalidatedFastClusters. All instances related to this are cached. Nothing is instantiated.
When we disable the above three systems, we see a dramatic drop in this in the profiler. But not completely gone. Only when we completely disable avatar rendering do we see all remaining updateInvalidatedFastClusters disappear.
We are already very careful about what properties we update and how often.
We do not change anything with the hierarchy at runtime once the cache is built.
Our player rigs are one model. There are no accessories or extra parts outside of the single weapon model, which is already in its own model as a sibling to the character model.
If you are just moving the player model far away, it shouldn’t cause a FastCluster rebuild, unless you don’t have a Model as the root. Without a Model, spatial / position changes will cause FastCluster rebuilds.
Now if you do have a Model as the root, then something else changes the hierarchy or property causing the FastCluster rebuild. It would be good to experiment to find out what that might be.
When a player dies we spawn their gun (physically simulated) at their death position, and it falls to the ground. This causes an updateInvalidatedFastClusters. The weapon model is a simple Model with ~ 10 parts/meshparts on average.
Adding a gun via a constraint shouldn’t cause a FastCluster rebuild any more, but it depends on the setup. Are you able to provide an example rbxl file through private message?
When a player dies they ragdoll (Skinned mesh). Enabling and disabling the ragdoll causes an updateInvalidatedFastClusters. All instances related to this are cached. Nothing is instantiated.
This FastCluster rebuild is expected. One way to reduce the cost of this is to user a more lightweight avatar model just for the ragdoll.
Our player rigs are one model. There are no accessories or extra parts outside of the single weapon model, which is already in its own model as a sibling to the character model.
It sounds like you have 3 models then? A root model, a child character model, and a child weapon model. Is that correct?
This is very unfortunate. Our ragdoll is already a heavily simplified assembly that we move the skinned meshes’ bones to, similar to how classical welds work. This is done in the heartbeat. This assembly also doubles are our avatars hitbox. We move the hitbox parts to match the animation so it can be shot at.
That’s fair, it is 3 models. They are all siblings in the workspace. All each in their own Model instance.
Server-sided character (This is just a block inside of a model)
Client-sided character (Pulled from the cache, moved to the server-sided character via renderstep). The hitbox parts exist within a Folder that is a child of the character model.
Client-sided weapon (Also pulled from a cache. Once selected we attach this via physics constraints. We also have a pathway to attach it via PivotTo() on a renderstep. No difference between the two from what we can tell.
It is difficult to create a standalone repro. It would take a bit of time to create a 1:1 test. Many of our systems are tightly integrated.
It does not have a Humanoid or AnimationController. I can try adding one, but unless there’s some magic black-box stuff going on I am not sure how it will make a difference, since there’s no “HumanoidRootPart”–We dont use a standard Roblox-like character, so we cannot satisfy the RigType property on the Humanoid or get it to assign the “RootPart” property
Adding a Humanoid will ensure that the parts in the character model are built together. Without it, it ends up using a spatial grouping approach which explains why you are seeing FastCluster rebuilds whenever you move the character model spatially. Adding a Humanoid should fix your issue.
The picture above shows the microprofiler with the Humanoid added! updateInvalidatedFastClusters is still present. I may be crazy, but it seems to defer the issue across multiple frames when a Humanoid is in the model. If this is the case, it will still feel equally laggy to the player when playing, since frames are still below 60 fps
I dont know if this is exactly true in my case. I see it when the character model is moved to the spawn position. However, while the players run around I do not see it. Every renderstep we do a BulkMoveTo on the client side character model to smoothly interpolate it to the server side characters position.
From your previous screenshot, it seems roughly half the time in FastCluster::update is spent in upload, is that right? It might be a pretty heavy model perhaps? If you can send us the model, we can take a quick look to see if we it can be sped up somehow.
Yes, there is a upper bound imposed on rebuilding Humanoids based on per frame time, which we consider to be the more expensive FastClusters, due to the nature of player avatar rigs in general. I would advise to keep the Humanoid to take advantage of this to reduce the single frame spikes.
I dont know if this is exactly true in my case. I see it when the character model is moved to the spawn position. However, while the players run around I do not see it.
It is grid-based, so it will only trigger a rebuild if it moves across grid cells (i.e. large position changes).
Here is a screenshot of the profiler upon respawning the players where I have disabled all scripts that touch the client side character (Animator, dropping guns on death, holding guns in hand, etc)
There is still a bit of upload in the fast cluster job, with zero change to the model except calling BulkMoveTo() to position it, as far as I can tell.
