updateInvalidatedFastClusters/updateEntity triggers on character descendant part property changes, causing spikes

Reproduction Steps
Reproduction

Play using the attached reproduction place file. Observe the microprofiler on any frame and find the updateInvalidatedFastClusters tag. For context, 300 regular parts are also created under the character to show the additive effect, but descendant MeshParts (including default character parts) add much worse overhead than regular parts.

Reproduction.rbxl (28.6 KB)

System Information

i7 9700k
RTX 2080 S
32GB
Windows 10

Expected Behavior
When I update a sub-part in a character model, I expect updateInvalidatedFastClusters to not re-process all the character descendant parts on the next frame.

Actual Behavior
When I update a sub-part in a character model, updateInvalidatedFastClusters triggers on the next frame.

Workaround
As this mostly concerns custom character visuals/accessories which change frequently, these visuals can be welded to the character but parented to somewhere else like Workspace. This does not trigger updateInvalidatedFastClusters.

I expect the workaround is unintuitive and confusing to most developers, as they parent character-related parts to the character and won’t understand what this microprofiler tag represents or associate its cause to updating character sub-parts.

Issue Area: Engine
Issue Type: Performance
Impact: Moderate
Frequency: Often
Date First Experienced: 2021-06-01 00:06:00 (+01:00)
Date Last Experienced: 2021-08-29 00:08:00 (+01:00)

5 Likes

Hi NiceDraft,
This is actually expected. FastClusters will be invalidated whenever a part is added or changed, to generate a single mesh and texture for all the parts to improve rendering performance, but apparently in your case, this is causing performance issues. Can I ask what your intention is to add hundreds of parts under an avatar and change them very often? If this is something really common among developers, we could do something handle cases like this in the client. Like you said, some or most developers might not know the difference between adding a part to an avatar and adding it directly to the workspace.

Thanks,
FishrotR

4 Likes

Updating characters this way causes unnecessary lag to games where the character updates frequently. For instance:

  • Shooters (where you die and respawn often, with animated guns and outfits re-parented to the character)
  • Anime or physical combat games (where you use a lot of “flashy attacks” and custom part-based effects inside the character)

300 parts were created to simulate the cost of ~20-30 mesh parts in our game (gun MeshParts inside the character). Regular parts are an order of magnitude less expensive than mesh parts in my use case. The invalidated clusters still take up a lot of frame time due to the frequency of player respawns. You can view this in the microprofiler here - make sure to select Competitive when you spawn: [PISTOLS] Aimblox BETA - Roblox

Player character updates can also replicate in batches, so I see problems like this:

And I’m running on an overclocked i7 9700k & RTX 2080S - we can expect double to triple the frametime per invalidated cluster on an average machine, and even more on mobile.

I see a lot of developers:

  1. intuitively parenting assemblies which move around with the character, to the character (e.g. custom accessories, custom tools, custom outfits, custom visual fx, will be parented into the player character model, as opposed to a non-character object like Workspace)

  2. animating parts inside the character (for example, a blast effect in an anime game is parented to the player character, changing rotation, size, and transparency every frame, thus re-uploading the whole player character geometry each frame)

  3. respawning players frequently in PvP games where players die a lot - this causes unnecessary lag spikes when anything relating to visuals is added to the player character

In my specific use case, we parent these to the player character (usually on spawn):

  • Guns & gun holstering

  • Red outline parts (for enemy players)

  • Hitboxes (per-platform)

In one update ~2 months ago, we experienced severe lag spikes when animating out the transparency of the enemy player outlines on death, which dropped playtime by 20%. There is still an issue of frequent lag spikes due to players respawning. I feel like this doesn’t need to be the case, given how simple the workaround is.

2 Likes

Another example I just found on a game with 10k CCUs - Zenkai Origins - Roblox - just open microprofiler and look at any frame, ~4ms on a high-end desktop CPU+GPU

There’s probably many more front-page examples given how common it is for scripts to update sub-parts in characters when relating to part-based effects.

Hi NiceDrift,

I agree that respawning happens quite often in pvp shooter games, but currently one respawn will lead to about 2 to 3 updateEntity() calls. In games like Arsenal, I think on average, there will be one spawn per second, which shouldn’t put much pressure on CPU, but updateEntity() caused by vfx update might be a different story, if any vfx update will trigger an invalidation, overall invalidation cost will be really high. Does Aimblox have the perf issue with vfx update you mentioned above?

You saw 4ms on updateInvalidatedFastClusters because that is the upper time limit we set for updateInvalidatedFastClusters() each frame. Any remaining invalidated fast clusters will be processed in the next frame if time limit for this frame has been reached. What I mean is that even on a low-end machine, updateInvalidateFastClusters() usually won’t cost more than 4ms, but it will take much longer for all the fast clusters to be updated. However, I think I still need to take a look why there are so many invalidation in a frame, and whether that is something pretty common or not.

I’ll do some investigations and will get back to you soon.

Thanks,
FishrotR

2 Likes

HI NiceDrift,
Sorry for late reply. I did some investigation on the behavior of updateEntity().
First I confirmed that updating properties of particle emitters won’t trigger any updateEntity(). You could try this in an empty map with only one emitter and verify. If you still see updateEntity(), let me know and share with me your test map.
Second, no matter how many parts under a humanoid are updated, the humanoid will only be invalidated once, although updateEntity() could be called 2 to 3 times. (We are working on something to reduce number of updateEntity calls on each invalidation). This means, no matter whether there are 20 or 200 parts added to your avatar, the number of updateEntity() calls will be the same. If in your test with 200 parts, 200 parts are updated one per frame, it won’t actually represent the performance of having 20 parts. I still don’t understand why you are using 200 parts to simulate perf or 20 parts. If you are trying to simulate the physics/animation/simulation cost on a low-spec machine, this is OK, but this won’t increase any cost on updateEntity() calls.
Third, no matter how many humanoids are invalidated, the total budget for all updateEntity() calls in a frames is limited to 4 ms. All remaining invalidations will be handled in the next frame. 4 ms shouldn’t be a burden for most of the games. I played AimBox. There are about 400 kills in a 4 minute game, which means there will be about 1000 updateEntity() calls in 7200 frames (assuming 30 fps). Only 1 out of 7 frames will have 1 updateEntity() on average.
Last, if parts are welded to humanoids through attachments, when attachments are updated, the humanoid will not be invalidated. In your case, movement of guns should be handled by attachments and animation. I’m not sure how you implemented red outline and hitboxes, but I think there will be a way to avoid extra updateEntity() calls when they are updated. Are you sure the lag spikes when animating transparency of outlines are caused by updateEntity(). If you could explain your implementation to me, we may be able to figure out a solution for optimization.

Thanks,
FishrotR

1 Like