updateGeometry causes low FPS on big maps with objects out of view

[MetatableIndex]

Reproduction Steps
When a player stares at a big map in a place where they cannot see objects far from them, it looks like all those parts that are not visible to the player are still rendered, and in particular, in the updateInvalidadedFastClusters updateGeometry bar this takes long to complete, causing lower FPS than what the player could be having.

To reproduce this bug, Roblox staff have been given access to The Red Wolves team in the following game:

Glacier

Once you join the game, press Play, select The Red Wolves and press Spawn. Face forwards, left or right, and this should be noticeable. Note that the Red Wolves spawn is in the middle of one of the map’s borders, as the game is 10k by 10k studs big.

Expected Behavior
I expect updateGeometry to not take 20ms to complete, at least avoiding updating as much as it does right now as most is out of view.

Actual Behavior

Issue Area: Engine
Issue Type: Performance
Impact: High
Frequency: Constantly

[fishrotR]

updateEntity() usually won’t take that long. I’ll take a look what happened in The Red Wolves.
Quick question. Does this long updateEntity() happen every frame?

[MetatableIndex]

Hey, thanks for the quick follow up.

It seems like this only occurs when there are parts actively loading / out of the game, such as is the case with streaming or custom load / unload solutions. Furthermore, the bar remains at an inconsistent higher value than 4ms for a variable amount of frames until it ceases to appear.
Perhaps the reason this might be more felt in the game linked in the post in particular is the fact it is a big terrain based open world with vehicles and a lot of players, meaning a constant stream in and out of models.

[fishrotR]

Hi MetatableIndex,
I had a look at the game, and what I found is that the reason why it is so expensive to update the geometry of the avatars in the game is that each avatar has to many mesh parts and those mesh parts are also pretty high-quality.
My suggestion is to combine all weapon meshes into one and import a lower-poly mesh for the weapon. Your game is really high-quality, but it’s not necessary to have such high-poly meshes for each part of the weapon. Players won’t see those details. And usually details can be baked into a normal map if you do need pixel-level details
If you need these different parts for gameplay features, such as swapping weapon parts, I suggest having simplified weapon meshes for non-local players. However you’ll have to do some tricky programming to get it done. For what is now in the game, I don’t see it necessary to have weapon parts.
Roblox engine currently doesn’t have mesh LODs and it’s a bit expensive to process complex meshes, because it takes time to merge all the vertices into the avatar mesh. We are making progress to make it more efficient, and hopefully we will be able to support heavier meshes and scenes more in a more efficient way in future.

Let me know if these suggestion helps and whether you need more help or have more questions.

FishrotR

[MetatableIndex]

Hey, thanks for the reply!

Sorry for taking so long to get back to you, since Roblox was down I was unable to further test this issue.

I appreciate the detailed explanation. However, upon further testing, it does not seem like avatar complexity is the issue.

I was able to reproduce the long frames described in op in an empty server with no one else online.

This seems to occur mostly whenever streaming is occuring. This is more frequent in live servers simply because of the amount of moving players, vehicles, parts, etc. that need to constantly stream in / out.
Worth to note this was not an issue weeks ago from what we can analyze of player performance. Don’t have an exact estimate as to when though.

Edit: Perhaps it might just be the local avatar affecting it? Still wouldn’t make much sense as this didn’t happen before though… could still be related.

[fishrotR]

Hi MetatableIndex,
Sorry I might be very clear. By avatar complexity, I mean all avatars and their weapon meshes, including the local player avatar and other players’ avatars. I think by streaming, you mean players joining.
Since all players have an FAL, which has about 10 mesh parts and each part is a high-poly mesh, any player join will result in a long updateGeometry().
Could you do a simple test? Do not add any weapon mesh part to avatars and see if updateGeometry() time gets much smaller.

If this issue didn’t happen before, did you add the FAL weapon meshes recently?

Thanks,
FishrotR

[MetatableIndex]

Oh, I misunderstood what you meant then.

That does seem to be the case! When using a normal avatar with no gun meshes on top the bar takes much less time. (roughly 1-3ms)

It still seems to take quite some time once a gun tool is equipped when using a standard combat morph (no gun mesh on the avatar) though, which is really not ideal for our game considering morphs such as The Red Wolves are critical elements of the game’s aesthetic, immersion, lore, gameplay.

We already do our best at optimizing the meshes constantly but I guess we’ll have to keep trying harder until optimizations to this are implemented into the engine.

@fishrotR

did you add the FAL weapon meshes recently?

Nope, the FN FAL is 8.7k tris, and these morphs and guns have been a thing for many months. The reports of fps drops related to this frame only started occuring more recently.

[fishrotR]

Could you duplicate your game to a new place with the standard combat morph? I can take a look from there.
Sorry, maybe FAL is OK, although I still think its mesh should be simplified.
And the FAL mesh combined with all its parts such as suppressor, magazine, plus 10 others is an overkill.
The M16A1 has more than 60k tris.

[MetatableIndex]

I believe you can use a private server to reproduce this, using the Red Wolves team. Going in the menu and selecting Test Subject will allow you to see with no standard morph, while selecting Red Wolves will select a standard combat morph with no gun on the back.

Glacier

I agree that there are weapons we definitely must optimize; my main concern though is that on an empty server on the Red Wolves team where no gun is on the morph and you only have the FN Fal I can still experience 9ms on this frame, which can be problematic considering this is not a lot of detail and is a critical element of the game. If the solution is to not include details on morphs or tools, or at least limit them heavily, then this severely limits the immersion the game can offer. My device is quite a beast (Ryzen 9 5950x, RX 6900 XT, 32 GB DDR4-3200MHz) and even I experience fps drops from this, so I can only imagine how other players with much weaker devices must be handling this.

Hopefully the upcoming optimizations you’ve mentioned can help, these really are critical elements of the type of game we’re doing and as such they are vital to our community.

[fishrotR]

It looks like even without any weapon, there are accessory meshes like helmet and bulletproof vest, but I agree these meshes are not heavy, but still, to process the vertices of these meshes are very expensive in the current Roblox engine due to the bad design of the geometry processing module.
Even with a powerful machine like yours, it won’t help much, because the processing work is not currently done multi-threaded.
I’ll create a task to improve geometry processing speed and talk to my colleagues for solutions, but I don’t think this will be something that can be addressed in a short time.

For now, my only suggestion is to simplify meshes and reduce number of vertices as much as possible.

Thanks,
FishrotR

[MetatableIndex]

Hey, sorry for bumping this topic again, but I’ve ran into an issue I am unsure how to resolve and causes updateInvalidatedFastClusters to take 16ms per frame for many frames guaranteed.

We have Night Vision Goggles in some of our morphs that can be toggled up / down by pressing N. They work by changing the DesiredAngle of a Motor6D in their grip part, which is welded to the helmet accessory (HeadAccessory)'s Handle. That Handle is then welded automatically by Roblox to the character’s Head once the accessory is parented to the character.

As long as the Motor6D hasn’t reached its target angle, updateInvalidatedFastClusters will continue to call every frame, eating up 16ms per frame while doing so. This causes enormous lag spikes. Important to note that this was not an issue a year ago when this was first implemented.

This does seem related to the avatar geometry as you mentioned, but all the morphs actually have many less triangles than they did when this feature was first implemented, which is what’s causing us the most confusion in this whole topic.

@fishrotR

Last, if parts are welded to humanoids through attachments, when attachments are updated, the humanoid will not be invalidated.

Most I could find regarding possible solutions for this problem was this post you made in a separate topic about this bar. What does this mean exactly, and how would we go about doing something like that? Would it work in our case?

Thanks in advance.

[fishrotR]

Hi @MetatableIndex ,
I can take a look at your performance issue. Could you tell me how to get the night vision goggle in your game?

[MetatableIndex]

Yep - in Glacier, joining the Red Wolves team should spawn you with a standard combat morph with night vision goggles.
To test the night vision goggles without extra equipment, you can join the Test Subject team and type !nvg in the chat; this should morph you the night vision goggles helmet.

[fishrotR]

Can I join the Test Subject team?

[MetatableIndex]

Yes, it should be visible in the menu in the same area you select The Red Wolves team.

[fishrotR]

Hi @MetatableIndex ,
I took a look at the long updateEntity() issue.
The cause is still the same, which is too many parts with high-poly meshes.

In a test subject team avatar, there are 86 parts with totally 158776 vertices. This is just too much. The goggle lens mesh itself has about 25K vertices, which is definitely an overkill.

Roblox Engine right now is really not very efficient to handle complex meshes like this for an avatar, which needs to be improved in future, but I still think the number of parts and complexity of meshes you are using for your avatar is unnecessary.

First, about number of part, from a gameplay perspective view, you don’t need to split meshes if they don’t contribute to any gameplay individually. For example, the many weapon parts in your game can be combined to one single mesh. The multiple goggle meshes can be combined, too.

Second, about number of vertices, some meshes are too high-poly for a 3rd-person character. Weapon meshes with that many parts and vertices in your game are usually used for 1st-person character in an FPS game. A third-person character should use much simpler meshes. If there is any details you need on that mesh, use a normal map. I’m not sure why you need to have different weapon and equipment parts. Are they switchable and will contribute to some gameplay? If not, please simplify them.
I suggest to reduce total number of vertices to 10K. If you think it’s not acceptable, then reduce it to less than 20K. And, try using normal map for details on meshes.

[MetatableIndex]

Hey, thank you for taking the time to look into this.

Are you sure you were on the correct team? The Test Subject team has no morph (meaning it only has the default Roblox avatar parts). Whenever someone activates their night vision goggles, they move down and you can feel this lag in any team.

Furthermore, we have checked the Red Wolves team morph and it is 76,253 vertices and 49,489 tris. If we take that the average morph has 80k vertices and the average gun has 20k (the most used weapons are all already optimized to this standard, some even below 10k), that is 100k vertices. How did you obtain 86 parts and 158776 vertices? Perhaps this could be some area we missed and need to optimize in.
Unfortunately most meshes that are split are this way because it is needed from a gameplay perspective, such as for effects of radio equipping / unequipping, mag dropping, attachments being attached, etc. This is the case with most morphs and weapons, at least those used the most. We are aware some are still in serious need of reworking and optimizing, but those are mostly unused due to being outdated.
We will be using PBR on new assets, thank you for the tip.

The lag observed with night vision goggles mostly comes from the fact that the avatar update spans multiple frames rather than a single one, due to the Motor6D not having achieved its goal. Is there no way to have it so Roblox only invalidates the character once it has reached its target, or have it be throttled such as is normally the case where updateInvalidatedFastClusters usually only happens every so frames?

This is a really tricky problem for us because our game is already a result of massive optimization work over its development time, and issues like this are mostly out of our control, with solutions that are not really applicable for us (without completely degrading the entire game).

The main issue I’m trying to point out here is not even updateGeometry anymore, it is the fact this is being called every frame when Motor6D is actively changing the part - it truly kills the frames.

[fishrotR]

Hi @MetatableIndex ,
I was testing following your instructions, which is to join the test Subject Team, spawn as a default avatar, type in “!nvg” in chat, then I’ll have all the equipments.

The number of parts vertices I got is from Microprofiler. This includes my avatar and all weapons and equipments attached to my avatar. This may include some items that you didn’t count but is actually attached. If you need, I could dump a list of parts and number of vertices of each part and share with you. However, even with 100k per avatar, that is still too much.

Number of parts is less a problem than number of vertices, so if you need to split those parts for gameplay purpose, you could probably focus on reducing number of vertices.

You are correct about updateInvalidatedFastClusters. How it works is that it has a 4ms budget. Any update that failed to fit into the window will be process in the next few frames. And every single update of the Mortor6D transformation will trigger an update. I know it sounds stupid, but this is how it works in Roblox Engine right now, and we’ll try to solve this problem in future. An workaround, as suggested by one of my colleagues, is to not parenting goggle parts to the avatar and use constraint to attach it to the avatar instead. In this case, an update of transformation on goggle won’t trigger invalidation of the avatar, because the avatar is no longer the ancestor of the goggle any more.

[MetatableIndex]

Thanks for the quick and detailed response,

@fishrotR

This may include some items that you didn’t count but is actually attached. If you need, I could dump a list of parts and number of vertices of each part and share with you.

This could be useful for us if you’re okay with taking the time for that, we’d be grateful for that. You can probably follow up about this in PM if you do proceed with it.

@fishrotR

An workaround, as suggested by one of my colleagues, is to not parenting goggle parts to the avatar and use constraint to attach it to the avatar instead.

I guess this makes sense with what we are experiencing. How should we go about doing something like this? It seems a bit out-of-the-norm for us as we are unused to having accessories parented outside of avatars. Would this work by having a folder for all goggle parts and applying the welds and Motor6Ds to these instead so only a singular main part is welded to the avatar?

Thanks for the help!

[fishrotR]

Yes, I suggest creating a dedicated to put all goggle parts including mount, lens, etc., and weld them in correct order. Then attach the base goggle part to helmet with an constraint. Could you give it a try? If there is any problem with this workaround, we could follow up in this thread or PM.

I’ll try to get a list of parts and #vertices and will PM you with that later.