Terrible Pathfinding Performance Related to Collisions

In my game, players are able to build while NPCs chase them. However, I noticed that terrible FPS drops ensue when certain (anchored) models are moved in/out of the render queue (set the CFrame of an anchored part with unanchored meshes welded to it). I couldn’t figure out what about the models was causing it until I tried turning CanCollide off, which completely fixed the issue. I’m not sure exactly why this is happening (something to do with the navigation mesh, I assume). Obviously, having to turn CanCollide on models off is a problem. Is there anything I can do? Has anyone else run into this issue? In the attached screenshot, the bottom three lines (from top to bottom) are:
getPrimitivesOverlapping
assemble
processPendingCollisionAssemblies/NOVSIGNAL

Assign parts to different collision groups. This way, you can control which parts should collide with each other and prevent unnecessary collisions.