How can I achieve letting npc climb a part to reach a player, or destroy that part to reach the player?

Hello, so I am making a game where you survive zombies and you can build walls to defend against the zombies, so I am using roblox pathfinding.

An example of an issue is that the player can lets say place down a ladder part, which allows for the player to climb upwards. But this ladder can also be used as a wall part to defend against zombies. Imagine the scenario were the player finds a cave, and decides to block the entrace by placing ladders infront of the cave entrance. I need to have the ability for the zombie to figure out if it needs to destroy that ladder to reach the player, or if it can simply climb it to reach the player.

I have tried using PathfindingModifier, and setting PassThrough = true on the parts, this way the npc can create a path through the walls and once the zombie is infront of that part it destroys them. But the problem now is that the npc cannot jump on the parts and climb them because it thinks its an cancolide = false part I believe.
So how can I make it so that the npc can detect if it needs to destroy the wall to be able to path to the player, or if it requires to climb that part up to reach the player without destroying it?

you could use touched event on torso and add tag ladder to ladder and if ladder has tag ladder then destroy the ladder