If the entire map is one big mesh, you’re going to have a problem. The collision shape of a mesh is different and less exact than the actual shape of the object–you just don’t notice it because meshes are rarely big enough to cause a noticeable difference. It’s similar to how cutting a hole in a wall with CSG would make it impossible to walk through.
You could try breaking down the mesh into smaller chunks, but I’m not sure how well that would work. A better option, though tedious, is to disable CanCollide on the mesh and then use transparent bricks to build your own collision hull.