I think I know the problem when the player ragdolls these collider parts are created.
I am specifically using this ragdoll module right here:
[Updated] Perfect R6 Ragdoll - Easiest Ragdoll System for R6 Avatars - Resources / Community Resources - Developer Forum | Roblox
I will see if I can make these collider parts non-collidable and see from there.