Hmm. This may be due to Parts within the workspace having Automatic network ownership, so the closest player will automatically request ownership of the part, thus allowing them to control touches. If you want, try calling part:SetNetworkOwner(nil) (to set server ownership) and seeing if the client can then block touches. Additionally, check if GetTouchingParts still works as intended even when the client has deleted it.
It does sound like Rays are still the way to go though, unless you’re really focused on 100% accurate physics for swords, in which case, go for it.
What I just do is use the method that raycast hitbox modules used. I didn’t use the module, it was limiting me kinda (having issues reading others script lol) so i made one using my owner method. It work pretty well especially in sword combat, it feels so accurate. Not even a bit lag. Just make attachments inside your sword and every render step, make ray from previous position to new position for the attachments. Every time it hits to something with humanoid or a wall etc. just save them in ignore table so raycast doesn’t hit it back. Not sure if the way i did is best but it works well at the moment.