How should I handle network ownership in my building game?

I’m making a game with a similar building mechanic to those found in “build to survive” games. A big part of this game is destruction of builds, due to this I’m having trouble with handling network ownership of fallen structures.

server ownership doesn’t work for my case as it’s slow and floaty which makes destroying builds feel jank. I want a way to set player ownership, I just can’t figure out how to do it without it being exploit-prone.

the first issue I’m encountering is just figuring out which player to assign ownership too. Say a chunk of a build falls, it’s likely that chunk has multiple blocks from different players so it’s not like I can just set it based off whoever built the falling object. I considered setting it to the closest player but I expect lots of scenarios where multiple players would be standing on/interacting with a falling object so that likely wouldn’t work. Updating ownership is an entirely different issue as I probably need some way to track all the fallen blocks server side to reupdate ownership when appropriate (fallen structures will likely be on the ground idle for awhile).

the second issue is exploit prevention. handing network ownership of a bunch of 2x2 stud welded together cubes to clients seems very risky, especially considering players can arrange the cubes however they want.

Has anyone dealt with an issue like this before? I’ve been thinking on it for awhile and can’t come up with a solution to even the first problem I mentioned and any hypotheticals I came up with were extremely exploitable.

update: after checking other games to see if I can figure out how they do stuff like this I think a better solution is simulating structures client side and then somehow updating to the server. I played around with this game as well as this game for a bit and they seem to have some sort of system where blocks are smooth for all players with the catch being they start to experience the traditional jitter that comes with network ownership related desync when multiple people are pushing an unanchored block. The only system I can think of that would produce these results is if physics was simulated locally but one client was picked as the “replicator” to get the final say when objects get interacted with.

I’m gonna try to implement this on my own but I still am not 100% sure how it would work and also think its still very exploitable.