Blender Animations Bone Constraints

I am trying to make an equip animation for this sword right now.

So as of this moment, the handle of the sword is constrainted using a bone constraint so that the sword can move with the torso. This is so prevent weird movement with the sword if the upper torso were to move, which it is doing so here.

The issue: In an equip animation such as mine, the hand obviously picks up the sword, however the issue is if i were to change the bone constraint to the right hand, it would completely mess up the entire animation; it would be as if I animated it entirely with the constraint being with the right hand. Here is the issue on video.

As you can see, if i change the bone constraint, it messes up the entire animation.

Question to the problem: Would it be possible to change the bone constraint WITHOUT messing up the animation. OR would it be possible to change the bone constraint based on a specific keyframe.