There might be a way to connect to touch events seamlessly without any getting corrupted. Sometimes I get a touch gesture that doesn't disconnect if I slide my hand across my phone screen. For now I disconnect all gestures when they reset. (I wish I could detect when they pressed escape/opened the menu so I can do it then)
Edit: For your case it probably has to do with using the button's ButtonUp event, instead of the UserInputService InputEnded event. The former probably only fires when their gesture is directly above the button, which isn't always the case while dragging.