I’ve had many moments when I accepted a party (Friend or not), and get booted from a game I’m in 
Sure, there might be a point where you actually want to join them instantly, but can’t you just hit the follow button? (I mean, if you accept the party and the game is already full; it does no good to instantly join anyway vs clicking a button.)