If you don’t focus and then unfocus each client then the gamepad will control them regardless of focus. I remember this being a bug back when gamepad support was added. I guess it never got fixed.
I plugged my controller in before and after I started a Test Server as well as trying in a new place. I can still controll all the players in my test server.