You could do it with a normal physical switch easily enough as it's USB and is designed to be hot plugged. A rotary switch which does 4 pole 4 way should be okay. I seem to remember that the XBox has an extra wire for the joypads to power the vibration feature. Since you're using the Logitech receiver then you wouldn't need to switch this one.

It probably won't like you plugging in a joypad into the same port that you've got the receiver though.

- Trevor