Interesting. If I use this Anker adapter for my HDMI adapter instead of the Mac Digital AV adapter, then everything works the way I expect it to work. It drives the HDMI switch the way I expect it to, and works correctly and displays a good image.

So this is a problem with the Mac adapter product.