Right, that's how it should be. If you delay the right side, it's like moving the right speaker AWAY from you, so your stereo image center moves to the LEFT.

And vice versa.