I had a tape adaptor that had a gear between the driven and non-driven reels.
I assume it is to avoid this problem.. It did cause a bit of extra noise though, so I had to turn it up a bit.. =)

However, make sure if it has autoreverse that you are playing the correct side.. =)