It's not even totally a bit reservoir issue. The fact that mp3s are encoded in frames that have a constant real-time length (that is, the fact that each frame must be 25.7 ms, or some such number) means that if your track isn't an exact multiple of that length, then it has to insert blank space. (BTW, the frame length varies depending on bitrate, sample frequency, etc. And that exact number I gave is totally made up.)

Also, in my experience, tossing the bit reservoir does something more along the lines of making the volume ramp up, rather than creating silence. Not that that's not equally as annoying.