Does anyone know if the MP3 format specifies some sort of padding where it's trying to keep the frame data to a multiple of some fixed offset from the start of the file or something?

No. The size of an MP3 frame depends only on the bitrate and layer and other fields of the header. There is no leeway for padding. If two applications disagree about the size, one of them has a bug (but note that Lame, for instance, adds an "Oooh look encoded with Lame" block after the last frame, which doesn't begin with a frame header so is interpreted by players as loss of streaming).

Peter