I find it absolutely fascinating that the lag happened in the silent bit a the middle of a track.

I wonder if it's related to the code that does the visuals? That's the only part of the system (I can think of) which behaves differently when the player is playing silence.

Perhaps if there's no audio data to process, the code for the visuals goes into some kind of race condition, executing small loops really fast and taking over the CPU?