Basically, the lyrics are read in a background thread instead of the main loop (including the button input) having to wait.

Maybe you compromized a bit too generously and that thread's at too low of a priority.

I did make sure I wasn't actively using ttsd (it was 98.5% swapped out), not sending IR, not typing in my telnet session, I hadn't recently touched any buttons on the player, nor had I just booted the player. pretty close to standard conditions.