The mad decoder works much better. CPU is around 15-17% using mad instead of ffmp3. Resampling to 44.1hz takes an extra 1% or so.

mplayer -ac mad -af resample=44100