I've been messing around with the different build options for libmad and have found that using --disable-debugging gives a noticable performance boost without compromising audio quality. If you build with --enable-speed its faster still, but this is not such a good idea on a piece of hifi kit.

I've noticed that both the rioplay and rrr source distributions include a larger (260k+) libmad.a which is what I get if debugging is switched on. The no-debug lib is approx 108k. I don't know whether the binary distributions are the same, but if so, changing them may speed things up a bit.

Cheers
Paul