I will second Matt's recommendation to make sure you've downloaded and installed the latest firmware from www.rioaudio.com . But also, you said:

I don't use EQ or the crossfader

Okay, but do you use the automatic gain control feature? This is another part of the player which affects the audio stream. I've never heard it cause the problem you describe, but I just wanted to check.

Next thing to check. Pause the playback and see if you can still hear clicks. If they go away when you pause, then yes, it could be a decoder problem. If they do not go away, it could be a hardware issue or something physical (like bumping the wires on earbud headphones).