I'm listening to my music and a song by ALicia Keys comes on. I decide I'd like to listen to that whole almbum but in a shuffled order and not just hear 1 more song by her.

Press the "3" key on the remote several times. Details here.

This will do exactly as you say: Listen to the rest of the songs from that Alicia Keys album in random order.