I've noticed this "heirarchical playlists" thing being mentioned all over the place, what is it?

If you have an empeg, you already know what they are. If you don't have an empeg, there is a picture of hierarchical playlists here.

Note that the picture shows the playlists being simple artist/album trees, which isn't very useful on the Karma because it can sort by artist/album already without playlists. But playlists can be freeform as well. Also note that you can link to a sub-playlist multiple times and they are "live", so if you update one copy, then all the other copies are automatically updated as described here.

For example, I have a playlist called "Low Key" that's for quiet background music. In many cases, I have copied individual tunes into this playlist, but there are a few complete albums in there, and those are linked sub-playlists. And I have a playlist called "Oh Yeah" that's for uptempo high energy stuff, which has similar sub-linked playlists. I have a third playlist called "Road Trip" for which "Low Key" and "Oh Yeah" are linked sub-playlists, as well as a few individual loose tunes.

Additionally, you can tag a sub-playlist with the ignore as child option for even more flexibility.

Oh, and can the Karma play mp2s?

Not sure. But first, a question for you. By MP2, do you mean "Mpeg-1 Layer 2", or do you mean "Mpeg-2"? Those are two different specs which might carry the same file extension.