Don't i need a modem to use Hyperterminal?

No, you don't. Hyperterminal can work directly from the serial port, as described in that FAQ entry.

However, if I recall correctly, I think that Windows might complain about the lack of a modem the first time you run Hyperterminal. You simply cancel/disregard the box where it asks about a modem. I think that's something I should add to the FAQ, thanks for reminding me.

It started by repeatedly saying "marked" then the volume started going up and down and the tracks started to change.

That's very strange. This is with the default 1.03 software?

If you tried to apply a custom kernel to it, that could explain it. For example, if you tried to apply Mlord's 2.0b3 kernel against the 1.03 player software, strange and unpredictable things could happen (kernel and player should match versions).

But if it's stock 1.03, then that's a real surprise. I don't know what could cause that other than stuck buttons or some kind of hardware problem.