If you just want to learn HTML, Ultra edit is fine. I picked up a book of html "programmer's reference" (Osbourne Press, Powell and Whitworth, 0-07-213232-9) a few years ago and find it really handy just to look up the correct syntax and attributes of any element. I used it when I was learning PHP too.

In my opinion, if you are developing a site to attract customers, you need the best you can afford. Which usually means having someone else do it. eg, I know how to hammer/drill/nail stuff, but I wouldn't do a shopfront.
That then takes the pressure off you to play without affecting your 'shopfront'. It's easy to make a site look bad.

I've found things like dreamweaver or golive handy to develop complex tables, get them just right, then use that as a template for the rest of my PHP code.

I think the idea for Knoppix is a great one. It would let you boot up your current computer into linux without installation or shared system problems. And you return to a known state of play each time you boot. If you can do the dual system (old PC running linux and main computer running Win) you can get more done and can easily switch to your windows system if something goes south with the linux.

Turning to apple is a good solution, but I don't believe that all things are in the same place as they are on a standard PC Linux install (correct me if I'm wrong). but all the commands are there.

It's an exciting and fun thing to do if you have the time.
Good luck!