yeah, its a bitch out there now.
As I run a server at home as well, have file and email scanner running 24x7 as well as a port watcher on the firewall.
You would scare the crap out of yourself looking at all the hits on the port watcher! I log all occurences and IP address and look for patterns just in case.
Not to be too paranoid but...Any regularly connected PC, especially DSL and Cable internet folks are open doors for all kinds of attacks. Many of these attacks are not just some script kiddie but a concerted effort by external forces.
Just check out the homeland security/government perspective on hacking and viruses.

A widespread virus or worm is an effective tool against a capitalist society based on computer money. Imagine if your PC is about to be used as a stepping stone to hack into the FBI, Pentagon or Chase Manhatten.
Viruses are only the half of it.

Be careful out there people, to others, its a war.