> Like... directly harming me by causing my insurance rates to go up

No, that would be the aformentioned "indirect harm". He is not harming you personally. Many things people do cause you indirect harm, from people talking too loud on their cell-phone on up...I think having freedom is more important than legislating every dangerous behavior in the world myself, but I am, apparently, part of the ever-increasing minority.

As for the other points you bring up I think msaeger's point about other dangerous activities such as, say, mountain biking, hit it well enough.