If the web was filled with only banner ads, I probably wouldn't bother. But there are far too many sites that still use popups, use irritating animated ads, ads that use CSS to obscure the entire page, etc. That said, even the static ads often cause problems because, as Matt said, they often significantly delay the loading of a web page. (I'm pretty sure that Doubleclick must be running all of their ad serving off of a Pentium 90 on the other side of an ISDN link.)

Basically, it's a workflow efficiency thing. The fewer UI interactions, especially ones due to unrequested data, the better.