I changed my 404 page as Tony suggested and things have improved a lot. However I don't think it was improved by reducing the size of the page, it looks like the frequency of the "attacks" is just falling off.

Even with the slim line 404 page I still get a peak with my up stream bandwidth maxed out every few minutes for 10 seconds or so as a couple of clients decide to pound on me.