Speakers blow because of distortion (bad signal, clipping, wrong frequencies)

but the minute ... the signal gets distorted it's gone

Sorry, can't agree on that - how can we reproduce a distorted electrical guitar?

IMO/E, two things kill speakers - going beyond their mechanical limits or going beyond their thermal limits.

Mechanical limits - running a way to large signal to the speaker (even short term) can make the coil move beyond its limits damaging the coil and/or cone.

Thermal limits - running to much power to the speaker may burn up the coil leading to shorts between turns of the wire or broken wire. Could also lead to failures of glues...

Take a 100W speaker and feed it a 100% distorted signal (square wave) of 10W and it won't complain one bit. The thing is that if you have an amp and speakers matched (say 300W each) and you drive the amp into serious distorsion it will put out more than 300W. If you compare a 300W sinus signal with a square wave of the same amplitude as the sinus signal, the square wave will be 600W.

It's not the distorsion that kills speakers, it's the associated increase in power - if you go beyond the power handling capability of the speaker.

/Michael