What I'm saying is that there's not enough energy in the cellular data signal to move a speaker cone on its own. (God I hope not, or else the folks worried about cellphone cancer might not be so crazy after all.) So saying that the cellphone makes interference noise with your "speaker" is a misnomer.
In order to get those dit-dit-dit noises to come out of a speaker, the phone has to be causing the RF interference at the amplifier stage or somewhere prior to the amplifier stage. For instance, if it is near the amplifier circuitry, near the line input circuitry, or near a live microphone transducer.
In many cases all of those circuits are near each other anyway. For example with self-powered speakers, with a car stereo, or with a computer laptop speaker system. So it *seems* like the phone is interfering with the speakers.