This might be a dumb question, but indulge me - with a PWM amp, does the speaker itself act as the DAC?

AIUI (and CMIIW) the digital amp provides a nice constant voltage (if it's any good) and does PWM at a frequency too high to hear (and too high for the speaker to keep up), meaning that the speaker smoothes the PWM to pass the audible frequencies and not the inaudible harmonics.

The PWM is then effectively analog still subject to noise pickup, unless it's re-generated nearer to the speaker. IOW, it may be treated as a digital signal or as an analogue one.