Not often enough for me to really commit each time to memory. Most of the time I'll encounter Siri issues due to the phone wanting to hang onto a distant WiFi router instead of transitioning back to the 3G network. The way Siri reports this is the same as if the server is actually down, but trying it again when the phone swaps to 3G has generally worked.

Not really making an excuse for Apple per say, but they did indicate Siri is beta. I imagine they said this due to the inherent issues with voice recognition and needing lots of usage to iron it out. Along with Apple's weakness in server scalability compared to someone like Google. For me, the occasional downtime and issues aren't large enough to diminish the overall benefits it has brought.