It's going to be really hard for any of us to narrow down the problem just due to the way YouTube, and the modern internet in general is set up. There isn't just one YouTube server or datacenter somewhere, it's spread out all over Google's network mixed in with everything else they run. I don't know exact details on their operations, but it's likely run similar to a CDN. Edge servers sit closer to users, avoiding hops over major backbone transfer points. Those edge servers contain the videos popular in that area, and carry lesser requested ones only when demanded. And once a problematic video is found, a later refresh may play just fine since the edge server now has it fully cached.

The other issue is effective bandwidth. YouTube isn't hit as badly by this compared to more real time internet applications, but it still can be a factor. The higher the latency, and the more hops involved with a route, the lower the maximum bandwidth is going to be. Even if you have a great connection from the ISP to your house, it's possible that you aren't being routed to the proper "internet close" edge server, and instead are being sent to one with less then favorable latency possibly with packet loss issues.

Google does seem to be trying to address it though, as recent revisions to their Flash player have several diagnostic options in the right click menu, including report an issue. Might be worthwhile to use, since Google is likely to have much more data on the situation at the time of the problem. Just a shame it's harder to report issues when using the HTML 5 or mobile players.