For what it's worth, a buddy of mine who works on YouTube mentioned:
... The OP could right-click on the video and access some debug information and a speed test which will help them get some more details on what might be happening.
If you think about it, it's pretty amazing that YouTube works as well as it does, given that (wild guess) it's not exactly Google's most profitable service. Why, I remember, back in the bad old days, when you'd put QuickTime videos on your home page, use a kludgy piece of HTML to link to the QT (via embed and object tags), and it would still not work very well.