Try it this way round....
When the player displays a movie, it only displays the changes between the previous frame and the current frame.
Imagine a movie where the background never changes and has a person talking.
The first frame will have all of the scene, but each subsequent frame only has the changes from the previous frame. So in order to FF, the player must read all the way from the very first frame.

The reason for the keyframes is to give the player something to display when you fast forward.
Without the initial frame it doesn't know how to display anything without first interpreting all of the frames that came before were you want to FF to.
So if you don't have the keyframes, the player would have to literally decode every frame in order to FF, which most likely wouldn't be that fast.

That means that the keyframes are just a full frame from which the following frames can change from, then all FF has to do is display all of the keyframes in sequence and not bother with the others.
The downside to keyframes is that because they are a full frame, the filesize is usually much larger than a file that only has a few fullsize frames. Which is why you will not be able to fast forward with most of the downloaded movies, they filesize penalty is to high.

I hope that made sense, I get the distinct impression that I was rambling again