OK, after the stupid post, here is some useful info (possibly)
The video stream alway has to have keyframes encoded into the stream to provide the FF capability.
Check out http://nickyguides.digital-digest.com/keyframes.htm for an explanation and fix.