Try clearing these bits... they're in my code, although I can't say I know exactly why.

port_opt.c_lflag &= ~(ECHO|ECHOE|ECHOK|ECHONL|ICANON);

Also this probably won't affect you since you're talking to the VFD both ways, but while we're near the subject, the cfsetispeed and cfsetospeed actually do the opposite of what they say (how I look at it anyway).. ispeed is the speed of stuff leaving the empeg and going to your device, ospeed is the speed of stuff coming into the empeg. If yours are different (at the time my VFD was running at 38.4 and my joystick 19.2) and you set it the way that sounds right you will waste hours trying to figure out why the hell you can't output correct data to the display or read keys from your joystick.