The VFD I chose (
GU256X64D-3900, because it's slightly under DIN-sized and -shaped, and has exactly twice the res of the Empeg in each direction) has a max speed of 38,400bps. If you want faster, pick a different display.. the 7000s are 115.2kbps, although as far as I remember it won't really work that fast because it starts reporting itself as busy and ignoring you if you're using the speed to do bitmap drawing. Some models (3900 included) have parallel input too, so you could theoretically use a parallel<->serial converter to drive it at 115.2 or 230.4. Drawing speed wasn't particularly important to me, aside from being an interesting optimization problem. I don't need visuals when I'm driving, and most of them look lousy in 1-bit 'color' anyway.
The display uses only the serial output, and the joystick uses only the input. There's no conflict. Some of the VFD models can send data back to you, but they don't tell you anything particularly useful.
You don't need to hack the kernel to make it work, although it would be helpful for a variety of reasons (suppress the serial output on boot, increased priority to update the display, etc).