Last night I got the RPi up and running, built a toolchain, and did a quick "hello world" test of the cross-compiler. The frame buffer interface in Raspbian seems pretty standard, so I shouldn't have any problem driving it, though making it fast might take a couple of iterations.

One unexpected snag: it appears the RPi GPIO pins are 3.3V, while the pins on the Arduino are 5V. There are some ways to work around this, but it does give me more motivation to find a way to factor the Arduino out of the architecture and do everything I can on the RPi

The other option would be something like this, which is a stackable Arduino that's basically plug-and-play with the RPi. It's only in beta, but if they're close to production, that might be the simplest way to get things going until CAN on the RPI is farther along.