Keep DVDs as ISO images if you want to preserve the menu structure (AnyDVD, DVD Shrink), otherwise rip the streams into an MKV container if you want to pare it down but not transcode (MakeMKV). And if you're sensitive to file size, transcode to AVC/H.264 in an MKV container (Handbrake). Those all seem to give the best results.

Best practice for filenames is also "Movie Name (year).ext" inside a folder of the same name minus the extension. That easily allows custom poster artwork and will later allow easily overriding metadata as well. "Movie Name" should match the name of the movies from IMDB.

Considering that the desktop version of the software is a BETA, they really should bring it to parity. That would allow a lot more testing on issues that are not specific to the Boxee Box hardware.