Would it be possible to have it scroll only when you use multiple fingers up/down? If so, what about this:

All single finger gestures used for numbers and maybe other "complex" gestures.

Two finger swipes used for the four physical buttons.

Three finger swipe up/down to scroll, left/right to switch between remote and playlist views.

Maybe pinch zoom gestures for volume up/down?

With this, I think you could do the number 1 as a straight line, and there would be no ambiguity with up/down for scrolling.

Anyway, no matter what, this sounds awesome. Have fun at the beach