I think you're right here.

If you look at the way that most cars do their buttons or dash illumination, they usually use milky white silicone plastic, with a colored LED behind (or white LED+filter).

Brian, have you thought about contacting the people who make replacement cell phone keypads to find out what compound they are using? I suspect that they are silicone-based too, but you can get diffuse translucent colored keypads for cellphones, so it is possible....somehow.

But as Tony suggested, colorless translucent buttons with a colored filter over the LED may be the way to go. Although you'd have to find someway of putting a filter over the LEDs, I'm sure that it would make the button casting aspect easier for you as you wouldn't have to create different colors for different people.