If it's not RC5, you will need a different state machine in order to recognise the codes. You can run multiple state machines concurrently in the IR driver - it already runs one to recognise the front panel buttons (which are encoded as if they were IR) alongside the RC5 driver.

ISTR there may be something in Documentation/ which talks about the RC5 format.

You can also put the IR device into raw mode, where it just tells you the timings between changes on the IR line, which is useful to get an idea of what a remote is sending you.

Hugo