maybe I am having endianness issues and am just SOL

Maybe you are having alignment issues and are just SOL. The ARM cannot load a 32-bit quantity from a non-4-byte-aligned address correctly (it silently gives the wrong answer). This often comes as a surprise to x86-trained kernel developers; in particular it took me a lot of hacking on ReiserFS to work around those issues in 2.2.

Peter