If you mean the Ctrl-T thing, I'm pretty sure that's firmware that bypasses the regular boot process. I don't think it ever gets to the kernel. You'd have to rewrite the RAM test application altogether and then upload it to the flash, and that area may be write-protected.

That being said, folks have claimed that a new kernel would be required to get the player to use the additional memory once booted normally. The kernel should produce some output on bootup in regards to the amount of memory it sees. What do you see there?