It's really just testing your video card at this point. The data you entered for the monitor doesn't really do anything other than limit the resolutions that XFree86 (the base GUI software for Linux) will display. If you're seeing a scrambled image, it's a problem with the video card settings most likely.

How is it scrambled? Is there anything recognizable on the screen, like the image has been shuffled? Is it continuously changing, like the vertical hold is off, or like an application is moving underneath?

My guess, even though I have little knowledge about digital monitors, is that it expects to see a particular frequency or set of frequencies coming over the digital connection, and your video card is sending something valid, but not what it wants. The fact that you're seeing anything at all is a positive sign. I'd try hooking it up to the analog connectors first and see if you get anything useful. If so, I'll bet that if you set the refresh rate to 60 Hz, the digital would start coming in, too. I'm unfamiliar with RedHat's setup for this, so I don't know if that'd be an immediately obvious selection or not. You might try fudging the monitor settings to tell it that it can only do 60Hz vertical refresh.

So maybe it's the monitor settings after all, now that I've gotten this far.