Plugged in the analogue cable and it works fine.

I'm guessing that means that the digital connection wants a very specific timing. If I were you, I'd try to figure out what that was and set it up that way. It's bound to be standard across multiple vendors, as you don't have to do anything special under Windows depending on what type of monitor you have. Then again, I suppose the monitor might be able to tell the computer what timing it's looking for.