Well, if you have a vague suspicion that it may be the nvidia drivers, try updating.

Otherwise, the only thing I can think of at the moment is to try disabling any swap space. Perhaps the machine is hanging up on swap i/o?