Oooh. He's got so much RAM that no one process can map it all (unless both the sysadmin and the process jump through hoops). Perhaps the backup program is sizing its buffers by system memory size, and running out of address space in MapViewOfFile? Is it possible to tell Windows to boot with only 2GB of RAM and see whether the problem goes away?

Alternatively, take a look in the Processes tab in Task Manager and see if the "VM Size" of the backup program grows endlessly, until it reaches 2GB and things start failing.

Peter