I think you've just had odd luck with the "upgrade" path you took. The Athlon system was around the time AMD was noticeably ahead of Intel. AMD was making major inroads with servers at the time, and the Tyan motherboard and setup was likely a very solid platform.

The Xeon at 2.4ghz doesn't give me a full idea of what exact Xeon it was. Some Xeons from that era were derived from the Pentium 4, a chip that did have performance issues. To drive the clock speed up, they kept increasing the instruction pipeline. Apple had an explanation of this in their keynote a number of years back (prior to their switch to Intel). http://www.youtube.com/watch?v=PKF9GOE2q38 is worth a watch. For your earlier AMD system, the pipeline was much shorter then what was likely in the Xeon.

The Dual core Pentium 4 chips were two physical processors chips placed into a single package. Beyond the general issues with the P4 architecture and the pipeline mess, this package setup led to other issues. Often, these processors were placed onto motherboards that weren't fully built to support multi-core processors. And since they were still two separate chips in one package, they didn't benefit the way proper multicore chips do. Proper multicore (one chip, multiple processors) can skip sending messages across a bus (slower then in chip messaging), and often share the same cache.

Oh, and the other issue that might have caused problems on the Xeon and P4 system, Hyperthreading. This was a trick Intel did to make a single processor core appear as multiple processors. If done correctly, this can boost system performance, since it helps utilize more of the processor. The P4 era hyperthreading was bad though, and at times could impact performance by a noticeable amount. Modern hyperthreading has been improved quite a bit, and better OS kernel support has also helped.

Thankfully Intel finally came to their senses and killed the P4, replacing it with the Core architecture. The initial Core chips were basically reworked Pentium 3 processors, with their architecture dating back to the very solid Pentium Pro days. The newer Core 2/i3/i5/i7 chips are a proper generational jump to a new architecture, one without the major design flaws of the P4 architecture.

Honestly, you need to move off Windows XP if you want to take better advantage of any modern multiprocessor (multiple chips, or multi core) setup. The OS plays a major role in how work is divided between all the processors, and the XP kernel is simply too old to properly account for modern processor setups. On the same multi processor multicore setup at a previous job, we saw a noticeable jump in compile speed simply by changing from Windows XP to Windows 7.