I defrag about 4 times a year on the machines I can't rebuild (corporate assets etc) and always notice a speed improvement. If you don't add/remove files much it won't be as necessary, but if you do a lot of installing and uninstalling you're actually better off rebuilding once a quarter or so (which is what I do with my home machines)

Of course all the above only applies to Windows machines. The filesystems I use under *nix have little need for anything like this, and they also cope better with uninstalling correctly.

YMMV