On Solaris, a good metric for if you need more memory is the "sr" field in vmstat. This says how hard the kernel is looking for pages to swap out. If it's consistently high, that means that the kernel is under constant pressure to swap and is having a hard time finding appropriate pages.

Maybe there's something similar in MacOSX?