I think maybe I'm gonna re encode everything at 320.
Does anyone have a good argument against this?

If you've got the disk space for it (in other words, you've put everything on your player at 320 that you'll ever want to put on it, and you've still got room left over), then there is nothing wrong with encoding everything at 320.

Just remember that the bit rate is not the only thing that determines quality. There is also the lowpass filtering. If you're doing 320 for everything, you should probably shut off the lowpass filter.