For one thing, prestige begets prestige. Plenty of people apply to Harvard and MIT because they're big names that they've heard of, and, hey, maybe they win the lottery and get accepted. Most of them don't, a feature that pushes up the rejection rate, which makes them more selective, which makes them more prestigious, and the circle of life continues.

Furthermore, private colleges and universities are generally smaller in enrollment than public colleges and universities -- part of that is that the small private schools take a cue from their public sector counterparts in other industries and constrain the supply, so that they can be more selective, and therefore be more prestigious and... (see above)