Another way of looking at it, is that only the particular process (web server, IMAP server, database server) at the other end can decrypt any of the message. So all the information that's needed to get the message to that process -- i.e., the IP address and port -- must be sent unencrypted.

Peter