Keep in mind the common port numbers you see (80, 443, 22, etc) only represent what port the server initially listens on. The client can send the initial request to port X, but tell the server to then respond to the client on port Y. Each protocol can handle this differently.

Add into the mix NAT, where the router will talk out to the server on port X, ask the server to respond on port Y, then internally coordinate with your machine to expect an answer on port Z. It does so by modifying the layer 3 and 4 parts, while leaving the higher layers alone.