I found a solution. Kind of. One of the solutions Google themselves propose allowing less secure apps to access the account. When I enable this, POP3 access works. If I disable it again, it stops working again. Very strange this seems necessary, especially since everything is happening in the very same google ecosystem.