Get a tester and test each point from the cumputer to the Swich.
Since you're not using a patch panel, it sounds really likely to be the line from the wall jack to the switch. The tester should tell you if you have pairs crossed. Get a nice one and it will even tell you which standard you are using.

I go with
WO|O|WG|Bl|WBl|G|WBr|Br
Hasn't let me down yet.
Maybe your RJ45s are not Cat 5 jacks?