Strange Linux networking problem
Posted by: andy
Strange Linux networking problem - 07/09/2005 19:42
I have got an odd Linux networking problem.
Two Linux boxes, called Dipsy and Vault (yes, really). Dipsy is an old, tired Redhat 7.1 machine, running on a Dell PII400. Vault is a much newer, Debian Sarge machine, running on a Via 500Mhz fanless box.
Vault can ping Dipsy perfectly. Dipsy can ssh to Vault without problem. All the other machines (Windows and Mac boxes) in the house can ssh and ping both boxes.
However, for some odd reason, Dipsy can't ping Vault properly. If you ping vault from Dipsy you get 80% or higher lost packets.
If you sniff the traffic between them you can see that Dipsy sends the requests and that Vault responds with the replies. For some reason that I can't fathom Dipsy doesn't see these replies.
Does anyone have any idea what is going on ?
Posted by: andym
Re: Strange Linux networking problem - 07/09/2005 19:47
Something freaky with Dipsy's network card drivers and ICMP packets?
/Backs away quietly
Posted by: pgrzelak
Re: Strange Linux networking problem - 07/09/2005 20:01
Any kind of router / switch involved? Given the age of the machine, it might only have a 10Mb card and you might be losing something with either autonegotiate or a speed mismatch.
Posted by: andy
Re: Strange Linux networking problem - 07/09/2005 20:01
I guess so. It doesn't have any problems pinging other machines though, just has problems with Vault. It also didn't have problems when Vault was a RH9 box.
Posted by: wfaulk
Re: Strange Linux networking problem - 07/09/2005 20:04
Does a tcpdump on Dipsy itself show the ICMP Echo Replies? (I assume that you're sniffing the network elsewhere right now.) What I'm getting at is maybe it receives the packets but doesn't notice them? Clutching at straws here. Anything look odd with MAC addresses? How about the switch they're connected to?
Actually, you didn't say if they were on the same network or not.
Posted by: andym
Re: Strange Linux networking problem - 07/09/2005 20:10
Maybe try and compare the ICMP replies from Vault with other machines on your network?
Posted by: andy
Re: Strange Linux networking problem - 07/09/2005 20:19
I was sniffing remotely, but using tcpdump on Dipsy has the same result. You can see both the echos and replies. MAC addresses all look fine.
They are both on the same network, connected via a couple of switches. I've power cycled the switches.
This problem has actually been going on for months. I had assumed, wrongly, that Vault's network card was playing up after my RH9 -> Debian upgrade (on the basis of the ping problems from Dipsy). Vault has been sitting gather dust for a while due to this.
It is only when I plugged Vault back in again today to try and get it working that I realised what was going on.
Vault is actually supposed to live 30 miles away in someone else's house, which is where it is going tomorrow. Hopefully putting a chunk of the Internet between the two machines will return Dipsy to sanity (if it doesn't I will be really puzzled).
Posted by: tfabris
Re: Strange Linux networking problem - 07/09/2005 20:21
Quote:
I had assumed, wrongly, that Vault's network card was playing up
From your initial description, it sounds more like the problem is with Dipsy's network card or drivers.
Posted by: andy
Re: Strange Linux networking problem - 07/09/2005 20:26
Quote:
From your initial description, it sounds more like the problem is with Dipsy's network card or drivers.
Yes, I realise that now. The reason I was suspicious of Vault was because I had problem in the past where the network card in Vault would drop packets when it had a memory DIMM that it took a dislike to (which was another odd problem).
Posted by: wfaulk
Re: Strange Linux networking problem - 07/09/2005 20:35
So a tcpdump on Dipsy shows the Echo Replies, but Dipsy doesn't actually see them? Weeeeeeeeeeeird.
Does Dipsy have more than one IP address? Well, more than two, including 127.0.0.1?
Posted by: andy
Re: Strange Linux networking problem - 07/09/2005 20:38
Ok, I've found the problem, but I have no idea why it causes this symptom.
Dipsy is my primary DNS server. Just over a year ago Dipsy was moved from connecting via one ISP to another, so its IP addresses changed.
Dipsy's /etc/resolv.conf contains three nameserver directives, the first is its own IP address. The other two are the ISPs two DNS servers.
I had forgotten to update resolv.conf after the ISP move, so the first nameserver directive was Dipsy's old now useless IP address.
For reasons that I don't understand this causes Dipsy to ignore 80% of the ping replies from Vault (even though tcpdump shows them arriving).
If I remove the bad nameserver from resolv.conf or put the correct IP address in the pings go back to normal.
I didn't believe this to start with, but I have verified that this is the case.
I don't understand...
P.S. in both cases dig returns the correct IP address for Vault
Posted by: wfaulk
Re: Strange Linux networking problem - 07/09/2005 20:45
Does your ping have an option to not do name resolution? If so, try that. Maybe it's not so much that it didn't receive the packets, but that it was having a hard time looking up Vault's PTR record.
Otherwise, that makes no sense. Actually, it doesn't make any sense regardless.
Posted by: andy
Re: Strange Linux networking problem - 07/09/2005 20:49
Quote:
Does your ping have an option to not do name resolution? If so, try that. Maybe it's not so much that it didn't receive the packets, but that it was having a hard time looking up Vault's PTR record.
Telling ping to not do name resolution does indeed solve the problem, as does pinging Vault's IP address instead of using its name.
All very odd.
Posted by: wfaulk
Re: Strange Linux networking problem - 07/09/2005 20:54
Sounds like you've got a bug in ping. I'd try downloading a different one, compiling it up, and trying again, just to satisfy myself.