There's clearly something I need to learn about networking (well, there's tons I need to learn, but one thing specifically at the moment).

As I've mentioned before, I support a network in DC that consists of a five-floor building with many student apartments. The network consists of a less than ideal WRT54GL with Tomato (they give me zero budget here), but that connects to a series of switches on each floor that then goes out to drops in all the apartments/rooms.

Last week the whole thing went down, and they called me in to find out what it was.

In the end, I sussed out that there was something connected to the network that was killing it, and I assume that it was a router because when I connected with my laptop I was getting a different IP than I should have been getting. The legit router gives out addresses in 192.168.168.x, and the mystery router was giving out IPs at 10.0.1.x. This additionally leads me to believe that it's most likely an Airport Extreme.

My question (sorry for taking this long) is: how do I prevent this? What can I do to fix it?

I currently have disconnected an entire switch on the network, taking out an entire half of a floor because the people who installed this network didn't label a damn thing (what's up with that, anyway?). I'm going in tomorrow to try to figure out who the problem person is.

Your computer should give you information about the IP of the computer that gave you that IP address. I know in XP it shows up in ipconfig's output.

Then ping that IP address. It probably makes no difference if it responds or not. Then look at the computer's ARP table (under Windows, "arp -a") and match that IP address to a MAC address.

Once you have the MAC address, you should be able to find what switch port the device is connected to, assuming they're not dumb switches. If they are dumb switches, you can at least use the MAC address to find the manufacturer of the NIC, which is often the manufacturer of the whole device.

To prevent it, you'd want to assign a different network to each apartment, and have them on different network segments, whether that's a different switch for each apartment, or, if you have decent switches, assigning a VLAN to each apartment. That way, they can only take themselves out. (You could also filter DHCP responses at the switch, but if your switch can do that, it can do VLANs, which is probably the better option anyway.)

If you don't want to do that, though, the only thing I can think of is to set up something that would try to suck up all of the offending device's DHCP leases, and that is going to be tough to implement.

LOL

Welcome to being a sysadmin. Every company I've worked at has regularly had their networks killed by DHCP servers (or routers with DHCP capability) being plugged into it at one point or another.

This is one of the few things that kills networks real dead for real long times and makes the sysadmin panic real real bad until he finds the problem.

By the way: A bit of googling tells me that the way you harden your network against this sort of thing is to use managed switches and enable a feature on the managed switches called "DHCP Snooping", which allows you to firewall DHCP requests and only allow them to reach your own trusted DHCP server, and ignore all other DHCP requests.

Thanks guys. I'll see what I can find out tomorrow.

My suspicion is that some kid wanted to get wireless in their room, but didn't have a clue how to do it and ended up connecting the WAN port to the wall jack in his bedroom. Ugh.

Sadly, I do NOT have managed switches. This is, after all, an organization that in the past has told me to get the least expensive thing that'll get the job done. It's a wonder I was able to spring for the L in the WRT54GL Nevermind that the poor little router has to support over 130 college kids. I'm amazed it hasn't exploded yet. It's sitting right next to another one (and a couple cheap switches) on a wooden ledge in a tiny closet with no ventilation.

Maybe it's time to go to the customers and ask if they are willing do drop some bucks into the kitty for the necessary hardware.

Don't know how much would be needed but $10/per would raise $1300.

What I'd do

Turn off the legit DHCP device
Get an IP from the dodgy unit
Find out the manufacturer based on MAC address of the dodgy DHCP
Ping it
Break out the separate floor switches until the device disappears so you can narrow down the floor which you've done
Go door to door until you find it (assuming the ports coming in at the floor switches aren't labelled)
If you can't find it (locked room etc), just keep pinging it and unplugging individual ports at the floor switch until it stops responding. At least then just that port is disabled.

More likely it's a LAN port (not WAN) that's connected to your LAN. If it's an Airport perhaps it's set up to join a wireless network and provide that on the Airport Ethernet port. i.e. configured the wrong way around.

Could easily be an Airport Express too. It can bridge both ways.

Well, just point out to them how much downtime they had, how much your services cost to fix it, and the price of decent switches.

D'oh! That's what I meant, of course!


Yeah, that's what my plan for tomorrow was. As you might imagine, they're probably a little hesitant to have me going into the apartments of 21 year old college girls! Usually someone goes with me even if there's nobody home. They have to cover their buts too.

But I'll be doing that tomorrow. Fortunately, I already narrowed everything down to about 12 ports on one of the switches on the problem floor. If I have to I'll test all 12. I could just unplug one cable at a time and see which one gives me an IP.


Thanks for the tip. It'll be helpful if I actually find the offending device to be able to tell the admin staff that it's the right one.

Thanks for all the help, guys. I really appreciate it.

Based on suggestions from Bit in an earlier thread, when this particular organization moves into the new space they're building at the moment, I've recommended some higher-end equipment that'll be able to do the things you're suggesting. I gave them three price levels to choose from and was shocked when they went for the highest one!

Thanks, everyone, for your help. It appears that the situation had been resolved by the time I went back to reconnect the switches on the offending floor. Perhaps the culprit heard through the grapevine that it was a resident that caused the problem, and removed their router from the network.

But sure enough, last night the exact same thing happened again. I went in this morning and this time it was actually a different floor causing the same problem! Dumb kids! I started by narrowing down the floor, then the switch on that floor, then fortunately the cables were labeled for once so I could narrow it down to the room.

With the "arp -a" command (thanks, Bitt!), I was able to get the Mac. I knocked on the door, a guy opened up. I asked "do you have a router set up in here?" "Yeah, but it's not working! I moved it back to this other room here!" Sure enough, I walk back there and the cable from the wall is plugged into a LAN port. And there on the bottom was the matching MAC. Stupid kids!

Thanks again, folks. Hopefully this won't be a problem in the new building. If they'd agreed to the proposal I'd made a year ago for putting wireless in the building, we wouldn't have had this problem...

Sorry for a third post in a row, but I wanted to ask you guys about Firesheep.

This won't be an issue in the new building for people on WiFi (I'll be setting it up with WPA2). But wouldn't people on the wired LAN be vulnerable to this until these services start finally using https?

It will be an issue because they'll have the keys. It isn't quite as simple to do but its possible to do. Encryption on WiFi just stops outsiders from accessing your network.


Normally you don't get packets for other machines on the network. The switch just sends you packets for your own computer however you can force it to send you everything or to start masquerading as the gateway.

Until you get some new switches and redesign the network, you're going to have issues. You need to separate each apartment from each other. At the moment, anybody who is plugged into the network is on the same subnet as everybody else.

The only reason anyone on a wired LAN would be in danger of having their traffic sniffed is if there is a hub somewhere in the setup (highly unlikely these days) or a man in the middle attack is set up. Switched networks only send packets to the devices that need them, thus the sniffer tool Firesheep sees nothing. Man in the middle would require a computer(or hacked router, etc) with 2 NICs set up to route transparently, and would only impact any computers downstream.

WPA at a minimum is needed to secure against Firesheep on WiFi. You seem to have that covered with plans to go to WPA2. Keep in mind though WPA personal is still vulnerable to other issues, and it's trivial to add support to some newer Firesheep release to work even on WPA/WPA2 personal setups. WPA enterprise is the next step needed to be a bit more secure.

If you secure the network hardware in the main switch room, and the uplink, the most someone could do is sniff their roommates traffic. At that point it's probably not your concern if it is happening.

You don't need to do man in the middle. You can ARP spoof the gateway or flood the switch with MAC addresses so it starts broadcasting on every port. Its very easy to do both with something like dsniff.

Didn't realize how trivial that was to do. Networking is definitely one of my weaknesses in the IT field, and I think for now, I'll continue to leave it to the experts to implement anything large scale and secure. I left HP before iSCSI and FC over IP was becoming popular, so I missed out on the training there. Fibre SAN networks were much simpler to secure and maintain.

I am amazed the whole thing survives.

I am in a not so much larger environment in terms of user number, and we have € 40,000 (or more) worth Cisco network equipment, and a 20 Mbps fiber optics data link.
Compared to your setup, it may look great, and still, without some serious bandwidth shaping, we would be in trouble daily in peak hours: bandwidth maxed out, p2p, and what not.

We do have some Graduate School population, here, which seems on average significantly more sensible than 20 year old kids, but still...


You do need better equipment and a redesigned network. And, I suggest you only give wireless access to residential areas, if possible That may save you some problems.

That quote doesn't really apply to what you're talking about, but yes, I'll be doing the things you mentioned. I'll be doing major QoS (knocking P2P stuff down to nothing as much as possible), and the entire building is nothing but residential, so I don't have any classes or offices to worry about.

I'm sorry, quote was too long. I just meant to refer to your new building and, in general, coping with users' doing unpredictable things in the network.

No class / offices / labs make things easier.

But still, most of our problems came (come) from residents and their personal laptops.

Our machines are configured so that is it ahrd for a user to create issues, and that proved good so far.

We are seriously considering Packeteer. So far we did not need to invest there and simple QoS worked well enough most cases, but we are adding more residential space (going up to 220 residents, plus 200 non-residents, plus offices), and personally I think it is very likely we'll end up with more trouble.
In an ideal world, I'd get packeteer and upgrade data link to 30 Mbps.

Keep in mind that by doing this, it may irritate some of the Blizzard game players on the network. Both World of Warcraft and StarCraft II use BitTorrent heavily for patching. They have fallbacks to using webseeding (torrent chunks delivered over HTTP), but it's noticeably slower then P2P on big patch days.

Thanks for the input guys, I appreciate all I can get. I'm not accustomed to supporting this kind of environment, so I'll be learning a lot of this as I go.

I'll think about the Blizzard issue, Tom, but for the moment it's not a top priority, sadly. Given how bad it's been in the old building, I'm going to do everything I can to make sure the basic needs are met.

I still can't believe they've been spending the last two years sharing 3Mbps with 120-150 college students. I can't begin to imagine how that works...

Don't do it. They're awful, especially at their extortionate pricing. Their logic for selecting traffic is so backwards as to make anything even slightly complicated virtually impossible.

Use anything else. OpenBSD's firewall, pf, for example, has packet prioritization, as do most of the other open-source firewalls, like ipfilter, iptables, and ipfw, I think.

Thanks for the advice on packeteer, Bitt.
Hopefully we can save that money, in any case.

It probably doesn't. We used to have 80 people on a 10Mbps link and it was hell.