I'm at the end of my rope here.

I'm doing a little consulting work for an organization in Washington D.C. They are essentially a "study abroad" campus for a well known university, and their building is more or less an old apartment building in the Cleveland Park area of D.C.

The building its self contains a few classrooms, a newsroom (there's a journalism aspect to the program), about 6 offices, a computer lab, and then four floors of student housing.

Currently, the building's internet is supplied over a bonded T1, so 3Mbps total. This (and half this) would be fine for everything I just mentioned except for the student housing, which consists of about 145 people. During the day, the offices get along fine, but in the evening the students complain of horrendous speeds.

The initial plan was to eliminate one of the two T1 lines, and put the students on something faster but less reliable. Well, we got that. Comcast came in and the connection was MUCH faster, but had an uptime of about 50%, and the company would not tell us what the problem was.

The next solution was to install a Covad-supplied DSL line. The speed was slower than Comcast and a little more expensive, but they had an SLA, so I figured it would be more reliable. The problem was, the guy who installed it was surprised they even sold us the product, because we were half again too far to receive the 6Mbps connection we were hoping for (which was slower than I wanted to begin with).

So that's where I am right now. We're back to the 3Mbps feeding the whole building. The students are rioting, and FIOS is approved for the city, but rollout probably won't start until the end of 2009, and who knows when it'll get to this part of Cleveland Park.

The campus has asked me for a solution by their fall semester, and they're not looking to spend anymore than they are now for two T1 lines.

Any suggestions?

^^^ Root of problem.

The likely issue is that 145 students are swamping the upstream, and this is going to be a problem with pretty much any solution that involves a conventional broadband connection. All it would take is a few people using video chat to kill the entire connection. Add to that the students likely use of peer to peer programs, and one or two people render the connection useless for the rest.

You clearly need not only a decent connection, but a good solution for traffic shaping. Home broadband connections were simply not built for 145 people to share.

Pulling random numbers out of the air time...

Lets say 75% of the students are online at any one time. That gives us 108 students. Minimal service I'd say would be 56k dialup for downlink anyway. That gives you roughly 6Mbps just to get those 108 an equivalent speed of a 56k modem. You'd also need to traffic shape the crap out of that link to stop a handful of people hogging the entire link.

How much is bonded DSL in that area? Whilst you may be too far for 6Mbps over a single line, can you get >6 Mbps over multiple lines? Install a proxy that blocks P2P or throttles it down.

Yeah, seriously. If you throttle all P2P traffic, throttle FTP downloads, and throttle Youtube and other video/audio streaming, the existing T1 line would probably be just fine.

For the cost of one of those T1 lines, you ought to be able to hook up about four (or more) of the DSL lines. Figure 3mb/s each, that's 12mb/sec -- quite an improvement over the T1 they replace, and you've still got the other (reliable) T1 there too.

MLPPP (multi-link PPP) is one possible way to "bond" the DSL lines, if the ISP can support it on their end. Otherwise a Linux box doing round-robin (or even smarter) routing can spread the bandwidth around.

Cheers

Thanks for the suggestions.

Any advice on how to do some of this traffic shaping? The student network is handled separately from the rest of the building, so we can get extreme on the solutions. Currently, there's just a WRT54GL running Tomato. I've attempted to set up QoS as best I can, but I've not had any experience doing that, so I really don't know what I'm doing with it. Through QoS, I've tried throttling P2P traffic, and I've limited the upstream bandwidth to around 1400Kbps.

Would you guys recommend another piece of hardware, or can I get away with what I have? That would be ideal, because as I implied, these folks don't want to spend a whole lot.

Excellent idea, but I neglected to mention that I wasn't even getting 3Mbps when the guy set it for that speed. The performance was terrible. It was getting around 300Kbps down and 70Kbps up. That was with my laptop plugged right into the modem they supplied. Even worse, the connection kept dropping. I could stare at the link light on the modem for a minute and watch it go out several times.

Since you said "FIOS", I'm guessing that means that you're in Verizon territory. I don't know if it's because of their isolated terrain around here, but Verizon service is outrageously expensive. You need to find a competitive carrier that has their own loop.

I just moved my office from a single T1 from Deltacom via Verizon to a 6Mbps symmetric from TW Telecom for an increase of about 40%. I'm not necessarily recommending TW Telecom, but you just need to shop around.

Actually, it's worth pointing out that getting the same T1 from Verizon directly would have also been about 40% more, for the exact same service. Well, it would have gone through Verizon's network instead of Deltacom's, but it's still the same bandwidth. So even just finding a regular CLEC may save you some money.

The DSL problems *might* be manageable with the right hardware (Thomson SpeedTouch brand modems) and the right line profiles.

The trick is to first use the Thomson modem to find out what the line quality is (detailed info from DMT tool or from the web menu of the modem itself).

And then convince the ISP / telco to set a correct DSL line profile based on that info. Otherwise, it will be set too fast, and will lose sync over and over, perhaps never stabilizing.

For example, our 5.2km DSL link here was set up with a 3008/800 kpbs ADSL line profile originally. Couldn't hold sync for more than a few minutes to an hour at a time, even as it automatically downshifted to slower and slower speeds.

Attenuation is 61dB downstream and 31dB upstream. S/N fluctuated between 3 and 10db, both up and down.

We got them to force the line profile to 2496/640 kbps, and to switch it from "fast mode" to "interleaved". This stabilized it, and it regularly lost sync perhaps once every day or three, rather than hourly. Big difference!

Now, it's even better, of course, and hasn't lost sync over the past seven days since the noisy adjacent line got disconnected. I might even ask for a faster line profile if this continues to hold steady.

A related idea is to physically position the modems as close as possible to where the wires enter the building, to reduce self-inflicted line noise.

Cheers

Just about all of these modems/routers that do QoS are doing it with a Linux kernel inside. The QoS / Traffic-Shaping is a stock kernel feature on Linux, so look around at the available documentation on how to use it.

Eg.
http://lartc.org/
http://lartc.org/howto/lartc.qdisc.html

The knowledge gained there can be applied to Tomato firmware setup, or used on a dedicated Linux box (any old/new hardware will do) that you could install to manage the bandwidth and routing.

Cheers

I'm surprised your WRT54G hasn't let out it's magic smoke already.

Here's a randomly chosen cricket graph of one of our residential /25 subnets at berkeley. So, about 100 people, we do some light bandwidth controls at the building level, campus does some more when we hit the edge of campus depending on where the traffic is going. That subnet of approximately 100 people is averaging 5Mbits a second over the day, and we still get complaints about speed occasionally.

If you want this to work well in the long run, they need to be willing to pay for a fast connection, real routing hardware, and real traffic shaping hardware. It's not going to be cheap. In the outside world, people pay $20-40 a month for their home internet connection which usually serves at most 4 people. $10 a month times 150 residents, and you should be looking at spending 1.5k a month on the connection for residents pretty easily. The residents are probably paying close to a thousand dollars a month to live there, and the internet connection is probably in the top five things they care about right after electricity and food.

A cheap or old PC running Linux with the necessary cards in it will do both of those with no problems.


The two T1s aren't cheap. They're significantly more expensive than a home connection.

Is there free software that can do the traffic shaping even once the P2P users start using random ports and encrypted traffic ? (without bundling all traffic not to/from well know ports into the same shaping profile)

Is there commercial software that can?

What types of service are available near you?

Do you have any cable internet in the area apart from DSL? Although by the sounds of it if you can't even get 3Mbps reliably then you must be pretty cut off from a comms point of view.

I have never really seen DSL lines successfully used to serve large groups of people. The asymmetric properties don't lend them selves well to the situation. A university near me tried this to cut costs and it ended in disaster (and them spending a whole load of money!) after we (the telco) refused to visit site until they took the load off the lines.

The best thing I could suggest for DSL would be to install 1 line per 10 people for students, but that would be 15 DSL lines which would start making a larger backhaul circuit seem cost effective. There really isn't any other solution than to stump up and pay for a fibre into the building. Short term pain long term gain.

Cheers

Cris.

Mmm.. without looking too hard, I'd say one could configure Linux to do that with a bit of thought. And perhaps add a runtime script or two to self-adjust it on the fly as needed.

As usual with Linux, all of the necessary tools are there. But they likely would need a bit of scripting glue to fit any given scenario.

Cheers

We have a similar setup in terms of student population. More offices, however.

We have a 20 Mbps, bidirectional connection, on fiber optics, and last Spetember students were maxing that out with p2p.

Some user education and some tweaking of the Cisco main router made things better. And, we are good now, but we've also been lucky in the fact that student body are not being too "bad".

As you cannot rely of users not using p2p (and you don't want to depend on that, of course), however, I am ready any time to have to resort to Packeteer, which is very good and very expensive. I mention Packeteer because that's what we use in our HQ, and I never looked at anything else. Packeteer will do anything you ask. But again, it costs a lot of money.

Bullshit. I've used Packeteers at two different companies, and they blow chunks. They do the actual job of throttling very well, but the matching algorithms are near worthless.

Say, for example, you want to give a certain priority to a set of networks. In addition, you want to give a different priority to HTTP data. The Packeteer will match the "first" one. But there's no way to determine what the "first" one is going to be. Actually, you can force one to be first, but it also puts it before any other rules. So if you have anything even remotely complex, you're screwed.

You'd think that you could have matching rules and sub-rules. But you can't. So if you want to apply one priority to other campus networks, another to HTTP traffic, and another to HTTP traffic on other campus networks, you have to create three separate rules. This is a pain to create and to administer, but it ought to work okay, right? Well, no, because there is apparently no "AND" rule, so there's no way to say "HTTP and destination". I feel sure that I must be wrong about this, despite having looked for it for over a year. Let's assume I am wrong; then it should work okay, right? Nope, because there is a maximum limit for the global number of rules you can define, and if your networks are disjoint and you have to list a dozen each time, you run out really quickly.

I would never, ever, recommend Packeteer to anyone. Well, if it was free or cheap, it would be okay, but as Taym points out, it's wildly expensive, and the cost of your license is based on how much traffic it will deal with.

In summary, forget anyone ever mentioned it.

Speaking of free and cheap.. again, standard Linux can do that complex rule/subrule thing pretty much any way one wants to set it up.

It may still require a few duplicate entries to completely clarify (to the kernel) exactly how one wants it to behave, but it can do what you just described.

The links I posted earlier give the gory details, and yes, they're gory when used without a GUI wrapper. I'm sure the GUI/html wrappers exist (heck, that's probably how Packeteers does it), but usually a GUI also implies surrendering some functionality.

Cheers

Yeah, the Packeteer has surprisingly limited function in that regard. Pretty much any other system works better. And it has a CLI interface that doesn't add any features.

The only advantage it has (and, IMO, this is pretty minor, though my IT Manager thinks it's wildly important) is that it will inspect and classify all passing data so that you can see what's been going through your network and easily add a new prioritization.

Edit: I take it back. It has one really neat hardware feature. It functions as an ethernet-to-ethernet bridge. It doesn't route. If the system reboots or loses power or in any other way isn't active, it still passes traffic. I don't know how it does that, honestly. An 8PDT normally-closed relay comes to mind.

Linux firewalls/filtering gui:

http://www.fwbuilder.org/

I've not used it but it's the most comprehensive I've seen.

I've poked around with the older v2 of fwbuilder. Good for firewall rules (though I prefer just the rules without a GUI), but it doesn't cover traffic shaping / QoS rules.

Cheers

I've made a Ethernet tap before and it'll do exactly what you want. You need to dedicate two NICs to monitor the traffic however.

Would it? The linked article says it's useful for intrusion detection and sniffing (i.e., listening to the signal), but says nothing about being able to *modify* the traffic, like a firewall or traffic shaper would need to do.

From the looks of the design, it looks like it would just let you listen in on the conversation, but not take over the conversation, as it were. I could be interpreting it wrong...

Ah yeah. Ignore me. Got the wrong end of the stick. I guess you'd do it like Bitt said and have some sort of normally closed connection.