Optimize huge internet file transfers

I have a client who shoots and edits video from a home office in DC for for various tv news programs in Europe. He needs to get files - frequently of several gigabytes - delivered electronically. Currently he ties up his computer for hours at a time on these transfers by uploading to clients' ftp servers.

We're going to set him up with a basic machine in his house so he can dump files there at LAN speeds and let it deal with the time to move data at Internet speeds. Whether it's an ftp client pushing, an ftp server his clients pull, or something else entirely is still up in the air.

It needs to be fairly idiot-proof for the recipient news departments, too, which is why I'm mostly looking at ftp. Is there a faster protocol that has widespread support?

He mostly works on a mac, but has the odd windows machine floating around, too. Am I missing anything else obvious?

thanks,

-jk

Maybe something like Dropbox?

tanstaafl.

Dropbox is obvious, but I think Cubby from LogMeIn was built with this situation in mind. I haven't used it very much, but I believe you can share any file/folder on your hard drive and generate a link for it.

The file that's outside of the main Cubby drive won't be sync'd to the cloud first. It just lives on your hard drive. Then, when someone outside your LAN requests the file, that's when it's transferred over the internet.

Don't quote me on that, though.

We've contemplated it. It would require subscribing to their paid service to have the capacity needed. And he'd still have the need for a secondary box.

-jk

Looks interesting, but they seem to have a 2 GB filesize limit. That would be a problem for him.

thanks,

-jk

I beta tested a feature from Egnite (https://www.egnyte.com) coupled with my ReadyNAS. It allowed me to throw data onto the NAS in a specific share, then have that share mirrored to another NAS or computer somewhere out there. Had quite a few options including making it available from "the cloud". Something like this may work out well for your client.

Make sure your client is fully aware of his bandwidth limits. If he's in DC there's a good chance he's on Comcast. Uploads count toward your total as much as downloads do, sadly.

ftp might have some annoying limitations and features, but on the speed issue you won't actually find anything that is significantly faster when you are talking about just copying a file from one location to another

(the same doesn't apply when you are doing something like syncing a set of changing files, when of course things like rsync that do diffs are far faster)

If the client has something like an ADSL line with low upstream speeds, they might be running into the problem of the upstream bandwidth being saturated for long periods which then breaks everything on the line that is trying to use TCP. When that happens you really need a way of throttling the upload speed to keep it from fully filling the line.

What do you mean by "he ties up his computer for hours at a time" ?

Uploading some files via ftp should have little effect on a modern computer, it should certainly not "tie it up".

I'm guessing it really means 'ties up his connection' since a fully saturated uplink will usually make ADSL unusable.

I think the best solution would be to employ some form of QoS so that he can still use his connection while he is transferring his work. Other solutions may be more convenient, but as said before nothing else is really going to be faster.

I've tried and failed to get QoS stuff working. In the end I gave up and got a second ADSL line instead.

I run my servers and BitTorrent client on one line and all my other stuff on the second line.

(that isn't the only reason I got the second line, but it was a handy side effect)

QoS works fine when configured correctly -- we use it here with good success. But very few people understand QoS, and therefore it can be very difficult to set up.

Edit: In this specific case, though, it's mainly upstream QoS that is needed, and that part is almost trivial to set up for the specific need in this case.

He uses fios and he reports a 2 GB transfer to europe can take a few hours, time he can't move his computer. He can use it for other things, of course, he just can't move it. He spends half his time out and about shooting video, the other half prepping it for broadcast.

A second computer is certainly the first step. I just want to make sure there isn't a significantly more optimized protocol than ftp and that his clients in europe could navigate. Copying via a windows share probably takes a third longer, and some of multi-channel file transfers need special clients. We're trying to optimize and keep it simple.

-jk

I've good a reasonably good grasp of TCP/UDP/IP (certainly better than 99.99%) of the population planet has). But despite many hours of trying I could not come up with a workable QoS setup.

The best I came up with was with everything throttled back to 25% of the upstream bandwidth, but with you only start with 800k that wasn't exactly workable

(and even then VOIP used to regularly get hit with a lack of upsteam bandwidth)

Though I'm not sure QoS on my Tomato router was working quite as it was supposed to, so that probably didn't help.

Instead now I have my CrashPlan backup throttled (within the service itself) to 45% and BitTorrent throttled to 30% (again within the BT app), leaving some space for SMTP/DNS/HTTP even when BT and CP are running at full pelt (which isn't that often). And I have a separate line for all our laptops/desktops/iPads/iPhones/Apple TV/set top box/etc

Works much better than I was able to achieve with QoS.

Agreed. And you shouldn't even need to mess with the router to achieve it.

It should be doable in either the ftp client or server (depending on whether he is uploading to a server or sharing the files out). Certainly things like the filezilla client let you set an upstream limit.

Just because it's "fios" doesn't mean "fast speeds".



What's the uplink speed on his fios plan?

If it's "25" or better, then a 2GB upload should take less than half an hour, unless fios also throttles, or if the other end is the bottleneck.

Empegger and BBS user Aleksander Milewski ("N6mod") used to work for a company that specializied in solving precisely this problem. I'm having a terrible time remembering the name of the company. Anyone else remember it so we can google it?

Basically they were trying to make a cost-effective and time-effective alternative to movie production companies and SFX houses having to ship hard disks full of video files to each other. (Because that was the way they were doing it at the time because it was faster than file transfers over the internet.)

It worked by having PCs at both ends of the link doing a continuous UDP stream that handled the data transfer communications in a round-robin fashion where the client and server didn't have to wait for a response before streaming the next set of packets. It could effectively saturate any connection up to its maximum available bandwidth, something that most standard file transfer protocols can't do due to the latencies involved in their handshaking protocols.

I don't know his plan yet, but I suspect the limit is not within fios, but rather somewhere across the atlantic.

this sounds interesting! I'll see what I can find.

-jk

Are you thinking of Aspera Tony? http://asperasoft.com/

Used that at a past employer, and it was pretty quick, to the point that it would saturate the uplink when possible.

Yep, that's the one!

It requires the whole client/server setup, but, if you control both ends of the link, it will give you the fastest possible transfer rate between those two points.

That does look interesting. It looks like they may have a web based on-demand version that I'll look into.

thanks!

-jk

Just to show that podcast advertising occasionally gets through, I offer Sharefile (a frequent TWiT advertiser).

However, I have no idea if that service would be of any use in this situation...