I've frequently run into a problem, and I'd like to know how to work around it. As always, this BBS is a gold mine of minds, and I'm sure someone here can help me.

Often, I want to exchange large files with a friend. Only one friend, directly, not share them with the world.

The following options work, but they all have limitations:

- We can email the files. But the limitation is that many people's mail servers have caps on the size of file attachments, either incoming or outgoing, or both. I hit that problem far too often, and splitting the files up into multiple emails is unwieldy.

- We can use the direct-file-transfer feature of instant messaging applications (or heck, IRC DCC for that matter). However, these don't work behind firewalls, and rarely work behind NAT. Often, the person on the other end of the line is behind a firewall/NAT.

- We can run an FTP daemon on our local machines, but those also run into the problem of being behind NAT or behind a firewall.

- We can use some FTP space that's been kindly loaned to us from one or the other's ISP. But these usually have size caps on them, so really big files can't be exchanged that way.

In most cases, the person at the other end of the line can chat with me via one of the IM services (I use Trillian so I can be on all of them), but the direct file transfer feature rarely works.

Ideally, I'd like for the IM clients to have file transfers that just worked behind NAT/Firewalls. But failing that, what other options to I have for file swapping big files? Anyone?

Hmm... Tricky...

I usually use FTP for anything that big, as long as it does not go over about 2GB...

Is there a way to encapsulate a file transfer and send it through as plain ascii within your IM session? Not using a file transfer mechanism, but through the regular text stream - just like the "send text file" option in hyperterminal.

I'm behind firewall/NAT and I don't have problems sending/receiving files using AOL IM (Trillian actually, but using IM via Trillian.) I think you need port 5190 open on both sides.

This link has some more info on setting up my particular router.

Text messages in the IM clients usually have a limitation per-message of just a couple of K or so.

And being the FTP server is nice if you're not behind NAT/Firewall. If you are the FTP server, and you're behind a firewall, no one can see you or connect to you. And borrowing space from someone else's server has limitations.

I think you need port 5190 open on both sides.

Like I said. Doesn't work behind a firewall.

You're assuming, with that statement, that both people on each end of the transaction have control over their firewalls. Assume that one or both do NOT have control over the firewall and that it's a draconian corporate firewall that only allows HTTP to get through.

Oh geez, then you're screwed.

Heh, thanks, Tony.

Why not install Apache, and set the file to transfer in the http-doc directory then kill Apache when you don't need to transfer a file? Unless you want to get fancy, you can have Apache running on whichever machine is sending out the file. That is a simple, but pretty ugly solution.

If HTTP gets through, that means port 80 is open, right? Why not just set up a FTP daemon (only when you want to send or receive a file) that listens on port 80?

Chances are that if the firewall is paranoid enough to only allow HTTP traffic in then it's going to allow access to the official webserver only. This is also not taking account the problems caused by NAT or masquerading.

- Trevor

The last time I used Yahoo IM, the file transfer service was indirect. It would upload to a Yahoo server completely before the recipient could start downloading it. And, the download was through http.

However, I would imagine that there is a file size limit due to it being stored first. Also, this was in the early days of Yahoo IM, so things may have changed.

Also, AIM transfers usually only fail if both people are behind NAT/firewall. I am using NAT and I can always transfer files to my 56K friends (but why would you want to).

Have you tried Fed Ex?

Sure, encode the file as a bunch of airbill numbers and recipients.

I usually use drive crypt to make a container that is triple blowfish encrypted, use a 128 bit key, name it something weird and put it on kazaa until the transfer is done, then remove. Klugey, but it works.

Only done it a couple of times. I usually just mail a CD. Unless of course, time is critical.

How big are these files?

Why not install Apache, and set the file to transfer in the http-doc directory then kill Apache when you don't need to transfer a file?

If I'm behind a NAT/Firewall, the person on the other side has no way of seeing or connecting to a web server or an FTP server on my machine.

use a 128 bit key, name it something weird and put it on kazaa until the transfer is done, then remove. Klugey, but it works.

Except when my recipient is behind a draconian corporate NAT/Firewall that doesn't allow things like Kazaa to work through it.

How big are these files?

Doesn't matter exactly. Files could be a couple megs up to hundreds of megs. This is something I just run into occasionally in my life with various people... I'm talking about finding a long-term solution to internet file transfers from behind firewalls in general.

If SSH is open on the firewall, you might be able to tunnel a web or FTP server over it. Or just use SCP.

Can you describe how those things are done, in more detail?

Tony, if this is just you and one other person, you don't do this every single day, and you don't mind using a slightly slower connection (640/256), I'd be happy to create an account for you on my server that you and your friend can share. You will not be able to use FTP, only SSH, but you can get a free wintel ssh client for non-commercial use that works just like a GUI FTP client.

Like I said, if it isn't 10's of Gigs every day and as long as it doesn't totally hammer my gaming bandwidth all the time, it's the least I can do for our heroic FAQmaster. Oh, and you'll have to clean up after yourself by deleting the files when you've exchanged them.

Also, being behind NAT/firewall is not that big of a deal, and only one of you will need to open the port. If you want to talk about doing this, I'd be happy to help. You can have an old machine running OpenBSD and your very own SSH file server in under a couple of hours. Very easily. With a proper OS (UNIX), an old, free P133 can totally fill a 10Mbps pipe, and I doubt you have that kind of connectivity... I can give you some suggestions if you want to go this route, too.

Send me an email or PM if interested.

Jim

Greetings!

And some further good stuff on ssh.

Thanks for the offer of FTP space. At the moment, I'm trying to find a long-term solution that doesn't depend on someone else's FTP server. However I will consider your offer.

How about using freedrive/xdrive/what-not harddrive-on-internet? They are (at least used to be) free up to so many MB and you can share files with other people. There might be some regulations on the use nowadays, but earlier they were very popular for sharing warez.

Stig

Freedrive redirects to Xdrive, which doesn't appear to be free. Although it does look like it would work for what I want if it were free.

What you have here Tony, to give a slightly different analogy is effectively the same problem as having two USB "slave devices" and trying to get them to talk to each other.
[in that you and your friend are behind NAT'ed and gods knows what else firewalls, and therefore cannot accept incoming connections,you can only make them - sort of how a USB slave device works]

The only way to achieve that will be by using the equivalent of a USB "master" as a proxy to communicate to both slaves.
[BTW: There is a USB experienced person on the board here who is proposing to do exactly this to add USB slave to slave capability to the Empeg - look in the projects board/section for more details]

Of course, the analogy runs out here as its actually a software issue not a hardware one, but then the USB slave to USB slave problem is mostly a software problem too.

But you could imagine a "IM" type of place, like a Telephone Exchange that does voice conferencing where two incoming calls are routed to a common "place" where both parties can exchange files and whatever.

I would have thought that one of the existing IM services would offer this sort of thing.

Hey wait a minute - what about "Microsoft NetMeeting" or similar "service" - yes its MS software/protocols (well sort of), but from memory the Netmeeting protocol has up to 65535 "channels" that can be used (with the right software) - so surely you could concoct something up using Netmeeting - assuming both parties are using NetMeeting compliant software - not sure about Linux, but windows has lots of NM clients.
It also supports file transfer, real time chat, shared white board amoungst other features (including shared audio/video).

You're right in your assessment.

Yes, the IM programs should, in theory, just do this natively, but I've not yet been able to get the file-transfer features to work from behind firewalls.

I could try netmeeting. I wonder if it would have the same problems that the IM programs have from behind firewalls.

50megs.com?

Just checked 50megs.com and it looked good until I hit this.

most zip programs can split, i'm told.

And I can split files into multiple emails. As I said at the top of this thread, that's unwieldy and I'd like to avoid it.

You only said that it was unwieldy, not that you wanted to avoid it, until now

Basically you need a rendezvous service , I guess. NetMeeting might be it.

I've never used it.

Can one send/receive files with NetMeeting?

yes, file transfer is built in

Wonder if it'll work better than the file transfer in all the IM packages I've used. I'll give it a shot.

I'd tend to do this in Unix, but I believe that PuTTY can handle SSH tunnels. You can download it from http://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html
Look for tunnels in the docs.

Putty is an SSH implementation, from what I gather. How can SSH help me connect directly to another user when we're both behind firewalls? Don't we need an SSH server for that sort of thing?

Ah, good point. I forgot that PuTTY is just a client and one of you would need to be a server.

Could you run a web or FTP server on a high port that isn't firewalled (assuming the FW isn't stateful?)

Are you both behind firewalls that are uncontrollable? Or saying, at least, that we should assume that?

Tony,

I have about 110 gig free (RAID 5 with nightly backups) on a T1. Unlimited transfer unless it gets abused. PM me for an account.

Are you both behind firewalls that are uncontrollable? Or saying, at least, that we should assume that?

Ideally, I'm looking for a long-term solution that will work no matter who I'm communicating with. At the moment, I'm behind a firewall that I can control, but I can't always assume it'll be that way. The person at the other end of the line may often be behind a firewall they can't control. So yeah, for the purposes of this exercise, let's assume that one or both of the parties is behind an uncontrollable firewall.

Thanks for the offer of server space. I've also had offers in private messages. I will keep these offers in mind. Ideally, though, I'd like to find something that didn't depend on someone else offering server storage space.

Thinking about it (which hurts too much this close to new year ) I think this is actually the opposite of the USB scenario, you have two devices that are both able to connect to other things (websites) but cannot be connected to, this makes them the same as a USB host, so all you need to fix the problem is a USB LinQ

But seriously there should be a pretty easy fix that should be small enough for you to host almost anywhere, assuming:
- The upload mechanism the same as the one used here for attachments works OK for both of you
- The download mechanism that most places use where you click a link, it displays a page showing that a download should automatically start, then the download starts works for both of you.

What then happens is the file 'receiver' goes to the a webpage, and a download is triggered 'automatically' at the same time (well give or take whatever the download timeout in the browsers are, I am guessing you may have a couple of minutes) the 'sender' chooses the file and uploads it from another webpage, each uploaded chunk of data gets immediatley sent to the person dowloading so there is no data actually stored on the server. Actually that does sound a lot like the USB solution, something dumb in the middle just forwarding data.

I guess it should be a fairly simple task for someone who knows that sort of stuff, or has acess to the upload and download examples I gave, I may be interested but I am hacking USB at the moment

To be generic, if that's the case, then you can only be assured (and even then, not really) that each of you can make outgoing TCP connections. There's no way to match those two outgoing connections together without external help. And since you don't want to rely on the kindness of strangers, your only choice is to rely on the kindness of some corporation that provides such a service.

Right now, you're trying to do that with IM services, but the file transfer stuff they use happens not to work with your firewalls. Ideally, they'd just multiplex the messaging and file transfer on the same stream, but it doesn't appear that any current IM service does that, not even Jabber. Any other facility out there is likely to charge for it.

You could probably figure out a way to actually get two outgoing TCP connections to join up with some deep magic, but I imagine that's outside the scope of your request, and even then might fail depending on how your firewall rewrites ports.

So, for right now, I think you simply can't do it.

Funk TCP/IP. Can anyone say Xmodem?

Almost everyone has a phone line and modem collecting dust.

There's no way to match those two outgoing connections together without external help.

Good point. That does seem to be the crux of the dilemna.

Ideally, they'd just multiplex the messaging and file transfer on the same stream, but it doesn't appear that any current IM service does that, not even Jabber.

Yup. That seems to be the crux of the problem with the IM services.

You could probably figure out a way to actually get two outgoing TCP connections to join up with some deep magic, but I imagine that's outside the scope of your request, and even then might fail depending on how your firewall rewrites ports.

Ah, but if it could be figured out, and one could write such a utility, I think one would have a very popular little applet to sell. Hm. Wonder what that magic would be? Perhaps something that piggybacked on an existing IM service's connection.

Fair enough. But transferring a file that's too big for many email servers over long distance at something less than 56K could be quite expensive. It also requires a significant amount of effort, which was one of the prohibited things in the original RFP.

How about writing a pair of perl (or python, since Tony's already got that installed) scripts to chop up the file and send it via email and then reassemble on the other side? Still too much effort?

File chopping works, and I don't need Perl or Python to do that (just a zip program). It's just a pain with the multiple email messages.

This is probably the best solution, as there isn't any way with two NATed connections to get them to talk to each other. (Bug or Feature? You decide) The problem of course with using a server in the middle is that the file has to be transfered entirely to and from the server.

The problem stems from how NATed connections work. If you're behind a NAT proxy, paths to your computer only become opened up when you send a packet out, or when you set a port-forwarding rule(which is eliminated by Tony's no-firewall-conrtoll mandate).

Does anyone know if/how a P2P client gets around this? I know kazaa works for me behind a NATed connection, but I'm not sure if transfers between two NATed connections work out or just fail.

What I was thinking of was figuring out how to make the incoming and outgoing ports match on both sides beforethey were actually sent. It's likely that that would be in violation of the DMCA, though, and therefore probably unsalable. It would also depend on knowing how the firewall rewrote those packets. Since they often rewrite port numbers in addition to IP addresses, you'd still have to have someone on the outside looking to see what those new port numbers were in order to generate them properly on the other side, and if both firewalls did that, which isn't unlikely, then you'd be totally screwed.

However, you could figure out a way to write a custom IM app that was specifically built to transfer files via the IM service itself, but most IM services have a jabber-killing feature whereby if a client talks too much, they get cut off, so you'd have to pipe it through slowly, which wouldn't be ideal, either.

Perhaps something that piggybacked on an existing IM service's connection.

Perhaps you could have something that could do the following:

a) packetize your existing file into nice chunks
b) ascii armour it
c) spit the chunks over the existing IM connection as ascii data
d) reassemble the chunks on the other side.

You know, the xmodem idea isn't that far fetched. The only difference is that instead of dumping the data block over the phone line, you would be dumping it over a text IM session, probably proxied via http...

Still, it sounds like reinventing the wheel. If only the IM clients actually worked for sending their data files across the existing sessions without trying to open a different port...

If Tony controls his firewall, he can run a webserver off his machine (as I do). I have a Win2k box that sits on my DSL line running Xitami. I run the server on port 81, so some anonymous h4X0r doesn't easily stumble upon my machine.

If he doesn't want to do that, maybe he can ask his ISP nicely to raise his quota. I had an agreement with my former ISP, CyburbanLINK, that I could upload a huge file to my webspace as long as it was taken off in a timely manner. Local ISPs are usually very nice.

I am actually behind a pretty tight firewall (even the IM clients don't like it) and the upload and download examples I gave still work for me, if they work for Tony then that would be where I would start looking.

It's likely that that would be in violation of the DMCA

Isn't *everything* in violation of the DMCA though? I took a dump yesterday that resembled the cover to The Rolling Stone's "Tattoo You" album and I got a letter in the mail today telling me I violated the DMCA.

It does sound like a good idea. Still requires a server to sit in the middle, but it's not as bad as asking for FTP space from someone.

Tony,

I think you are going to be stuck with someone in the middle however you do this, as neither end can be directly conected to. If you have the program/script/webpage yourself you can host it wherever you want rather than using some IM piggyback that dies when the decide to change their 'private' protocol.

If you have the program/script/webpage yourself you can host it wherever you want

But it requires write-able disk space somewhere, though. Still, I might be able to work that out with my ISP.

I got lost in this thread just after tony said large file

In my line of work we don't use large files anymore we use air powered grinders

ROFL

hi Tony,
If you want to go the file chopping and delivery via e-mail route...

There is a commercial product (for windows PCs only) called Mailrules developed here in New Zealand.

I've used the product and it works well.

One of the features it has is the ability to automatically zip up and send entire folders/directories (or just single files if desired).

It can automatically Triple-DES encrypt the outgoing files and will automatically chop up into smaller chunks large files and send multiple messages with automatic chunk reassembly and decryption at the recipient (any missign chunks are detected and re-requested), plus you (as sender) can get email confirmation back that the file(s) made it to the other end.

So you can get a end-to-end transparent file xfer using Email. This transfer can be "one demand" or scheduled. It can be used to do the equivlaent of auoitmating large ftp transfers just using email.

It will do sending and receiving and will poll selected folders for particular files, and when one is found, will then zip ip up and email it to the recipient using the rules you configured.

If you're interested, visit This Link for the details and a download of a 60 day trial version.

I must point out that I don't get any benefit from recommending this program - I only recommend it because its useful and sells well around the world.

But its only Windows for now.

If you do try it and like it, the guys are open to suggestions for improvements - just email them your suggestions - they may well incorporate them sooner than later.

I tried using NetMeeting recently, but I can't connect to their directory server.

The software is coded to use "http://directory.netmeeting.microsoft.com" as the directory server, but that site doesn't respond, even in a plain old web browser. Didn't respond last week, doesn't respond this week. It's almost as if it's been taken down completely. However I don't see information to that effect when I search the MS knowledge base on the topic.

Has Microsoft simply dropped support for the Netmeeing directory server?

This whole thread seems like a paradoxical question. You're asking for a general-purpose way to do something through a firewall that firewalls are specifically designed to prevent.
Without an outside data storage location (or one of the firewalls in question) under your control, you're SOL.