I am somewhat puzzled by the idea of on-line backups.

I think I must have a fundamental misunderstanding of something because I can't see how it can be done in any reasonable time fame.

Take my system, for example. 4 TB of data = 32 Tb of data, upload speed averages 1.5 mbps.

32x10^12 bits divided by 1.5x10^6 bits/sec = 21.33x10^6 seconds = 8 months, more or less. How could I conceivably spend the better part of a year, 24 hours a day seven days a week, doing a backup? It doesn't seem feasible to me.

Am I overlooking something here?

tanstaafl.

Your math looks right. However, 4TB of data is way over the average amount of data most people have. The current online backup solutions are geared more towards the average user who has backup needs in the range of 10s of gigabytes.

Beyond that, some of the backup providers do clever tricks to minimize data transfer. One of them out there that does a complete system backup won't actually transfer files over the wire if they see a duplicate already stored in their system from another user. This cuts down on a lot of bulk data from Windows and installed applications, leaving just the users specific data going over the wire. Others just ignore all system/application files and only back up what a user has in My Documents.

One final trick I know of is that some backup providers allow you to mail them a drive for the initial backup, this ensuring only differences from that backup have to go over the wire.

For me, I only backup about 30 gigs offsite, with most of it sitting on a Time Capsule at my grandparents house two states over. The rest of my data I'm not overly concerned with losing. I do have local backups in the house to a RAID in a NAS, and if a large enough disaster strikes to wipe that out, I have bigger problems to deal with.

Most people with multiple TB's of data don't need to send it all up to a backup site because the content is already backed up on hundreds or thousands of systems around the world and they just need to fire up a bittorrent client to download it again.

It took me weeks to send up just my digital photos as I've only got 640Kbit/s upload capacity. Yeah, by 2015 the CRTC says I should have 1Mbit, but it's not proposing that be passed into law or anything.

Even if I had the required transmission rate, I've also got a 120GB per month usage cap. 4TB would take a lot longer if I had to meter it out. Probably over 6 years if I was going to be doing any downloading at all. 2.8 years if I was using up the whole cap for only uploading. Yuk.

I've got probably 100-200 GB of critical data that I'd consider backing up online. That's in addition to the local and offsite backups though.

You mean 640Kbits/s, don't you?

That's pretty grim, even down here in the wilds of Mexico I've got double that on a bad day, and as much as 1.9 Mbit/sec on a really good day.

I have the fastest internet connection it is possible to have here around Lake Chapala, at 7 Mbps down, 1.8 Mbps up. That is one third the speed I had in California, and the service fails on average once every couple of days for periods ranging from a few minutes to a few days. As a bonus, I also get 75 TV channels to watch, but since I disconnected my TV set five months ago, I don't get much benefit from that. I could have just straight internet (without the cable TV) but for some strange reason that costs about 40% more than internet plus TV. Go figure.

tanstaafl.

Doug, your math is right, but as Tom said, most people aren't backing up 4TB of data.

What do you have that's taking up 4TB of data? If it's DVD/bluray rips, then that's also not the typical thing that most people are backing up. If it's all photos, then dang, you're a pretty prolific photographer Good for you!

Also as Tom said, very few of these backup companies are still offering unlimited storage. Some are, but they still bank on the assumption that most people are going to go with them because they think they'll need the unlimited space, but end up only using a few hundred GB at most, probably around 100GB on average (which might still be a little high).

Also figure that while your upload speed is typical (for the US at least, I don't know about Mexico), there are others with faster speeds. I have nearly 5Mbps, and it took me about a week for my initial upload of around 300GB. I do wish I had Bruno's 640Mbps connection, though Are you in Kansas City, Bruno?

After the initial backup, the differential backups are quick.

Online backup isn't feasible for everyone, but I'm a big fan of it because it works well for me.

I still think it's good to have a local backup as well...

Ha! I accidentally capitalized a lowercase "k" as an "M." Woops!

I'm in the same situation as Doug, but I've "only" got about 3TB of data. Most of it is indeed DVD rips I've made of my DVD collection. However, this took so much time, I really never want to do that ever again. I would love to back it up somewhere offsite, but as Doug says, it's just not feasible at current speeds.

Upon reflection, there's really only about 2.5 TB that would require backup, the rest is on-site backup of the original data and I obviously wouldn't be backing up my backups.

A great deal of my data is audio (my music collection, every single episode of NPR's Car Talk since February 2002, more than 3,000 audio books, plus photographs (mine and SWMBO's), games, income taxes, Rosetta Stone, etc. I have 6.5 TB of capacity, of which (including backups) about 5 TB is being used.

No doubt I could go through and get rid of at least one TB of data I'll never need nor look at again, but with hard drive storage so inexpensive ($80 for 2TB at NewEgg) why bother?

It looks like my best solution is to get a couple more 2 TB drives (WD20EARS, probably) to use in my external dock as backup drives and keep them at my neighbor's house.

tanstaafl.

I've been investigating and trying out Crashplan (I think the recommendation came from here). It kind of gives you the best of both worlds. You can back up locally on your LAN but also remotely to a friend or to the Crashplan servers. If backing up to a friend you can "seed" it doing a local backup and then transfer the backup somewhere else. Once seeded it only backs up differences. All of this can be done free and probably in return you could back up your friend if similarly inclined.

So far I'm pretty impressed and I've looked at quite a few "cloud" only solutions all with certain limitations that were frustrating.

Elephantdrive - slow uploads (they wanted me to break it into multiple uploads) and the software was using 600MB of memory/pagefile.
Sugarsync - just didn't get into it
Opendrive - limited versioning and backup type functions but economical
+ others

plus the many others that I reviewed but didn't actually install

Crashplan is definitely the new darling, easily beating out Carbonite.

Crashplan offers (for a not-insignificant amount of money) to send you a drive to back up to and mail back to them. I don't know if they do multiple hard drives, and if not, that still means only 2TB at a likely maximum, if not less.

When my Mozy plan runs out I'll be switching to Crashplan. I'm already recommending it to clients.

Crashplan seems like a super nice service and super cheap if you only have a reasonable amount of data to back up. Considering I have an unreasonable amount if I include everything, that wouldn't work out.

I think I'll stick to RSYNC from my ReadyNAS (which is the 2nd copy) to my web server for photos, but I might consider Crashplan for other document-type files which I also need to keep absolutely safe.

So, this brings me to the ULTIMATE question. Does anyone have a solution, commercial or otherwise of a truly completely idiot-proof photo backup procedure?

This is my ideal: A small box with an SD card slot connected to my parents' network. They pull an SD card from their camera and put it in the box. AUTOMATICALLY all the files are copied off the SD card, verified and then they remove the card. That box then uploads all the photos to a backup site. Which can be my web server or Crashplan or something like it. That box should probably have a drive in it to hold the copies. Optionally all the photos would also get sent up to some social site like Flickr or Facebook - but require privacy settings so we can restrict viewing access.

The bottom line is I'd love a way for them to be able to unload the images off their camera memory cards without interacting with a computer.

And if something like this doesn't exist, who wants to help me create this product and commercialize it?

Bruno, eye-fi does that, but wireless, so no box involved. Well mostly, it does not delete files.
In our case, photos are copied to local desktop, (if on home wireless, if not will do later) and Picasa (hidden by default),it can copy to other photo sites. From the desktop they could be backed to crashplan etc.

The camera does need to be on for an upload to occur, but if you have a load to do you can put the SD card into the USB reader and it will copy automatically from there.

edit

I second the Eye-Fi. I keep begging my mother to let me set her up with one, but she's resistant to technology change.

It's too bad, because for people like my mom, the Eye-Fi is magical. You simply bring the camera home, sit it on your desk, and power it on. All the photos are automatically transferred to the computer in a folder structure you define (I have it set for YYYY-MM-DD). Like Phoenix42 said, you then have the My Pictures directory (or wherever they're saved) set to backup and it's all automated. Much easier than having them move SD cards around.

I have one in my camera, but I'm probably going to move away from it because I want to shoot in RAW+JPEG, and RAW files take an eternity to transfer. JPEGs are pretty quick, though.

The other primary selling point of the Eye-Fi, though (at least the high-end ones) is the automatic geotagging. On the high-end cards, you get a lifetime geotagging service. It's not GPS, but it does an amazing job at estimating based on surrounding WiFi. Honestly, it's pretty impressive. It's fast, and it seems to get the location in even the most unlikely places. Anyway, it's a great feature for the parents.

I'm going to check exactly what model camera my mom has and the Eye-Fi (and me setting it al up for her) will hopefully be a nice mother's day gift. She'll be ecstatic at the prospect of more easily using her digital images.

Can you set up multiple backup destinations? For instance, can I send to FTP *AND* Costco *AND* Flickr *AND* xxxxx ?

I want to say yes, but I'm not certain, I'll verify with my card if I can configure an additional web target beyond what is currently configured above.

Even just two or three targets. I definitely want to send to FTP at a minimum, but also sending elsewhere would be great. Somewhere easy where I can teach her to do something with them. Perhaps Facebook if I can set the default album to be completely locked down and then give her the option of sharing individual images. I'm pretty sure she knows how to move around somewhat in Facebook (the new AOL)

At least with the version of the Eye-fi software currently on my machine, you're limited to uploading to one home computer and one online destination. FTP counts as an online destination, so if you upload to FTP, you can't also upload to Picasa, Flickr, and friends without some hackery. Some folks on the Eye-fi forums have suggested Pixelpipe as a multiplexer for the online services,

I almost always want to clean up my photos before uploading them, so I just have it deposit them on my file server via FTP and do my uploading to Picasa using the Picasa desktop app.

Hardware: A small little Atom-based system (eg. one of the Zotac HD-ID* boxes), or even a netbook --> say, a $200 Acer model with SD-slot.

Software: Ubuntu Linux (configured as "read-only"), and some simple shell scripts to monitor the SD slot and do the uploads.

That's what I would do, were it me and my folks.

Cheers

Mark, a small little tiny hand-sized system is what I was thinking. But I'd be looking for something that I could install and configure at most. I don't want to spend any time writing scripts nor inventing the technology if it's not something I'm going to turn around and sell. Mainly because it could take away a lot of time/energy from other things I'm already doing. Plus if my wife found out she'd kill me. Lots of projects around here to finish still.

How to set up PixelPipe with Eye-Fi - the instructions were easy to complete, even though it's unsupported by either Eye-Fi or Pixelpipe officially. There used to be posted on the Pixelpipe blog, but that link has apparently been deleted.

So far I've tested relaying from Pixelpipe to my photo-specific web site via FTP. Very slick. The Eye-Fi software has a pretty crappy UI, having been written in Adobe Air and not though out well enough, IMO. But it does get the job done. I'm surprised the company hasn't set up its own service to pipe photos to multiple sites at once.

This *should* work out well for my mom. So long as she doesn't lose her camera again. I'm having to go a month early on her birthday gift as well by getting her a new camera to use with the Eye-Fi I just got her for mother's day.

Wow, that is scary.

I've been looking at some cloud-based backups in the last days, and I remebered reading this thread a while back. So, Crashplan does look like the best out there, I think, but I am a bit concerned about security. I may be paranoyd, but I'd really NOT want Crashplan be able to look into my files, irrelevant as they may be. So, does anyone know how their service works at this regard? Clearly, file >transfer< is encrypted. But, then, once on their servers, I suppose they are accessible to their system admins? I'd love my stuff to be a bunch of meaningless bits to them, and nothing more.

Also, I too have 500GB of "valuable" files that I'd like to backup. I have a SHDSL, 2Mbps symmetric data link at home. It would still take forever to transfer all... But still, if the security part was convincing enough, I may be considering this.

I believe if you opt to apply the extra private password option then your data is supposed to be completely unaccessible to their admins. I think there is also an option to provide your own key. There is stuff in their FAQs that covers it

So very different to some of the other off line backup systems.

Taym, Crashplan encrypts your data before it leaves your computer to go to their servers or even to go to another computer in your home. This page has more info.

The nice part about Crashplan is that if you do have a very slow connection, at least it also includes the ability to back up locally as well. The application then detects which method will complete the fastest, and starts with that one. That way at least you're safe from hard drive failure and any environmental scares that would be localized to the area of the original data (burst water pipe over your computer, thief takes your computer but doesn't see system in another room, etc).

Thanks guys for pointing me to the right page. That's very interesting. I am seriously considering to set up some "reciprocal" backup with friends, to begin with.

Excellent idea, I'd forgotten that you could do that for free. Very good idea. Of course, you'll need to make sure they have connections that will accommodate this too.

There was an email to a podcast I listen to recently that said they'd backed up several hundred GB of data to Carbonite. His computer crashed, and when he went to restore the data, he realized he was VERY quickly reaching his bandwidth cap. He called Comcast and all they could tell him was to try downloading the data over the next few MONTHS. Ugh, caps are going to be a big problem very soon...

Yes, Matt, that concerns me a bit, I have to say.

Even assuming Crashplan has some recovery service where they ship a HDD or a set of BDs to you - which I don't think, or at least I have not yet read any such thing - still it would take time to get your data back, especially if you're not in the US. Your files at a friend's house is going to be available "immediately", if the whole thing is well planned.

I've been using CrashPlan for just over three weeks now. In the past I used Mozy and I've also for a long time used my own set of rsync scripts that backed up to some "unlimited" web space that I had access to.

Mozy never really worked for me, it took 5 months to do my initial backup and it just couldn't cope with very large files (my large virtual hard discs caused it to stall the backup for days). The "unlimited" web space provider recently spotted I was storing 100GB+ on their server and asked me to remove it. So I needed something new.

Over the last three weeks CrashPlan has backed up 60GB of my data to their cloud. That covers my photos, documents, source code, email and other important things. In comparison, it look 5 months to backup the same data to Mozy !

I still have 150GB of music and other assorted files to backup. After that I'll try backing up my larger virtual disc images (it has already coped with some 6GB ones that Mozy choked on).

I'm also using it to backup the whole of my Mac Book to my server, works well and is painless to use. It is also running within my main Windows virtual machine, continuously backing up all my local source code repositories (CrashPlan on that virtual machine is also simultaneously backing up to their cloud).

So it all seems to work pretty well. I am about to sign up for their $6 a month service (to cover more than one computer and various extra features).

It isn't perfect though, I have a few complaints.

The first one is fairly minor, it isn't very good at estimating the remaining time. It very naively assumes that if it has backed up lots of compressible stuff, that the rest of the backup set is similar. For example a few minutes ago it was claiming that I had 2.3 days left to backup 150GB...

Its bandwidth throttling is very inflexible. You don't get to vary the throttling based on time of day. Also the selection of speed limits are oddly restricted, for example you there is no option between 300kb and 1mb !? I'd really like to have it use very little upstream data during the day and have it fill most of the line over night.

It has, for me, a very serious bug/bad bit of design. There are a bunch of settings around whether you are backing up over the LAN (local network) or WAN (the Internet). Settings like speed for example. So you can tell it to use unlimited speed backing up to other local PCs and limiting the bandwidth used when backing up to their cloud.

All good, except they have screwed up their LAN/WAN detection. Rather than making use of netmasks, they decide that if they are backing up to a public routable IP address that it must be a WAN destination. So for example if I want my MacBook to backup to my local server, that will be seen as a WAN connection as they are both on public IP addresses. So I can't restrict the Internet bandwidth used by the Mac when backing up and also have it backup locally at full speed.

The final major negative point is the lack of account wide de-duplication of data. If you have two copies of a file on a machine, it will only back the data up once. If however you have the same file on two different machines and they are both backing up the cloud, that data will be sent twice. I believe they have plans to address this, which will be good.

I do wish that the local backups were stored as plain files, rather than stuffed into their own proprietary databases. But I understand why that is, as they actually backup blocks of data rather than files.

I have also had the backup stall a few times, for serval hours, seemingly because the server my client was talking to was having problems. Something I will have to keep an eye on.

You have to worry of course about whether it will stay unlimited, though they have at least been very clear to explain that they intend to stay unlimited and their reasons for thinking that they can when others have not. Only time will tell.

Someone pointed out to me that the $6/month family service is actually "as low as $6/month", but only when you pay for 4 years in advance.

You can enter a custom value of any number but I agree it's not the most flexible. I'm still trying to figure out a way to let it run full blast all the time and be throttled at my firewall/router.


Remember they are encrypted anyway. I kind of like the 4GB blob method. Hundreds of thousands of files can be unwieldy.


They do also say they will give a pro-rata refund if you cancel. Doesn't help if it's taken you 6 months to get everything up there though.


I paid the full 4 in advance for the cheap rate based on the pro-rata refund mentioned above.

There are some niggles I have with it but overall I'm happy. My upload speeds are fairly slow so it's taken me a few months to get to about 300GB "in the cloud". The real important stuff is done though.

I created separate jobs backing up the higher priority stuff first. One niggle there is I have a local and online copies. It does the local copy only first and then assumes that's enough until every thing else has at least one copy. I get the concept but would prefer the higher priority stuff always take priority for all locations.

Just to add my configuration.

I have a NAS running unRAID. It runs Crashplan and backs up to their online service. This is where the important stuff is (photos, MP3s, files etc).

My laptop, MythTV machine, work laptop etc backup to the unRAID NAS locally. Some parts also back up directly to the online service.

Crashplan can't backup its local backup store to anywhere else. Not really any point in that. Just backup directly.

Something I've found quite useful is the ability to restore through the webpage. Once or twice whilst away for work on a corporate LAN I've needed a file. I can't VPN out or anything so I just restore it via the webpage to the machine I'm on.

I did also find an issue when the linux NAS didn't have the right codepage causing Crashplan to skip a number of files e.g. "José González". Not really Crashplan's fault though I only found it by doing an audit of number of real files vs number it said it would restore.

Oh and another minor niggle. The backup page counts number of files + number of folders as "number of files"