Long Term Data Backup

Yet again I am rethinking how I keep my clients pictures safe. At the moment I have at least 2 copies locally and one copy on JungleDisk (which is working really well). But this is only of the JPGs. I now need to off load RAW files from 2009/10 to make space, so in effect put them into archive.

I was thinking of just dumping them out to an internal hard drive and then keeping that in my new fire and water proof safe in the house. How long do you think the data would be good for on a hard drive sat in a safe not powered up ??? It wouldn't be totally critical as the final JPG images would be backed up elsewhere onsite and on JungleDisk too.

Also, can anyone recommend any really good backup software ??? I'd really like something that backs up JPGs to one place and picks out the RAWs and puts them somewhere else.

Cheers

Cris

What volume of RAW files are we talking about per year ?

At least 2Tb. Maybe more. For 2009/10 I only have just over 1Tb though. This year I have more than 3x the work of the previous 2 years combined! Hence the rethink on data storage.

Cheers

Cris

With that volume, hard discs do seem to be the most practical option. I think I'd want to two duplicate hard discs for each year.

The empeg community has prover that hard discs can store data for a reasonably ling period, how many of us have 10+ year old discs that still work and the data written 10 years ago is still readable...

Also I'm guessing that the nature of RAW files means they are good candidates for recovery even when hit by a couple of unreadable blocks.

Powering all the discs up once a year would probably be a good plan. You probably want one of this drop in hard disc adapters.

Yeah, a pop-up toaster style of dock, eSATA or USB3 interface to the PC, and a pile of bare drives (in their original anti-static bags).

2TB WD "Green" drives go for CAD$70 around the corner here.

After attending a seminar on long term data storage for broadcasters a couple of years ago, the only solution they were willing to recommend was tape, more specifically LTO. The guy gave specific examples of customers that had bought a disc, filled it up, stuck it on a shelf and a year later found the disc wouldn't spin up.

Of course, this recommendation was based on the fact the data these people have was 'worth' anything from tens of thousands of pounds to millions. If the long-term survival of my business depended upon the integrity of my backups then tapes in a fire safe seems to be the only solution. Otherwise, if you're only holding onto the data as a courtesy (non-contractual) service, then buying a load of discs may be the answer. But I'd definitely store them off site.

Yea that was my plan, I just fitted an eSATA card to the Mac Pro the other week.

My system is getting pretty clogged with data I hardly use but need to keep. I think I'd like to have one HD for each year from now on, seems to make sense. I'm going to use up all those odd drives I have knocking around first.

The safe is pretty cool...

http://www.sentrysafe.com/Products/277/QE4531_FIRE-SAFE_Water-Resistant_Safe

It has USB through the door, so I plan to have a 1Tb 2.5 drive in there as a 2nd copy location in Lightroom, so until the JPGs are output and uploaded to JungleDisk the RAWs are nice and secure, well as reasonably as they can be.

So what you are saying is that Data is pretty safe on hard drives stored in a fire and water proof safe ???

Cheers

Cris

Yea, it's non-contractural. The JPGs are safely stored off-site, this would be for the original RAWs. I was thinking of charging a premium to store peoples pictures at a premium, say £30 for 3 years, £45 for 5 and £100 for "life" (being the time that Cris Matthews Photography is running).

Actually Andy you'd a good one to ask because the RAWs of your wedding will be amongst those I am talking about. Would you be bothered if you came back after a short while asking for a few changes and I said I couldn't because I had lost the RAWs ??? (assume I hadn't given them to you). I could be going over the top here

Cheers

Cris

Heh.. I don't exactly have first-hand experience with safes, but my understanding is that the only thing that is fireproof/waterproof about them is the safe itself.

I really doubt that a hard drive would survive a house fire when inside one of those.

Cheers

Good question, as you know full well, I'm not a normal sane person. So, no I probably wouldn't expect you to keep those files in perpetuity, unless I'd specifically paid you to do so. Other people may assume otherwise though.

I would've said some clause in your contract that states that files are kept for up to 1 year (maybe), after that there's no guarantee you'll still have them.

The disc you gave me has been copied onto my laptop and ReadyNAS the day I brought it home. I 'think' I know where that disc is now, but I'm not 100% sure.

I'd definitely get a solicitor to look any agreement like that over, how much could it possibly cost you if did lose everything? Would you have to put a financial value on the data?

A hard drive wouldn't survive even 30 minutes in such a safe unfortunately. They're generally rated for paper and more recently for optical storage and NVRAM. The link claims 2 hours for flash storage products. Those temps would likely kill a hard drive though long before though.

I've had am empeg with 100GB and 30GB drives sitting in a safe for two years, which I just pulled out in the past three weeks. It's not doing so great, with a lot of issues reading, it will just stop playing a track at a random point. I haven't opened it up yet nor tried reformatting the disks.

My current media backup strategy is spare 1.5TB drives in their anti-static bags and off site. The data they're backing up sits on a dual-disk redundant 6 disk RAID 6 array. I've only just started this, but my plan is to make backups either monthly or bi-monthly.

Some of the data is replicated again on one or more additional drives that stay here at the house and get accessed once every few months.

While the tapes aren't hugely expensive individually, the price of the drives (~$1K) is prohibitive unless your data is actually worth real money.

That's a good plan, but you're planning on keeping that data on-line with regular backups. This means that you're regularly accessing the media (because it's online or near-line). I do something similar -- the data's on a NAS box (RAID0, so no redundancy), with monthly backup to external HDD. The external drives are rotated; the "father" and "grandfather" sets are in my drawer here at work.

Cris is talking about a different problem -- offline archival where he doesn't keep the original data online. I'd be really nervous about discovering in 5 years time that my data's toast. Regularly reading (and re-writing) it would make me much happier. Of course, if your data's not actually "worth" anything, then it's not such a problem.

Ah yes, it's really archiving, not backup, if there's only going to be a single copy.

Cris' business sounds like it's growing quite well, so perhaps a $1k investment in a tape drive wouldn't be a bad idea. If he can move on the value-added archive up-charges it would make even more sense and could be amortized much more quickly.

Making a second copy to optical disk could provide extra insurance.

Yea, this is something I'd get sorted in my contract and limit my liability to what ever the client has paid me. This is already in my contract with all my clients, so I think a few tweaks and it would cover data storage too.

As for the safe, it's rated to keep the internal temp below a max 177 degrees celsius for 2 hours. Most importantly it keeps sealed for 24 hours, so if the fire brigade filled the house full of water it would still be ok. It doesn't get much better than that without spending a whole heap of money!!!

Maybe I am over thinking this whole thing

Cheers

Cris

Exactly. The JPG images (edited) would be backed up. I'm talking about a greater volume of archive space for the originals.



This is what is causing the problem at the moment. Looking at my jobs coming up over the next few months it's clear that I need much more space than I thought, a conservative estimate puts me at 3Tb of RAW files over the next 6 months. This will shrink of course as final image selection is made and some are deleted but that is way more than I can cope with if I don't archive some of the older stuff.

I have all online backup at the moment with several NAS boxes and a Drobo (as well as JungleDisk) I am wasting a lot of that expensive space on files that I'd like to keep, need quick access to on rare occasions. In my head I think 3 years after an event seems a reasonable amount of time for me to keep the RAW files. So I'm trying to plan for that.

I also need to find an automated backup/archive solution for the Mac. The whole reason why offsite tape storage won't work for me is that I am lazy and it needs to just work on it's own.

Cheers

Cris

Oh, and they all seem to be SCSI. Does anyone still do external SCSI these days? Surely it's all USB or eSATA (or Firewire)... Actually, does eSATA support tape drives?

Like I said, if that data happens to be the all the footage from a prime-time TV show that's still in post, then losing it could cost you 100's of thousands, even more possibly.

As it stands, Cris can probably get away with hard discs, but if he ends up entering into agreements with a bunch of people to store material permanently with a financial penalty for failure, then the cost of an LTO drive, some tapes and a safety deposit box in a vault might start to make sense.

The ones I was looking at were SAS.

My take on a home safe like that is that it primarily prevents theft, anything else should be considered a bonus, not a core feature.

To protect against anything else you probably want to look at using an environment that has been built from the ground up to protect important materials. Like a vault or secure storage facility.

What does the final 'S' stand for? It's still a bit enterprise-y.

I'm not aware of any tape drives with interfaces other than SCSI, SAS, or FC.

That said, SAS controllers aren't terribly expensive. Nearly irrelevant in comparison to a drive.

I thought I read on their website that it was good for 20 minutes in a mere 8" of water? And that's not taking into account the fire burning away the water seals..

I also expect 177C to be higher than the maximum allowable storage temperature for any hard drive, though a quick google could test that theory.

The difference between tapes and disks is interesting: tapes hold less data, and are much slower, but have the huge advantages of (1) no mechanical/electrical parts to die, and (2) they tolerate partial failures.

So I suppose that would make them the first choice for a "file and forget" style of archiving things. Redundant (duplicated) disks are much better when they are accessed often enough to notice failures.

Cheers

You're using a Mac, right? Check out SuperDuper.

I get so used to swapping seamlessly between SAS and SATA drives on the poweredges that it's easy to forget they're actually different systems. It's only when I come to take one the SAS drives and try it in a desktop machine do I remember!

There are a few SAS cards that can be had for less than £100, if you're spending over a grand on a drive and tapes, I'd say you could stretch to a SAS card, hardly a deal breaker.

I think that's just if it hasn't just gone through a fire. When they go through a fire, there's sufficient thermal expansion of the fire safe to seal the box watertight, without water seals. Hence the 24-hours... that's how long it takes, once removed from the fire, for the box to shrink back into a state where it can be opened. At least, that's how our fire safe box is supposed to work.

I've just asked SWM^H^H^HAmy this question, her response was simply that the moment you give over the copies, it stops being your concern. If she wanted things tweaking, she'd have asked at the time. If, 6 months later, she changes mind but your house has burnt down, it's her luck out. Likewise if she'd lost the files it would be her fault. The only time this wouldn't be the case is if she'd paid you money specifically to keep them.

All valid points. But it would give me a lovely warm feeling if you came back to me after a few years and I had all your pictures safe and sound. I think at that point I could charge quite a bit for "recovery" from my "vault" *cough*insurance*cough*payout*cough*

You can tell Amy I have put her pictures on a DVD in a plastic bag in the garden under the Rose bush to see how that works out for data archive

The new server is running a treat, the portable drive is in the safe and shared on the network, all my old 1Tb drives have been put to use and I have a couple of odd SATA drives that will be the first for archive use. I'm getting just over 30MB/s transfer which is much much better than my ReadyNAS Duo.

I was worried about the drive being spun up in the safe the whole time, but tests this afternoon have shown that is successfully spins down after a short period. Bonus! I was worried about the drive heating up in the safe and the hot air not having anywhere to go.

I suppose time will tell if this is a good solution or not.

Cheers

Cris

I'm guessing that is under conditions of a "typical" house fire. What if the fire burns longer than two hours?

In any case, I doubt that a hard drive would survive for any length of time at 177 degrees C (350 degrees F). All the plastic bits would melt, possibly the case would warp, and you are actually perilously close to the melting point of many common solder alloys.

I think the best you could hope for would be that the platters themselves might be undamaged enough that a recovery $peciali$t might be able to save some of the data.

tanstaafl.

I thought that too, but the same would apply for DVDs and USB sticks too. One advantage of living in a newly built house is the general level of fire protection is pretty good. As it's over 3 levels the building regs state firedoors have to be fitted, which should hold back quite a bit of a blaze for a good 30 minutes.

I think this is all so in my mind I can say I took all reasonable actions. It's also nice to meet my insurance requirements for any cash etc... I hold.

Cheers

Cris

I've always found the idea of putting digital media in a fire safe bizarre. It's digital - make two copies, separate them geographically. A fire is far less likely to happen than a drive failure.

My rule of thumb - if it's not spinning in two places, it's as good as gone.

You use the drive whilst its in the safe?

Yea, that is the whole point. Like I said I'm real lazy, the drive int he safe acts as a temporary 2nd backup of the RAWs and is done automatically on import into Lightroom. Once the Wedding is edited the job is archived onto another drive to be kept offline in the safe, freeing up more space for the next job and so on.

The thing I want is to always have at least 2 copies of the RAWs and 3 (one offsite) of the JPGs.

After a couple of years I'd be happy to go down to 1 copy of the RAWs for longer term archive.

Cheers

Cris

So the drive is permanently connected? How do you get the wires in/out of the safe?

Thank you Mark

There is no hole, it's wired in with the door mech. It's the only one I could find that would allow me to do this without spending a fortune.

Cheers

Cris