Some RAID questions...

Guys,

I've been thinking of using a hardware RAID ward for a pure storage setup. The idea is I would use a small harddisk (or maybe even boot from CD-rom/USB stick, not sure about that yet) in the pc/server for the OS, and then use a RAID 5 array for storage.

I'm now looking for a good RAID card. The reason I want to use a hardware card instead of software is two-fold: for one, I know RAID5 is a CPU intensive array setup. I want to minimize the speed drop and secondly, the idea is that, if I want to upgrade the server pc or maybe even migrate the storage RAID to a different machine, it would be as easy as physically transferring the drives, installing the card and its driver and voila, the data is back but on another pc/server.

I'm asking for your help because it's been years since I've focussed on hardware RAID cards. Last time I looked, SCSI-2 was actually still popular, go figure. I have no idea which cards are still considered to be good cards. I'm sure Adaptec still makes good cards, but there might also be good alternatives I know nothing about. That's why I'm asking for your help.

Basically I want a good, speedy hardware RAID card for SATA disks. I guess I will need about 8 connectors on it. External SATA port may also be handy for connecting this type of device in case I still need more storage.

Good driver compatibility for the card is obviously a must, not only for windows, but also for Linux, since I may try and set up an OpenFiler server and try its iSCSI feature.

A last thing I want to ask is: Are hardware RAID controllers still as picky about the drives used as they were 'in the old days'? With this I mean: I've read about dedicated NAS devices like the Synology CS-407 with which is possible to 'upgrade' a RAID5 array by replacing the disks one by one and having the array rebuilt after each disk replacement. (eg. You've got 4x 500 GB, and after you replace them with 1TB disks one by one, you'll end up with a 4 TB RAID 5 array, well 3 TB of usable space). Is this also possible with nowadays RAID hardware RAID controllers? Or even better maybe, has it become possible maybe to add an extra disk to the array when you're running out of storage space maybe? (that would be cool)

Soooo... which controller(s) would you guys suggest?

Thanks!

How much space are you looking at using?
I would look at Adaptec and 3ware raid cards. I haven't tried it but my understanding is you can use SATA drives on a SAS card. If this is true I would really think about using an Adaptec SAS card. This way you can get a card with a 512 MB of ram used as cache and a battery backup. This way you can enable write-cashing on the controller. At work we are using the Adaptec 4805SAS Contoller with 6 146GB Seagate Cheetah's in raid 10. This is crazy fast but low on storage space.

At home I'm using linux software raid5 with 5 320GB Seagate drives on an Athlon 64 X2 4000+ 2.1GHz 65W cpu. I even have it clocked down to 1 GHz using cpufreq and I still get 50+ MB/s when copying data to/from it over the Gigabit network.

You really do not want hardware RAID. There are several prior threads on this very BBS that go into details.

There's no real performance loss in software RAID on Linux -- usually software RAID gives *better* performance, in addition to keeping your data safer.

Cheers

Every Smart Array controller in the Proliant server line has always supported both of these features, going back to the first card in 1995. So it's not new to RAID controllers, but it wasn't common on the workstation class RAID solutions until later. My ReadyNAS uses a hardware RAID controller, and also offers both of these features when using what they call X-RAID. Theirs is built around always allowing scalability, and picking the right redundancy at the right time. If you start out with one disk and enable X-RAID, it just shares it out normally. Add a second disk, and you now have RAID-1. Add a third, and the array is migrated to RAID-5. And now it looks like the X-RAID2 in the ReadyNAS Pro will go up to RAID-6 if you want it to.

Hardware vs software, I'll let the other people here debate that one. Going either way has risks and benefits unique to each, and generally either one can do well for performance and reliability.

What are you intending to use it for?

I've had a 3ware RAID card for a couple of years now, with 4x500Gb SATA disks attached. Driver support under both Linux and Windows (2003 x64) is excellent.

These days, I'd probably opt for software RAID, whether Windows or Linux, mainly because if the card fails, you've got to find a compatible one to access your data, whereas with software, you just boot up another PC with the same OS (I do back up my data to an external disk every month or so, however). Software vs. hardware performance is irrelevant these days for most purposes.

Not sure yet. This is all connected to the 'DVD jukebox' server project I'm doing at home. Basically I want to rip my 700+ DVD's and store them on a server, to have them at my fingertips (on any TV in the house) when I want them. I also rip them as ISO's, and don't compress them any further to DivX or Xvid or so. This means a normal movie usually takes 5 to 8 GB of space. So I will probably start with 4 or 5 TB of diskspace, but it would be sweet if there were an easy way to expand the RAID volume in case I should need extra disk space (which I no doubt will at a certain point in time) and want to add another disk to the array.

Another option is that I would upgrade my 1 TB disks when the 1.5 TB's become available. (already announced by Hitachi) With the Blu Ray movies becoming increasingly popular, I can imagine this need for disk space will only increase in the future...

I've have the luck that I've got reasonable easy access to new hardware, so the chance of me upgrading all my disks at once to reach a higher storage level is not imaginary.

I have no experience whatsoever with SAS. Doesn't that require a special type or (more expensive) drives?

Adaptec and 3ware are indeed brands I've read a lot about. Is it still true that Adaptec is still de facto the standard?


I know, I remember those discussions. But going the software RAID route also have a few limitations. For one, I'm limited by the number of sata ports on the motherboard, which is usually six now (sometimes eight, but that's always with an added chip). That's not the main reason I would choose not to use it though. The main reason is as I said that I've got relatively easy access to new hardware. It's not imaginary at all that I would swap motherboards for this server once a year or so. Can I then take this Linux software RAID array with me to the new motherboard?


I know. I've probably picked up on the expandable RAID array thing from exactly those ReadyNAS boxes, which made me think there were probably computer hardware RAID cards out there by now which could do the same thing. I just don't know which ones, or how that feature is called on a spec sheet.

Oh? Does this mean the hardware in the PC is irrelevant then? If so, this does make my second point of not using software Linux moot. Maybe software RAID is the way to go then.

The only thing that scares me is that I have almost no experience with Linux up to now, and if something goes wrong, I'd have absolutely no idea how to fix it...

I've been thinking about doing the same thing with my DVD's. I was looking at getting something like one of the items below.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816701011
http://www.newegg.com/Product/Product.aspx?Item=N82E16816133001
http://www.newegg.com/Product/Product.aspx?Item=N82E16816702010

Reading the Serial Attached SCSI (SAS) wiki. You can connect SATA devices to the SAS controllers.

I found many sites detailing how to expand a raid 5 array in liunx. Here is just one example: http://scotgate.org/?p=107

Absolutely. Just plug them into whatever SATA ports are available, any mix of motherboard and/or multiple add-in cards (any slot type) etc.. Linux will sort them out into the original RAID.

Try *that* with a 3-year old Adaptec PCI RAID card -- on a new motherboard that doesn't have any PCI slots. Or whatever the slots will be 3-5 years from now.

I'm advocating software RAID here, because it can perform as well or better than "hardware" RAID (which is really just software RAID on an embedded CPU, probably running Linux or VxWorks), and mainly because it's the only safe way to ensure continued access to your data after hardware failures/upgrades.

Use Linux, Windows, OS/X, or a *BSD system -- whatever suits you, though only the first two have universal hardware support.

Cheers

Cheers

Archeon - the linux-raid mailing list is very helpful There is also the wiki I set up : http://linux-raid.osdl.org/ which is the semi-official linux-raid FAQ nowadays.

Linux raid5+6 can grow by adding disks or by replacing disks with larger ones.

Adding disks of the same size is trivial, install and 'grow' onto the new disk.

Upgrading all disks is fairly easy - just remove a disk and swap in a larger one, then resync the array. Do this for each disk. Then tell the array to 'grow' into the new space.

Upgrading your mb - not a problem.
Moving from SATA1 to SATA2 - not a problem.

Anything else?

Frankly your biggest requirement seems to be flexibility - Linux SW RAID has that in bucketloads!

(I just installed my 4 new 1Tb disks today; I also have an old 1Tb RAID6, a MythTV on a 1.2Tb RAID5, a 600Gb mirror and a mirrored / somewhere.)

While it might be true for Adapted RAID, it's definitely not the only way in the hardware world. I could take a 1995 era Compaq server with some old SCSI drives in an external enclosure connected to the same 1995 era raid card, and follow these steps to migrate it to my 2008 HP server:

1. Power down the 1995 server
2. Detach SCSI cable
3. Attach SCSI cable to 2008 server with 2008 RAID card
4. Power on 2008 server
5. Press an F key during boot to confirm RAID import.

Hardware raid isn't all bad, just as all software raid isn't all good. It all depends on the vendor, in this case either a particular hardware RAID vendor ensuring they support their products, or an OS like Linux ensuring it has a good software RAID subsystem.

For my personal stuff, I've gone both ways. I've had 3 generations of Linux servers as my storage boxes since 1998, and the second and third both relied on software raid to group several disks together. The 3rd box had disks added as I needed more space, and eventually was retired when the OS drive failed and I decided to replace the entire setup due to age. I then migrated to a ReadyNAS NV+ with hardware RAID, and new SATA drives instead of the old PATA drives of the linux box. The NAS has a 5 year warranty on it, and the company has several past products all using the same family of compatible hardware RAID in them, so I'm comfortable in knowing I'll be able to get at my data with it for the lifetime of the product, just as I was comfortable with my Linux software solution.

In the end, it all comes down to research before you invest in any solution, and then picking one that will meet your needs and that can be fixed if something does go wrong. Drives fail, as do other components. No matter what direction you go, you need as disaster recovery plan. And always keep in mind RAID is not a backup solution. Never trust your actual critical data to only be in one place, be it RAID, or your desktop computer, or even in the same building.

That is very cool and I didn't know such deeds were possible. But I still suggest that there's an element of luck there in that your RAID vendor didn't go out of business in the intervening thirteen years.

Peter

One thing that does annoy me about RAID is the absolute need for matched drives. I really like the idea behind the Drobo, Windows Home Server, ZFS, and other solutions that can guarantee data redundancy on the volume without requiring identical sized drives. Hopefully these implementations will be mature enough by the time I do look to replace my ReadyNAS down the road. Similar concepts were starting to show up in enterprise storage too before I left that job, and it was really nice.

The only thing keeping me from using Windows Home Server was the lack of federated logins (i.e. single-sign-on on all of the home PCs).

So I run Active Directory on Windows 2003 instead.

Very nice! Indeed, those are some sweet devices. Also good to hear that SATA drives can be connected to SAS controllers.

Also, great link about that expandable Linux RAID5 array! Thanks for that!

This is very good to know, and good advice, as always. Thanks Mark!

Thanks for the advice and link LittleBlueThing! I'm reall starting to believe Linux *is* my best bet, and I probably should go with that. Even if it'll probably cause me a few days of frustration because it's all new to me. But hey, it's a challenge!

On that subject does any one know if any of the Linux distributions treat RAID as a first class citizen yet ?

Last time I installed a RAID box 18 months ago none of the distributions made RAID easy during installation. I tried to take a look at the latest Ubuntu installer the other day to see if things had improved, but it kept crashing as it loaded the kernel on the two machines I tried it on.

Do not, and I repeat, do NOT go the Windows Home Server route if you don't want to get very frustrated. Trust me. Here's a detailed of my experience with WHS. Should be enough to throw anybody off.

(this is a cut and paste job of something I've posted somewhere else)

Oh yeah, on another note, I stumbled onto a project called Ubuntu Home Server. But alas, it seems that project has come to an early halt...
--------------
I wanted to setup a 'video jukebox'. Basically ripping all my 700 or so DVD's I have onto harddisk and stream them from a server to a media extender.

I had a choice between Linux, a regular NAS (which also run Linux, but is FAR easier to set up) and Windows. A couple of months ago, WHS was released, and it sounded like the product I had been waiting for all that time. I read a lot of reviews about it, and they all were raving about this '(one of the) best products MS had ever released'. NONE mentioned the big iussue I stumbled upon. An issue which I believe is unpardonable because everybody will bump into it sooner or later.

In my case it was sooner.

I started to save up (this took a couple of months), and I was overjoyed when I finally had all the necessary components to build my own WHS in-house. These were a recent G33 chipset motherboard with an Intel dual core chip (E2200) on it, 2GB RAM (RAM is cheap these days), and 3 harddisks: one Samsung 250 GB drive as the C: and 2 Samsung 750 GB drives as the Storage drives.

I setup the server and everything went smooth. Then I started to copy all my rips (which I had mostly done beforehand and stored on my desktop pc) to the server. I got some serious good speeds (after having installed the most recent drivers of my onboard NIC's - VERY important!), 70+ MB/s. Not bad, certainly better than a regular NAS. I was happy, but because this still was going to take a while, I let this run overnight. I got up in the morning, only to find an error message: disk full, continue, abort?

Disk full?? How could this be? I still has more than 600 GB of free space left on the server according to the WHS console. I clicked continue and the copying process happily continued. Strange. Why did this error message pop up in the first place then? After a quite a few movies were copied, another strange thing happened: my network speed dropped immensely. And I DO mean immensely. Something from 60-70 MB/s to less than 1 MB/s! Just like that! I had no idea what was going on, but then I started to do my homework, and came across some online articles which explained it to me. I was also the victim of Drive Extender, or at least of how the technology works.

During the copy, the 250 GB (OS disk) disk filled up. Once it was almost full, Drive Extender kicked in, effectively almost annihilating my transfer speeds. Needless to say, I was and am not happy about this!

But that's simply how it works: if you want to add a file to your server, you copy it to a share. In the background, this file gets copied to the C: drive (OS drive). After the copy is finished, Drive Extender kicks in and starts moving the file to its place in the storage pool. Once this Drive Extender kicks in, you'll see immense speed drops on the server, so that's not good. Drive Extender also usually waits until the copy is finished until it kicks in, unless the landing zone runs out of space: then it kicks in immediately, and brings the network speed to an absolute crawl with it. This is what I saw was happening to my server...

The best was still to come though. I had copied a lot of files to the \\server\videos share. I also copied a lot of them to the \\server\music share, because I knew there would be duplicates. That way, I intended to remove the duplicates by hand from the music share and copy the rest to the videos share. And that's where's the sh*t really started to hit the fan. Every movie I tried to copy took on average between TEN and FIFTEEN minutes. If you know I have a few hundred movies to copy like that, that is simply not workable. This was only because Drive Extender forced the movie to be copied from the drive pool back to the OS drive, and then back again to where it needed to be. A process, which on a 'regular' server, RAID or no raid, would take mere seconds!

If I had known this beforehand I would have NEVER started with WHS. The nature of what I intend to do with my server dictates that I'll ALWAYS be working with large files, so this will ALWAYS be an issue for me. This is simply not acceptable. I wished I had gone with a regular Windows 2003 setup which at least offer the possibility to use regular RAID arrays, and not this Drive Extender nonsense. Or I could have gone with Ubuntu Linux server combined with Samba, which would have cost me absolutely nothing, with the exception of a bit of time because the learning curve is more steep and I'm no Linux guru.

As said before, everybody will run into this, if not only for the way they market WHS. They say: "Oh, you can use a small older drive for your OS drive, 80 GB is sufficient." WRONG! Since it's the 'landing zone' for any copied file, AND it's used as some sort of 'buffer disk/workspace' for every file in the storage pool, this should be the fastest drive in your system! If anything, pick a WD raptor! But it still would end up being dog slog compared to a regular RAID array when working with large files.

Some have advised to use a drive of at least the same speed and size for the OS drive is being used for the storage drives. In my case this would be a 750 Samsung drive. Only to be used for a 20 GB OS partition and a landing zone for copied files? I'm sorry , but that's asking too much! A 750 GB drive simply as a buffer only to overcome the downsides of Drive Extender? That's ridiculous! If you ask me, this is a serious design flaw!

I wished they had given me the option to setup WHS with a regular RAID array, and leave this drive extender nonsense where it should be: uninstalled. But they didn't. A true shame, because WHS also has it's merits: it's really very easy to use, the backup facility is awesome eg. But it also has a few quirks, Drive Extender being the worst of them, but also eg. the lack of an onboard FTP server. Because of the nature of how WHS's file system works, (storage pool, with no drive letters), this simply does not work with regular FTP programs. Instead of coming up with a fix for that, Microsoft simply decided to give FTP the boot entirely. I mean come on! Which server does not support FTP out of the box? I know I can install one myself, but that's not the point. The point is when I'd do that, it would probably muck up the entire file system in no time, because FTP'ing would jump over the filesystem's head, creating havock.
But still, I believe FTP is such a standard protocol, it's like buying a car without wheels. It simply should be there, no question about it.

So what am I going to do now? Honestly: I don't know. Try 2003 server? Maybe... Try Linux? Tssss.... I can always use FreeNAS, or NASLite... maybe even OpenFiler or Open-E, those are supposed to be pretty good... Or maybe sell the whole server PC and buy a regular NAS instead. Synology has a very nice model coming up... I don't know anymore, I'm truly in limbo here.

But most of all I feel cheated. Cheated by Microsoft, but most of all but all those 'reviewers' who never discovered this issue, or,if they had, decided to leave it out of their review. From all I read, and I read dozens of reviews, WHS was the best product since sliced bread. Not ONE word of comment (except maybe the review of Tim Higgins from SmallNetBuilder.com, but he didn't like it for other reasons - lack of protocols, no print server, no backup of the server, ...he received a lot of bad comments by all those MS fanboys because of it as well!) about how bad Drive Extender can be...

In short: WHS is a good OS... as long as you only install ONE drive in your system. Once you install a second drive and Drive Extender kicks in and takes over, you're toast.

You mean 'hardware raid'

On linux sw raid you can mix *any* block devices:
* PATA
* SATA (1+2)
* USB
* Firewire
* MFM (!) (I powered up a 1988 Amstrad 2086 MFM drive and read it just before retiring my last ISA-slotted machine last year - I should have RAIDed it for the hell of it )
* RAM disks
* Network block devices

Not exactly 'matched'
(although I am dubious about including external drive(s) in a RAID.)


I'm pretty sure it's first-class on Debian and most others for 'ancillary' filesystems (ie not / & friends). I am surprised at your experience.

I thought I got an option to have a mirrored / last time I did a Debian install - and ISTR SUSE just made it first class as /.
The issues are primarily around the bootloaders which need to grab the kernel and initram from a BIOS accessible drive... not that easy from a RAID0,5,6,10 etc. Not too bad from a mirror.
Roll on the days when PCs arrive with a boot CF drive...

Archeon - on another note - are you ever going to be affected byt the MS 'copyright check' type stuff? ISTR they try to identify copyright material in the background - wasn't that an issue with the VISTA 'never-ending delete' problem? I know very little about the Windows side of the world (long may that continue!)
Anyhow - feel free to yell if you need help on getting it going...

I setup my Linux RAID 5 array with ArchLinux about 15 - 16 months ago. I tried Debian, RedHat, and Ubuntu and they all just had different issues getting them installed or getting Samba working.

Just last month I installed ArchLinux at work. I used this wiki to setup a software raid mirror on the machine. When I setup the machine last month the installer didn't have options for setting up the raid volumes. Following the wiki I was able to get the machine setup in about two hours while stopping to talk to clients on the phone.

No, I should clarify. I mean I dislike the requirement that the size of space used on each disk for the RAID has to be identical. IE, toss in 2 initial 160GB drives in a RAID 1 (Linux software raid, or hardware raid), then toss in a newer 200GB drive, and only 160 of it can be added to the existing RAID. Sure, that leftover 40gb of space can normally be used for something else (yes, hardware and software RAID here too), but since the first two drives into the RAID were 160, that is now the set size.

My 3rd Linux server ended up starting with 3 160gb drives in RAID 5 and an 80gb drive to boot off of. One failed, was replaced by a 200, and so I used the extra 40gb space to backup some files off the boot hard drive. Later I added a 500gb, used 160 of it to expand the RAID, and left the rest of the space as an "UnRAID" share that was used only for data I didn't care if the server lost.

With a Drobo, or ZFS, or something that doesn't use RAID, I could have just kept tossing drives in of various size, and still had data redundancy. Odds are if Linux had stable support for this a few years back, I would have gone that route. Problem is, once I made the decision to go RAID with 160gb drives, I was locked into that until a lengthy migration to completely new and independent disks.

And to also clarify, hardware raid can mix devices just fine too. Worked with several raid controllers that didn't care if the devices attached were SATA (1.5 or 3.0 gb, I or II is a bit of a misnomer), or SAS, FATA or FC, SCSI or Tape. The term RAID also didn't mean Redundant Array of Independent Disks, and instead was changed to Redundant Array of Independent Devices when the servers were shipping with RAID memory configurations. Wheres the software RAM RAID drivers in Linux? :-)

Thanks for the offer, I'll probably take you up on that.

I don't think I'll be a victim of the copyright check as you call it, because all my MS software is genuine. I also don't believe I'll ever use Vista, I'm sticking with XP. (for as long as I possibly can, and so far Vista has given me no reason whatsoever to upgrade. I mean: slower computer, with little or no added functionality? Why would I want that?)

Isn't this exactly what unRAID does? Or any other RAID3 capable OS?

Late opinions of questionable merit:

1) In 2008, unless you *must* do it to provide a transparent disk volume over 1TB to lots of people, I think any sort of parity-based RAID like RAID-5 is a bad idea. It only increases risk of heartbreak (unless of course you have nightly rsync and you really are not concerned about loss of your RAID5 disk system.)

2) With affordable single disks now going to 1TB and beyond, I would say RAID-1 -- simple mirroring -- is the way to go if you can...

3) Some mobos come with up to 6 SATA ports. They could support up to 3 disk mirrors in software RAID,

4) With simple RAID1 on a multiport mobo, you could "leapfrog" -- set up system on ports 0-1 mirror and data/var on ports 2-3 mirror, then copy /var to new mirror on ports 4-5 when new 2TB disks drop below $300 CAD each. All in software RAID off of a cheap commodity mobo without any driver considerations for the various low-budget RAID chipsets on different mobos.

5) We use OpenFiler at work for a backup rsync server. It was a cinch to set up -- took like 30 minutes. Yay!

6) While hardware RAID benefit may be debatable, a 3Ware card let us have 16 SATA ports in a single box (so we can "leapfrog" quite easily). Hard to find a cheap mobo with 12-16 SATA ports.

7) Some of the cheaper hardware RAID cards like the 3Ware 8006-2LP mark the disks such that the disks are completely unusable unless they are attached to that specific controller.

8) When I had some problems with a workstation that had a RAID-1 on a 3Ware 9550SX-type controller, I was pleased to discover that 1 of the Mirrored disks could boot just fine when attached to a simple SATA port on the MOBO.

So, hard to tell what might happen with some of these controllers without testing aforehand, but testing worthwhile.

I will only do parity RAID now if it is in a high-end system/NAS that gets nightly backup.

Archeon- I didn't mean the windows license. ISTR a suggestion that MS was trying to analyse your media files in order to determine if (in MS' opinion) they should be subject to DRM. As a result file copy operations and network transfers were being held up because windows was trying to analyse the file on the fly.

See here: http://techxpress.wordpress.com/2007/04/29/vista-slow-file-copydeletemove-vista-the-woe-starts-now/

and here: http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.html

(caveat - this may be wild FUD/rumour)

jimhogan :

I see an unexplained opinion that I interpret as "My personal risk analysis together with my wallet says that I can afford to buy more disks and more hardware rather than use a more complex but cost effective solution".

Remind me what the I in RAID stands for?

Other people would rather buy 4 1Tb disks to get 3Tb of storage than 6 1Tb disks - your cost/benefit may differ...

That has never been a requirement for Linux software RAID.

EDIT: Oh, I see that the original claim (above) has now been retracted, and replaced with an I don't understand how RAID works complaint. Fair enough.

To use different size allocations on different stripes of a RAID violates the concept. But you can do it on Linux if you really want to. This requires a translation layer between the RAID and the devices, to shuffle blocks between "devices" so that they all end up with similar stripe allocations.

That software layer is called "LVM" (or LVM2) on Linux.

Cheers

Eh? RAID5 only causes heartbreak if two or more drives fail in quick succession. This is a slightly increased risk compared to simple mirroring -- where your heart only gets broken if the two failing drives are the two halves of the same mirror set -- but still a vastly decreased risk compared to JBOD operation. IMO the risk is close enough to zero that you'd have to search really hard to find someone whose RAID5 had failed in a situation where RAID6 or RAID1 would have saved the day.

Peter

You would be backing up anyway if the data is that important. RAID != backup.

I don't see any problem with parity based RAID systems. Okay, if any two drives fail together in a single parity RAID then you're screwed. If you have to guard against 2 drive failure then use double parity. If that still isn't enough then restore from your backup. If your drives are failing quickly enough that you don't have any time to get your hotspare going then you've got bigger issues.

At work we've got several NetApp filers running RAID DP which are replicated to our 2nd site. Even so, we still snapshot and backup regularly.

In both instances I've had where the RAID was corrupted, it was the controller that went fritzy. Never the drives.

Yep, worked with LVM on Linux, along with similar concepts in the hardware RAID arena as well. And yes, I do understand that RAID is a block level operation, and thats where my issue with it comes it. It can be great for many applications. But I think for home use, something that works at a file level can be much more beneficial to use. Going to a file level redundancy easily avoids any mess with LVM or any other methods for avoiding the complicated setups to workaround the base concepts of RAID.

So far, my best impressions of storage products (be it Linux RAID, or hardware devices) has always been better the less I must know about the inner workings of the solution to set it up and operate it on a daily basis. (Probably a loaded statement there, since my last one gave the impression I don't know about RAID ) The EVA storage line was really fun to work with at my previous job. It was this big rackmount system, capable of going up to 252 disks. As for setup, there were definitely optimal ways of doing things, but the hardware took care of it. To create a virtual disk to share out over the SAN to servers, you simply told the controller that you wanted an X gb sized vdisk, with redundancy like RAID 0, RAID 1, or RAID 5. You didn't tell it what physical disks to store the data on, instead the controller would decide where the most optimal place was, in terms of both performance and redundancy. With 18 different disk shelves it would usually favor spreading the data across as many different ones as possible This was done to ensure that if an entire shelf went down, data was still there and accessible. It also would migrate data around as the configuration changes. So if you started out with only 6 shelves, then added 2 more later with disks in them, the controller just took care of migrating data. And the disks inside never had to be the same size. So it was a great system to allow expandability, either over a period of weeks, or a period of years when drive sizes have increased dramatically.

This is part of it but not all, I don't think. I do like the "less complex" part of anything.


Indigent?


Part of my feeling comes from my sense that while many folks should know that RAID != backup, in practice folks get a little lax or forget that they have a RAID config that could fail. In the past 5 years I have seen 7 to 10 postings to mail lists both inside and outside my employment sphere of cries for help to the effect "My RAID 5 is hosed! Does any body have a XXX controller? Does anybody know how to (fill in the blank)?" I think I remember at least 1 Proliant (no idea of actual controller make) and 1 3Ware in these posts. From what I could tell, most of these ended unhappily. A couple were not backed up.

Anyhow, folks used RAID5 with some combination of disks that would be considered small by todays standards (36GB SCSI drives, say) to get a total volume yield that was not attainable in any other way. 300, 500, 750GB or 1TB drives weren't in the picture when some of these were set up, but some of these folks would be completely set for the next decade with 300GB available disk for a given application and they would not need to go to a parity RAID solution to get that.

I also see pretty regular queries fo folks looking to find spare 9, 18 and 36GB SCSI drives to repair or maintain RAID sets that were created man moons ago and that are locked to a particular drive size.

I have an example of backup server that is a 16-drive-slot chassis, but which presently has 8 slots free. By only adding disk as needed (in pairs) we can wait for prices on 1TB drives to sink or wait for 1.5 or 2TB drives. Not locked into the previously affordable drive size.

Anyhow, my arguments are not what I would call scientific. I have a 3TB RAID5 NAS in my care and that is just fine. I don't need same at home. I can't seem to fill up my 500GB mirrored drive set. Using your 3TB example, though, my solution would be to get 2 1TB drives today and then add 2 2TB drives in 2010

I have seen at least one case where somebody set up a RAID5 where I just couldn't fathom why; it was like somebody did it "because I can" or because it was somehow superior. I guess I am on the other end of that opinion continuum.

In fairness, I think at least a few of the "Help! I'm screwed!" email calls for help I cite in last post were folks who only figured out they had a problem when that 2nd drive failed. But I did see one single-drive failure where restore on to a hot spare just would not complete. Also one case where all drives were ultimately shown to be fine but where controller just lost its freaking mind. With RAID1 you have some hope of just plugging a drive in and seeing a file system.

Well, of course. Those pesky meteorites are RAID-agnostic.

Since they don't offer DP, EMC will tell you that NetApp' RAID-6 DP is a bad thing I've had both EMC and NetApp and will say that, on balance, I'd pick NetApp next time. Part of it is that EMC as a company is just way too big. NetApp more focused, easier to navigate support resources. Same end result though: hard drive shows errors....replacement hard drive automatically shows up as if by magic. I checked last week and uptime on the Filer in my old department is 1035 days. Not bad.

When it is time to buy a new NAS in like 2010, I hope I can figure out how to use OpenFiler (or equivalent FOSS product) and achieve the same magical effect.

Somehow I don't think OpenFiler is ever going to figure out the "shows up as if by magic" parts replacement :p

Well, if I lived in the UK, the folks behind OpenFiler would gladly sell me a NAS:

http://www.xinit.com/main/storagenetworking/NAS/sns316012.html

but I can't tell right off whether it comes with Magic(TM).

We buy rack-mount boxes from a local vendor who we hold in very high regard (and who are very FOSS/Linux friendly). We've talked with them about whether they would ever consider an OpenFiler-type solution as a line of business. But supporting core storage with true Magic takes you into a whole different risk profile, so I doubt they are going to do it.

So maybe we will have to survive on ersatz, emulated Magic -- buy a box from them and then keep our own spares around. With our current OpenFiler box, we get email failure notifications both from the controller (3Ware 3DM2) and from Linux MDADM, so it could be worse, but the fault detection/notification systems in top-tier NAS like NetApp set a high bar.

One reason we are motivated to adopt a FOSS NAS is that the commercials NAS are a black box and discussions with NetApp or EMC always get, well, complicated. "What, you say you are running a *Samba* domain? That's really not supported you know old chap!"