#296657 - 10/04/2007 20:45
Hardware RAID Advice
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
After much messing about with software RAID 5 with FreeNAS I have given up. I'm not finding it's a solution I feel my data is safe with. This could be down to the fact I am still a total Linux/BSD noob, but FreeNAS also seems to be locking up my system quite a bit so I don't think it's all my fault  So I am looking for a hardware solution, I am using IDE drives (3 Seagate 250Gb and 1 x WD 250Gb RE) the rest of the setup is pretty bog standard. I think I may use clarkconnect for the NAS part as I already run it as my firewall/router etc... Any advice? I want it to be cheap as the system is manily made up of parts I already had and the idea of the project is to be low cost but highly stable. I'm not that botherd about speed but so far I have found a Socket 478 P4 2.8Ghz, 1Gb RAM, random MB (think it's an ex-HP MSI OEM from a system that was split at work) and a dodgy D-Link Gigabit NIC. I could splash out on a LSI MegaRAID i4 they seem to be pretty reasonable on eBay. Cheers Cris.
|
|
Top
|
|
|
|
#296658 - 10/04/2007 21:35
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
The 3Ware cards are the most highly recommended for this.
-ml
|
|
Top
|
|
|
|
|
#296659 - 10/04/2007 21:48
Re: Hardware RAID Advice
[Re: Cris]
|
addict
Registered: 01/03/2002
Posts: 599
Loc: Florida
|
I'm currently using Arch Linux and software raid on an AMD X2 with 5 320GB Seagate drives. One of my drives failed Friday, the replacement should be here today. Other than the drive failure I didn't have any problems with it. The next raid device I setup might be another linux box or this.
_________________________
Chad
|
|
Top
|
|
|
|
|
#296660 - 10/04/2007 21:53
Re: Hardware RAID Advice
[Re: mlord]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Hmmmmm why is it that the best is almost always the most expensive.
Even eBay doesn't throw up many bargains there.
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296661 - 10/04/2007 22:07
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
I don't know which specific Linux kernel FreeNAS is using, but it doesn't really matter: they all suck big time in fault-handling. The upcoming 2.6.21 kernel is the first one I'd really trust to do most things correctly, but even with that I have patches (my own) to fix obvious dead-in-the-head design relics in the SCSI layer (which is normally used for SATA drives). So.. it's good, but about to get real for the first time. Hopefully by 2.6.23 or so, it should really kick saa. EDIT: ahh.. FreeNAS is BSD -based?? Well.. that explains things a bit better..  Cheers
Edited by mlord (10/04/2007 22:08)
|
|
Top
|
|
|
|
|
#296663 - 11/04/2007 04:01
Re: Hardware RAID Advice
[Re: Robotic]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Quote:
I'm curious if it's the lockups or your desire for 'safety' that has steered you away from software RAID5.
I have software RAID working on my firewall (also webserver etc...) and before putting anything really important on it I went through a series of tests to ensure that if a drive fails I know what to do and know it works. I did suffer a HD failure on that system and I didn't loose any of my data, although it wasn't as smooth as I would have liked.
So far I haven't been able to do the same with RAID 5 and FreeNAS. In fact at the moment I can't even get the thing to mount! So I think it's a matter of trust. I figure a hardware solution will be more able to recover from a drive failure. So I guess it's data security not the lockups.
Quote:
If it's the lockups, the specs you quote are WAY above what I built for my NAS box, so something's wrong with your software selection or maybe execution.
Yea, the specs are a little high, but it's what I have to work with. I am moving towards a solution that takes the pressure off my main clarkconnect server, as I would like to run that mail server on that soon. I think I have already decided to bin FreeNAS and stick with clarkconnect, I know that works and is stable.
Quote:
If it's 'data safety', I'm wondering what specific points have you found to convince you hardware RAID is the way to go.
I read this article. Saw the interface for the BIOS and thought "That's what I need!". If I can find a cheap enough card I think my data will be safer with a hardware solution. Am I wrong?
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296664 - 11/04/2007 10:24
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
I figure a hardware solution will be more able to recover from a drive failure.
Just remember that "hardware solutions" are really just more software in a different, less-accessible, less-hackable, place. When drives go bad with a "hardware solution", it's more likely to "cope" without intervention, but there's more chance of losing everything too.
I have several *very* high-end "hardware" RAID boxes here, and all of them use Linux software RAID internally.
Cheers
|
|
Top
|
|
|
|
|
#296665 - 11/04/2007 12:57
Re: Hardware RAID Advice
[Re: mlord]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Quote:
but there's more chance of losing everything too.
How come?
I think for me there is no chance of getting data back from a Linux RAID 5 as I just don't know what I am doing. Is there a distribution that would offer me a better interface so I don't need to get too tehnical if the worst happens.
Quote:
I have several *very* high-end "hardware" RAID boxes here, and all of them use Linux software RAID internally.
I am very happy with my clarkconnect software RAID, it worked very well, it's RAID 5 that seems to be the problem, but I really want to give it a go as I want the max space out of the drives I have (read : need space!).
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296666 - 11/04/2007 13:10
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
Quote:
but there's more chance of losing everything too.
How come?
Because many of them use some kind of proprietary headers on the drives, and when the "hardware" part dies, there's no easy way to get software that can understand the quirky layouts.
Cheers
|
|
Top
|
|
|
|
|
#296667 - 11/04/2007 14:06
Re: Hardware RAID Advice
[Re: mlord]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Oh yea, big problem there, esp if I use a 2nd hand eBay card.
Well, now I am totally confused. I don't want to spend much more money on this, can anyone recommend a Linux disro I may get on with ???
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296668 - 11/04/2007 14:31
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
Oh yea, big problem there, esp if I use a 2nd hand eBay card.
Well, now I am totally confused. I don't want to spend much more money on this, can anyone recommend a Linux disro I may get on with ???
Cheers
Cris.
Well, totally sight-unseen, I would recommend Ubuntu Server. The new "Fiesty" is due for final (initial) release later this month, and would probably do an excellent job for you.
(I don't currently have any RAIDs set up around here; the big iron bits I just do manually, because I'm working at a layer or two beneath the RAID s/w. In other words, I'm not much use to you unless you can pop over with your system for an evening. )
|
|
Top
|
|
|
|
|
#296669 - 11/04/2007 14:35
Re: Hardware RAID Advice
[Re: mlord]
|
pooh-bah
Registered: 15/01/2002
Posts: 1866
Loc: Austin
|
Out of curiosity, what would you do if he showed up with his box?
|
|
Top
|
|
|
|
|
#296670 - 11/04/2007 14:38
Re: Hardware RAID Advice
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Mmm.. when using the regular (non "server") Ubuntu, the RAID installers are on the "alternate installation" disc images, rather than the default "live" CD images.
Dunno about the "server" edition, but if you go to download a copy, it sounds like the "alternate install" CD is the one you want for RAID.
-ml
|
|
Top
|
|
|
|
|
#296671 - 11/04/2007 14:42
Re: Hardware RAID Advice
[Re: mlord]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Quote:
I'm not much use to you unless you can pop over with your system for an evening.
Great! I'll be on the 20:00, see you in 12 hours!
Now do you think I can check in a 4U Rackmount case as hand baggage....
On the more serious note, I will give your suggestion a try and see how I get on.
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296672 - 11/04/2007 14:46
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
I will give your suggestion a try and see how I get on.
Ack. I'm pulling down the current daily build of "Fiesty Server" to try on one of the racks here later today or tomorrow. If I remember, I'll report back on how it looks for RAID stuff.
Cheers
|
|
Top
|
|
|
|
|
#296673 - 11/04/2007 16:42
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Personally, I would suggest running Solaris on it. Two of my biggest problems with Linux are that it regularly completely revamps large subsystems and its documentation is for shit. Both of these problems have a huge impact on RAID. While Solaris has overhauled its RAID subsystem from time to time (twice in the last ten years, I think), the user interface has remained largely the same. In addition, its documentation is very complete and, I think, understandable. It may be important to point out that Solaris is free of cost these days, and has been for many years. It is possible that you might have a hardware incompatibility, though. I haven't kept up to date on Solaris x86's hardware compatibility.
Edited by wfaulk (11/04/2007 18:02)
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#296674 - 11/04/2007 16:57
Re: Hardware RAID Advice
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
Two of my biggest problems with (???) are that ...
Missing noun, there.
Quote:
... it regularly completely revamps large subsystems
Ah.. MS-Windows. Okay.
Quote:
.. and its documentation is for shit.
Yup, must be MSWin.
It certainly couldn't be Linux on that first point, at least not for RAID stuff -- the interface has been stable for much of the past decade. The second point could apply to almost anything, though.
Cheers
Edited by mlord (11/04/2007 16:58)
|
|
Top
|
|
|
|
|
#296675 - 11/04/2007 18:07
Re: Hardware RAID Advice
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
I will never say that Windows is better than Linux, but that doesn't mean that Linux doesn't change stuff out with little ramification. They might decide to swap out the RAID system tomorrow with a completely new interface. Just because they haven't yet doesn't mean they won't, and their track record doesn't inspire confidence.
That doesn't mean that any of the things they've swapped out or the things they've replaced them with are bad, just that they constantly change, and if a concern is the ability to reproduce, that might be a significant legitimate concern.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#296676 - 11/04/2007 18:50
Re: Hardware RAID Advice
[Re: Cris]
|
addict
Registered: 23/01/2002
Posts: 506
Loc: The Great Pacific NorthWest
|
Cris;
One of the questions that I don't believe has been asked yet is why? Why do you want to use raid5? I currently have both the Promise SX6000 and a 3ware RAID 5 controller sitting on the bench. I don't think I'll every go that route again. The cost of harddrives has drop so much that I no longer feel the need to use something as complex as raid 5 for a hardware backup solution. 320gb drives warranted for 5 years are available for $80.00 USD. Why not just go RAID1 or use a large usb drive to backup your data?
Through the course of my usage of the above two controllers I had a few problems. The promise card declared non-existent simultaneous multiple hard drive failures. Not a good thing. Deleting and recreating the array without initialization saved the day but the pucker factor was huge. The 3ware would rebuild the raid1 array I had on it from time to time. I had both a raid5 and a raid1 running on this controller.
At any rate in todays world with the low cost of storage is this really the best solution?
|
|
Top
|
|
|
|
|
#296677 - 11/04/2007 18:55
Re: Hardware RAID Advice
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
that doesn't mean that Linux doesn't change stuff out with little ramification.
Huh? Linux is an operating system kernel. It didn't exist in 1991. By 1992 it was a complete OS. By 1999 many of the major subsystems (network, RAID, various filesystems, etc..) had been replaced with better implementations.
For the past decade (give or take a year), *none* of this has had any significant external regressions. Lots of new stuff, for sure, but the old stuff is still there and still works. I still do my company invoices on a WordPerfect binary from the mid-1990s.
We actually have a very firm rule requiring no userspace impact when kernel internals are rearranged.
Quote:
They might decide to swap out the RAID system tomorrow with a completely new interface
Not true. Sure, "they" might *add* another RAID implementation, but the one that's there now will also have to stay put for a very long time -- we require it.
Okay, so now look beyond the kernel itself to the whole system. Redhat is the longest lived example, and they've tried very hard to remain more-or-less "the same" in clients eyes. Which is why I dislike them, of course: their distro is still living in the late 1990s, while newer ones (like Ubuntu) with no legacy-base have been free to build in mid-2000s code.
Cheers
|
|
Top
|
|
|
|
|
#296678 - 11/04/2007 19:12
Re: Hardware RAID Advice
[Re: Neutrino]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Quote:
Why not just go RAID1 or use a large usb drive to backup your data?
All good points! The basic idea was to create something cheap (ie using as many parts as I already have) and something that offers as near to 100% up time of data 24/7 as possible with DIY kit.
I am very very lazy when it comes to backups, I want a NAS to sit on my network to take regular automated backups from my 4 main systems in the house. A USB drive is out of the question. The HD's in this system have been taken from various other ideas for back up involving USB solutions.
As for RAID 5, I guess it looked really good on the box in effect giving me 50% extra space from 4 drives. I thought it would be easy, as I have found other RAID solutions in the past. I am starting to wonder if I should start looking at the other solutions again, but it has to sit on the network.
FreeNAS has been a bit of a disapointment, it looks really good, says it does everything I want it to, but it's not very stable yet. I think I will go with Mark's suggestion as this was also used in an article I linked to further up in the thread. I also may try clarkconnect as I am confident with that.
One thing I really wanted was a system that would boot from a USB stick. FreeNAS does this really well. Power consumption is quite important for me, and the idea of not having HD's spun up until really needed sounds like a good idea to me. Heat is an ongoing issue with my under stairs server room, which seems to be stable at the moment, but I think 5 HD's spun up will tip that balance
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296679 - 11/04/2007 19:21
Re: Hardware RAID Advice
[Re: Cris]
|
carpal tunnel
Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
|
Quote:
FreeNAS has been a bit of a disapointment, it looks really good, says it does everything I want it to, but it's not very stable yet.
There's a couple of people I know that use it without any problems. Could there be a hardware problem? Their hardware is a little more antiquated than yours but other than that it seems pretty solid.
_________________________
Cheers,
Andy M
|
|
Top
|
|
|
|
|
#296680 - 11/04/2007 19:30
Re: Hardware RAID Advice
[Re: andym]
|
pooh-bah
Registered: 06/02/2002
Posts: 1904
Loc: Leeds, UK
|
Quote:
Could there be a hardware problem?
I don't think so.
The lock ups are pretty consistant. For example, once the RAID 5 is complete, it locks when you hit the format button, every time! I have tried different RAM, but when I pulled them out I noticed the originals were Crucial, and that this MD and CPU were my CCTV server for about a year, in this case. Never crashed once, which is pretty amazing for XP I think.
I can bring it over on Sunday if you want to take a look for me
Cheers
Cris.
|
|
Top
|
|
|
|
|
#296681 - 11/04/2007 19:55
Re: Hardware RAID Advice
[Re: mlord]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Well, my usual example is the NAT/FW code. How many times has that been swapped out over the years? Three? Each incompatible with the previous.
How many attempts at a /dev filesystem have there been? How often did that break existing things? Glibc incompatibilites have abounded. Gcc problems. The list goes on and on.
And I intentionally use "they", specifically because it's not a single entity. And that lack of a single entity is the cause of many of these problems. It's not as if it doesn't create some good things, too. There's a lot to be said for a lack of monopoly (though RedHat had something of a de facto monopoly for many years), but it also results in incompatibilities, reduplication of effort, etc.
And, yes, I recognize that "Linux" only really applies to the kernel. You know what I mean; don't be disingenuous. If there was a better term for what I was talking about, I'd have used it.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#296682 - 11/04/2007 20:07
Re: Hardware RAID Advice
[Re: Cris]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Here's my pro Linux software RAID strory:
All my tunes (flac & mp3) are on a RAID-5 array of 6 250 GB drives. When one of the drives died, I replaced it with a 500 GB drive lying around. The broken drive was under warranty, and a replacement drive arrived two weeks later. I forced the 500 GB drive to failed state and inserted the replacement drive, assuming I could just rebuild the array. Then Murphy hit me: a second drive from my array made horrible noise when I applied power. It was dead. Now two drives were faulty and my array would not start, even after switching back to the 500 GB drive. The 500 GB was non-fresh and would not be considered as part of the array. Luckily mdadm had a --force option that would bring back the array to life. All my data was there and intact. I wonder whether any hardware RAID solution would save me in this situation.
Here's my con hardware RAID story:
I once lost a lot of data on an IBM ServeRAID array, where the RAID controller would give an I/O error on a sector of the virtual disk, where every drive that built the array was just fine. Recreating the array solved the condition, but the data was lost.
Pim
|
|
Top
|
|
|
|
|
#296683 - 11/04/2007 20:09
Re: Hardware RAID Advice
[Re: wfaulk]
|
addict
Registered: 14/11/2000
Posts: 474
Loc: The Hague, the Netherlands
|
Quote:
f there was a better term for what I was talking about, I'd have used it.
GNU/Linux ?
Pim
|
|
Top
|
|
|
|
|
#296684 - 11/04/2007 20:23
Re: Hardware RAID Advice
[Re: Cris]
|
addict
Registered: 11/01/2002
Posts: 612
Loc: Reading, UK
|
I set up this page: http://linux-raid.osdl.org/index.php/Main_PageIt was a while ago and I've not finished <blush> I got permission to copy and modify the linux-RAID FAQ and I hang out on the linux-raid list a bit too. I run a 1.2TB RAID5 for my myth box and a 1.2Tb RAID6 for mrs lbt's video editing habit. Our important stuff is striped on raid0 (and then backed up using snapshots). I have replaced 4 hard disks showing SMART failures since xmas and have another 3 to return. Currently my RAID6 is 'degraded' (ie has a dead disk) but since it can survive 2 dead disks it would still run fine if another disk died before tomorrow. Oh, tonight I finally got around to setting up wake-on-lan too. So I guess if you have any issues.... I've got some hands on experience and I'd be happy to help
_________________________
LittleBlueThing
Running twin 30's
|
|
Top
|
|
|
|
|
#296685 - 11/04/2007 20:41
Re: Hardware RAID Advice
[Re: wfaulk]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14492
Loc: Canada
|
Quote:
Well, my usual example is the NAT/FW code. How many times has that been swapped out over the years? Three? Each incompatible with the previous.
And very heavy overlap among them. For several years the kernel could still be used with any of the three methods. Gradually, the first one (ipfwadm) was dropped, then the second (ipchains) got dropped quite recently after years of "depreciation". The current userspace interface, "iptables", has been in place for about 9-10 years now. *That's* stability in this world.
Quote:
How many attempts at a /dev filesystem have there been?
The original "static /dev" from 1991 is still supported and works fine. Along the way, we picked up "devfs" as a clearly marked "EXPERIMENTAL" method, and it was carried for many years. Only a month or two ago did it finally get dropped from the bleeding edge kernels. Over the past two years, various new features in the kernel have enabled proper hotplug support, and a userspace "udev" device manager app has grown up to use those features. But again, the original 1991 method still works just fine.
Quote:
How often did that break existing things?
Zero, or near to it, unless one deliberatly switched to a different method.
Installing an entire new system, eg. Ubuntu, of course will give the new methods as default. As it should.
Quote:
Glibc incompatibilites have abounded.
That's a BIG stretch. More correctly, all modern OSs abandoned the original UNIX executable format "a.out" in the early 1990s, in favour of the newer ELF format. This also meant a new set of ELF libraries, separate from the a.out ones. Both can/do coexist. My current bleeding edge Linux kernel *still* can run both kinds of binaries. Very VERY backward compatible.
About 10 years ago, GNU libc (glibc) had a major revision, again producing a new set of library interfaces that are quite different from before. So again, a new set of libs and a new program loader (ld.so.2) got installed in parallel with the old. My old Wordperfect program still uses the old libs, on the latest Kubuntu systems. Fantastic backward compatibility there.
Quote:
Gcc problems.
That's not Linux. And it's not a problem anyway. Just continue to use whatever old version you like. Until very recently, kernel developers were all still using the ancient gcc-2.95 to build kernels. I think it still works today, though we've moved on.
Quote:
The list goes on and on.
It certainly does! Stellar backward compatibility, no major upheavals in established distros. The hot new ones like (K)Ubuntu of course install only the latest stuff by default, but the older stuff is there too.
One can definitely fault a Linux distro on many points, but not this one.
Been there, done that, still doing it.
-ml
|
|
Top
|
|
|
|
|
#296686 - 12/04/2007 00:31
Re: Hardware RAID Advice
[Re: pim]
|
carpal tunnel
Registered: 08/06/1999
Posts: 7868
|
mdadm --force is a scary thing to use late at night trying to recover the array. For those that have never seen it, --force is buried in the manual under a ton of warnings, then when run, the binary also warns you again in a full screen of text and a second switch you have to use to then really confirm. I had to go through it back when I had one drive fail, and another was on it's way out. It let me get most of the data off the array, with only the files at the bad points being lost. I think it ended up being one CDs worth of MP3s that I had to rerip from that.
In fact, if I remember, thats when I posted here asking about having surface scans done from time to time to catch errors quicker. Double failures are the worst with RAIDs, hardware or software. I got spoiled by the Compaq RAID hardware always checking disks when they could, as it often rooted out failures way before the drive completely pitched over dead.
I've had good experiences with hardware RAID, but the type of solutions I worked with in my past job are likely way outside the price range of home use. One benefit I really remember that usually is only in the realm of software RAID is the portability of the data. A 1995 Smart Array controller could die in some server, and even if a like for like replacement wasn't available, the drives could be plugged into a 2007 Smart array (likely 7th or 8th generation by now) and still be able to read the data. A few newer features like full array expansion and such might not be available, but the data would still be in tact and could be moved off to a newer system. Sadly similar compatibility stories are rare on the hardware side.
|
|
Top
|
|
|
|
|
|
Previous Topic
Index