I'm finally getting around to dealing with a fileserver that I bought, and once I started dealing with it, I realized an issue that I hadn't considered, which is how to set up the RAID.

I have a system with eight 1TB drives. I suppose I might be able to hook up a pair of other drives, or even a flash drive of some nature for the OS, but let's assume I don't want to do that.

The only software RAID that Linux can boot is RAID1. (Let me know if that's not correct anymore.) I'd like the OS to be RAID protected, but I don't really want to lose a quarter of my available space for that. I suppose I could carve out 16MB or so from a couple of drives, but then I effectively lose that across all the drives, and I don't really need an 8-way mirror, even assuming Linux has support for that. Ideally, I'd carve 4MB out of each drive, and RAID10 them, but I don't think Linux will boot off of RAID10. Still, losing that 16MB per drive isn't a huge loss, and this is certainly the way I'm leaning right now.

The other obvious solution is to use hardware RAID, but my controller only supports RAID0 and RAID1, and I want to use a parity-based RAID for as much of the disk space as possible.

Am I missing any other options? Is there a boot loader that supports Linux RAID 5, for example?

When I used software RAID, I just did RAID1 over all the drives in the array to make the boot partition. I think that is what you've got to do.

If you go for hardware RAID or have bigger than 2TB drives then you'll come across the other problem I found which is that the old style MBR partition tables can't handle > 2TB.

Actually Bitt, you *do* "need an 8-way mirror", so that each drive is bootable. Unless you can predict exactly which drive(s) will be failing.

So just partition off 50MB from the front of each drive, and RAID1 across those partitions. Then forget about it.

Do whatever RAID you like over the remaining capacities.

Cheers

Also, note that this is really a bootloader/BIOS issue, not a Linux issue. Once the Linux kernel has been loaded by the bootloader/BIOS, it can deal with any kind of RAID you like.

But until *Linux* is loaded, you're likely dealing with simpler programs (bootloader/BIOS), which won't understand most RAID setups.

Cheers

A two-way mirror is enough. If any second drive fails before the first is replaced, the RAID5 is dead too.

Peter

Sure, the first time it happens.
Then, two years later, a different drive fails. Bam.. no more boot.

Well, okay, maybe. But the safest way by far is to just RAID1 /boot across the array.

Cheers

Perhaps I'm misinterpreting what Bitt is trying to do. But if you've got "/boot" RAID1 on /dev/hd[ab]1, and "/" RAID5 on /dev/hd[abcdefgh]2, you are safe against any one drive failure (but no pair of drives). Once you replace the failed drive, you rebuild both RAIDs, make sure hda and hdb are both bootable, and you're back where you started where you can survive any one failure.

If it were my box I'd be tempted to put swap on /dev/hd[cdefgh]1.

Peter

Well, I don't see myself putting anything more than /boot on the small RAID1 set, and I don't see any reason to make that very big at all. I mean, it's pretty much just the kernel, and how often does that get bigger than a few megabytes? As such, swap would be ((a few megabytes)/2*6) worth of space, and that seems kind of irrelevant.

I was strongly considering running without swap anyway. I've got 6GB of RAM in the system.

I just wasn't sure if Linux RAID supported n-way RAID1 where n != 2.

Sure, no problems at all.


Well, it holds the vmlinuz, initrd, System.map, abi (?), and config files.
Plus GRUB and it's various files.

On my notebook here, with four kernels (but only *two* initrd files), that's 33MB.

It's really easy to accumulate kernels through updates and the like -- they generally leave the old ones laying around just in case the newly installed one fails to boot for any (peculiar) reason.

50MB is a tiny, tiny fraction of 1TB (1000000MB). Amazing, that.

Likewise. I really don't see any point to swap space, given a decent amount of RAM. The logic people use for that stuff is all messed up.

Consider: 512MB RAM, so people will say "you need 512MB of swap". Bump that up to 1GB of RAM, and the same folks then up the ante to another 1GB of swap. So just put 2GB of RAM in the box and forget about it!!

The only real requirement for swap, is when using the default in-kernel suspend-to-disk feature, which writes RAM out to swap and then powers down.

But if one is using TuxOnIce instead, then it can use an ordinary file as the target, rather than needing a special swap partition. Or at least that was the case last time I looked.

-ml

Yeah, people (including the idiots over at RedHat) still go by that old saw of "twice your RAM", which stopped making sense when Unix virtual memory implementations moved from swapping to paging. Which is to say, like 25 years ago.

My personal rule of thumb is: "how much memory are you going to use at once? Double it (because you'll underestimate) and make sure your RAM plus swap equals that." (Or, in less frugal times, buy that much RAM.) Regardless, that means the more RAM you have, the less swap you need, and vice versa, which is the exact opposite of that old wives' tale.

The purpose of swap (on a modern system, anyway) is to ensure that if a process does get carried away, it degrades gracefully rather than hitting a brick wall. Sure, if you can prove you'll never need more than X amount of memory -- if you're effectively building a very large "embedded system" -- then you can just buy that much RAM and walk away. But for most boxes -- development machines, for instance -- you can't make such cast-iron guarantees.

Really the amount of swap you need depends not on how much RAM you have, but on how fast your CPU and disk are -- the Mac Pro can get itself half a gig into swap before I notice it's happened. But because it's hard to work that up into a rule of thumb ("five times the result of hdparm -tT plus a byte per four CPU Hz"?), and because CPU speeds, disk speeds, and RAM sizes all proceed vaguely upwards together, the rough estimate of needing an amount of swap 1-2x the RAM remains useful and sensible, even if not for the same reasons it was originally promoted.

Peter

Perhaps. But exactly what are those reasons now ? Your post above didn't really present anything credible.

Even with swap, there's still a brick wall. And systems now have way more RAM in them than the total of RAM+swap would have been just a couple of years ago.

Regardless of swap, Linux can/will discard pages to free up RAM when needed, so a little RAM goes a very long way with 2.4/xx/2.6.xx kernels.

Swap is obsolete for nearly all situations.

Cheers

With swap, it's less like a brick wall and more like one of those emergency sand traps you sometimes see at the end of long downhill roads: going into it slows you gradually to a halt and prevents a hard crash.

The point is, a heavily swapping machine lets you do something about the problem; sometimes, there's nothing you can do that wouldn't have been done for you by the OOM killer if there were no swap -- but at least, with lots of swap around, you get the choice. (Stopping the feed that's overwhelming your database, without having to kill the database server program, that sort of thing.)

Peter

Err, in my own experience, a heavily swapping machine generally always crashes X11 in the end, and is so unresponsive that one cannot get a command in sideways to help it beforehand.

Linux will kill processes to free memory when RAM+swap are full. You'll get to that point more quickly when using RAM+RAM than with the much slower RAM+swap anarchism.

And again, "lot's of swap" has through history usually meant no more than 1GB or so for most systems. And that was before Linux became entirely page based, allowing it to easily discard clean pages to free additional RAM on the fly. So just put 2GB of RAM in there and forget about it.

And embedded systems (eg. empeg) generally don't need or want swap, as it buggers their RT response and swapping to flash is generally a no-no.

Look at how well LiveCDs and the like work, entirely from RAM, with the added burden of having to store any modified files in that same RAM. No problemo usually, even on 512MB machines.

Cheers

Agreed 100%. If the swap wasn't there, the OOM killer would just do whatever it would do when the swap ran out, and it isn't busy thrashing for dozens of minutes up to that point.

That said, sometimes it is unfeasible to have as much RAM as you need, due to cost or architecture, and you have to use swap. But, IMO, it makes more sense to add swap files on the fly than have your system come to a crawl because of a runaway process that you're going to want to kill anyway.

My desktop machine has <512MB.

I do find swap useful particularly when my wife opens a ton of programs, and I need to get back to working on something else that's going to consume lots of RAM (say... photo editing), and I don't know if she's done with them. Minimize them, off they eventually go into swap.

IMHO, that's not a pattern that's going to change, regardless of how much RAM is installed in the system, so I'll probably always keep a swap partition around -- though I'm not going to follow that old Red Hat rule on sizing.

I was thinking specifically of machines utilized as servers, not interactive workstations, but you're right about swap being useful in that case.

On a server machine, though, I don't think that a process that's inactive that's using a lot of memory should be running to begin with. There are obviously exceptions, like databases with spiky usage patterns, for example, but generally speaking....