Mark Lord's recent post about mhddfs has made me think about somthing I've been wanting to do for ages: build an unRaid server. I have absolutely zero experience when it comes to Linux servers (which is why mhddfs is too daunting for me) and this is the reason this solution is so apealing to me.

Now, I'm transforming my old Windows Home Server box to do this. This box contains a Gigabyte GA-G33M-DSR2 motherbord. This board uses the Intel ICH9R as a southbridge.

Since I want to use a couple of WD 3TB drives with this system, I have a question. I know, in Windows, I couldn't use the SATA connectors on the motherboard with 3TB drives since the ICH9 wasn't compatible with disks larger than 2TB. This is also why the early versions of the WD 3TB harddrives came with a small PCI-express x1 card containing two SATA interfaces. Using this card, everything did work as it should.

Now, somehow I've got this idea in my head I won't need to use that card if I build an unRAID system, since this is Linux based, and Linux doesn't case about this ICH9 limitation since it does the disk handling in software. Am I correct here? I cannot find anything online to back up that statement, but IIRC I've read this somewhere once.

Also, anybody got any experience with unRAID? How do you like it?

Thanks!

I briefly looked into this sometime last year and decided against it. The performance was abysmal. Apparently you can use cache drives to speed that up, but I haven't looked into the specifics of that yet.

FreeNAS, something else I looked at, was also slow/poor and not worth the savings on the hardware for me.

Huh?

Never heard of that issue. I did buy a 2.5TB drive which included a (no extra cost) add-on SATA controller, but I assumed that was simply so that it (1) would support the full 6gbit/sec speed of the drive, and (2) had an onboard BIOS that could boot from a drive larger than the "2TB boot limit".

My MythTV box with the 3TB drives is based on ICH8.

Cheers

unRAID sounds exactly like what most of us probably would want to use in a home RAID setup. Much, much better than legacy RAID configurations for home use with single-drive redundancy.

Still requires a full separate back-up, though.

MMmm... perusing the documentation, I'd have to say that managing space on unRAID sounds overly complex.

It's doing pretty much the same thing as mhddfs does, plus a parity drive, but it seems to rely more upon the user to decide which drives will hold which files/folders. (?)

IIRC, Western Digital supplied that add-on card with the first version of their drive, which was 3gbit/s. The 6gbit/s version which was released a couple of months later didn't have that card shipped with it anymore.
The issue was that Windows or Intel didn't release drivers for their southbridge chipset to be able to support drives larger than 2.2 TB. Article with info on this issue here.


I guess this answers my question that Linux has no problem with this! Great! (with Windows, it didn't even work with an ICH10)

Agreed on both counts. It seems pretty ideal to me. It's solid because it's based on Linux, all data is shown in one big storage pool which is expandable without much effort, disks of different sizes can be used and it's redundant if one disk drops out. All the benefits of RAID5 (well, not the extra speed, but so be it), but none of the downsides.

Primary downside of unRAID versus RAID or versus mhddfs: very slow write performance. Probably not suitable as a backup destination for multiple computers.

This can be covered with a cache drive, which is used to hold all the data as a buffer, and then, at a predetermined moment in time (I believe at 3:30 am), unRAID starts copying everything to the array.

I still have an 80GB Intel SSD laying around here which seems perfect for this job (providing I don't write bigger files than 80 GB to the array in one go, which should never happen after the initial backup has been made - incremental backups are not a problem). This means the writing speeds to the server would be a lot faster even than the read speeds (since I would be writing to an SSD )

Yeah, a suitably large SSD mitigates the write times well enough. No redundancy there until after it completes the copy-back to mechanical media.

I wonder when they'll update their software to use the built-in caching that newer Linux kernels support, instead of that copy-back scheme..

What they are doing for redundancy sounds simple enough to reconstruct for use with any Linux system -- gotta write an MD driver to do it. But it sounds like they also have a nice web GUI and stuff to make it all much friendlier.

I may try out their freebie version (supports 2data + parity, no cache) later.. I've got a friend in mind who needs something better than the dinky little NAS box he's been struggling with.

Cheers

That "feature" suggests that unRAID is totally unsuitable as a Linux back-up destination. Oh well.

(mhddfs has the same limitation)

I wonder if perhaps they are actually using mhddfs as part of their implementation -- or a modified copy.

I'm sure if anybody could find out, it's you!

I have an unRAID system with about 6TB of space (1 x 2TB + 3 x 1.5TB + 1 x 2TB parity). I run the 5 disks and haven't bothered with a cache drive. I get about 15-20MByte/sec write speed and read speeds peaking around 35-40MByte/sec

There is some user level security - you can create user shares rather than just one access for all

The file system that does the merge I don't think is using mhddfs. The mount shows as just


I use it as a Crashplan destination to backup machines around the house and also backup its major stores to the Crashplan online backup system.

I run it in a HP N36L NAS enclosure I got fairly cheap. It's low power but not super low power as the drives don't get to spin down much (but unRAID does support that). I get about 75W normal load. I also run two VMs. One is a torrent downloader and the other is my firewall running ClearOS. I have VirtualBox providing the virtualisation.

It does have a level of control of which files go where. You can assign shares to particular drives or exclude particular drives. You can also set it up to not split across drives below a certain depth.

Any questions - ask away.

I must say the speeds dissapoint me somewhat, especially the read speeds. Normal HD's can easily reach 90 MB/S nowadays, so I have no idea why unRAID isn't capable of providing at least 90% of that speed. Since you're using pretty large drives, I don't suppose your drives are particularly old/slow as well. Are you using a low power (but also low performance) CPU like an Intel Atom or something?

Maybe FreeNAS with its ZFS file system is also a good option. I'll have to look into that.

Those speeds are roughly what I'd expect out of a NAS box, if it's mounted via CIFS.

Agreed, but most NAS boxes use low power/low performance CPU's like the Atom or some ARM CPU. I expect better performance than that when using a desktop CPU.

Most commercial consumer NAS devices don't get anywhere close to 90MB/s.

Sure they do. But then you're in the either in the 2-bay or $500+ range. Check out the NAS charts at Smallnetbuilder.com.

No, they really don't. I'm not talking about "some" but "most." You can go take a look at real-world performance numbers and for a good NAS, you're looking at over $500 just for the enclosure without any drives at all. And you need at least 4 drives or you won't have any redundancy. And without redundancy, you shouldn't be using such a product unless it's strictly for a scratch disk. At this level you may have something near 90MB/s, but even then it's not guaranteed.

Remember that there are a ton of NAS products out there. The best performance in consumer devices is typically from the bigger players like ReadyNAS, Synology and QNAP. Not WD, Buffalo, Dlink, etc.

And even these guys have plenty of products in their lower price points that don't get anywhere close to 90MB/s real-world read or write.

It's one of the reasons why my ReadyNAS Pro used to cost over $1000 just for the enclosure.

Of course I was talking about the cost for just the enclosure, without the disks. And about Readynas, Synology, Qnap and Thecus. All the others just don't count. All forementioned brands offer enclosures that do 90 MB/S in price ranges of +/- $500.

If you consider diskless $500 NAS'es no longer 'consumer range', that's another discussion.

Yes it's an AMD Athlon(tm) II Neo N36L Dual-Core Processor running at 1.3GHz. However it should be capable of maxing out a gigabit connection. The slower 1.5TB drives report 90MB/sec and the faster 2TB ones report about 130MB/sec raw read speed with hdparm.

The system cost me $170 plus $50 for some extra RAM (unRAID doesn't use swap and the unit only came with 1GB which wasn't enough once I started running Crashplan let alone the VMs) and something like that again for the unRAID license.

Almost all access is via samba shares to Windows 7 or XP machines.

So I don't know exactly why the slowish speeds but a local same disk to same disk copy is only of the order of 10MB/s. I expect this is coming from the fuse.shfs i.e. it's running in user space based on Mark's comments in the other thread. That local copy runs the Atom CPU pretty much to the maximum available so that's probably my limiting factor. Edit: There is probably some significant overhead in the RAID calculations on a write too. It shouldn't affect read though.

A "dd if=<large file> of=/dev/null" runs about 60MByte/sec though (made sure it wasn't cached).

All that said almost all my access is over a 802.11n x2 connection so the fastest I ever see is about 15Mbyte/sec throughput anyway. So it hasn't really bothered me all that much.

Wikipedia has some information on UnRAID. It sounds like it creates a file-level parity against a system somewhat like mhddfs.

Well mhddfs has no parity. It's just combining multiple file systems to look like one. UnRAID does that as well as the parity.

Parity is calculated at the disk/block level, not at a file level as far as I can tell.

I meant "creates parity against (a filesystem like mhddfs)", not "creates parity against a filesystem (like mhddfs)".

Well, with ReadyNAS, up until somewhat recently, you had to spend well over $1500 to get over 90MB/s including drives. Over $800 just for enclosure more recently.

Their Duo and NV products couldn't achieve those real-world rates, not even close.

The "Ultra" series in at least a 4 drive enclosure may be what can get those marks at the lowest end, but I believe only in some read circumstances, not write.

So I would say you have to spend at least $500 on an enclosure for top speeds.

Well, it's not really shfs, though kind of odd they named it that. The mount flags don't really give any clues, but it is a FUSE filesystem, and what it does is extremely similar to what mhddfs does.

What they seem to do underneath, is simply a drive/sector level parity sum across all data drives. It's not a "file level" parity, but a sector level parity.

That latter part is very handy -- RAID layers do that too, but then they also force striping upon us. This unRAID just does the parity, without the striping. End of Story.

The reason for the sucky perfomance is partly because they're using FUSE rather than an in-kernel solution, so it's going to be CPU heavy as well as incur lots of read latency, just like mhddfs and for the same reasons. Writes will be much slower, because of the read/modify/write required for parity updates.

But some of the performance loss you see could simply be due to not having correctly tuned Samba for MS-Win accesses. Didn't we have a thread on that here recently?

Cheers!

Thanks Shonky! That's good info and worth looking into. I'll have a look if I can find more real-life speed numbers on the net, preferably on systems with a regular desktop CPU. (which is what I'll be using since I have one laying around here anyway)


Why would they choose this type of design if using an in-kernel solution would be much easier on the resources and thus faster? Any specific reason? The same goes for the Linux build-in cache solution you mentioned earlier. Why would they not use this? Because the option wasn't there in the beginning and it's too much hassle now to implement it so they just keep on supporting their own solution to the problem?


No idea, I must have missed that. Could somebody point me to it please? Thanks!

So you guys inspired me to do a bit of testing. It seems Samba at least is not at fault:



Interestingly during the small blocksize test dd used about 25% CPU and shfs about 70%. During the 1M blocksize test, shfs sat around 10-12% and dd around 3 or 4%. So

Because it is very difficult to convince certain key kernel developers to include any kind of "union filesystem" (or look-alike, such as mhddfs) into the kernel proper.

They could still do it in-kernel (outside of Linus's tree), but that would require more maintenance than doing it externally with FUSE, and getting the code "right" is a heck of a lot trickier (and more dangerous) in-kernel than outside with FUSE.

So they've taken the sensible and easier route with it, and your data is safer as a result. But performance will be more reliant on CPU horsepower using FUSE.

The in-kernel generic filesystem caching thing is fairly new (a year ago?), and has it's own quirks. But mainly I suspect they simply already had their own solution, and at this point they're just sticking with it for backward compatibility. Besides, what they do does work well enough for non-critical use.


I forget where I read up on it, but I think it boils down to adding this line to the [global] section in /etc/samba/smb.conf:

socket options = TCP_NODELAY SO_RCVBUF=262140 SO_SNDBUF=262140 SO_KEEPALIVE

MythTV Users list it seems you might have seen it:
http://www.gossamer-threads.com/lists/mythtv/users/501377?do=post_view_flat#501377

Trying it now.