Which filesystem? | Off Topic | unofficial empeg BBS

You are not logged in. [Log In] empegbbs.com » Forums » General Discussion » Off Topic » Which filesystem?

Topic Options

#275786 - 07/02/2006 23:03 Which filesystem?
genixia Carpal Tunnel Registered: 08/02/2002 Posts: 3411	I'm rebuilding my home server. It will have 2x80GB IDE drives (previously existing RAID1 array) and 4x250GB SATA300 drives set up as RAID5. On top the RAID arrays I am going to run LVM. I like the flexibility that LVM affords, and I really like the snapshot feature which will be useful for server virtualisation. On top of that though... which filesystem? Obviously it has to be journalling. The CPU is going to be busy enough rebuilding arrays anyway. I don't want to wait hours for filesystem checks too! The old ext2 on RAID1 took long enough to fsck a 60GB partition. I hate to think how long a 720GB would take in addition. Ext3 would be the obvious solution. But is it the best? Other options would be Reiser, XFS and JFS. I'm also open to running different filesytems on different logical volumes to match performance to intended use. The large RAID5 array is going to be /home. Aside from typical flotsam and jetsom that ends up in /home, there will be lots of large media files there - mp3s and mpegs from my replaytvs. Everything else will be on the RAID1 array - /, /boot, /usr, /swap, /var, /tmp. _________________________ Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.
Top

#275787 - 07/02/2006 23:15 Re: Which filesystem? [Re: genixia]
drakino carpal tunnel Registered: 08/06/1999 Posts: 7868	I've been a fan of ReiserFS for a while, as it just seems to work so much better then ext2/3 in situations where a lot of files might be in the same directory. For me this comes in handy with maildir mailboxes across the system, but might also be handy on a media system. I also trust the recovery tools to get things back if something goes horribly wrong, having once rebuilt a filesystem after a raid 5 meltdown of 1 failed drive and a second with a few bad areas. Once I forced the raid back online, reiserfs tools quickly got back everything possible.
Top

#275788 - 07/02/2006 23:48 Re: Which filesystem? [Re: drakino]
wfaulk carpal tunnel Registered: 25/12/2000 Posts: 16706 Loc: Raleigh, NC US	On the other hand, I've had some extraordinarily bad experiences with ReiserFS, including with their recovery tools. The lack of FS-specific backup and recovery tools bothers me, too, perhaps needlessly. I have had some good experience with XFS, and the fact that it can be grown easily might be a plus for your SAN-including plans. I've had no problems with ext3, but it's not particularly advanced. I have no real experience with JFS. _________________________ Bitt Faulk
Top

#275789 - 08/02/2006 00:25 Re: Which filesystem? [Re: genixia]
Robotic pooh-bah Registered: 06/04/2005 Posts: 2026 Loc: Seattle transplant	Would this link be helpful? http://en.wikipedia.org/wiki/Comparison_of_file_systems I don't have any specific info to give you and I'm sure that most are better informed than I. Searched a little on Slashdot for 'file system' under the Linux index for you, but didn't find any conclusive articles. I'm sure I've seen posts on slashdot regarding this topic (slashdot being a haven for Linux lovers, afterall) but my search-fu isn't strong tonight. _________________________ 10101311 (20GB- backup empeg) 10101466 (2x60GB, Eutronix/GreenLights Blue) (Stolen!)
Top

#275790 - 08/02/2006 00:49 Re: Which filesystem? [Re: genixia]
mlord carpal tunnel Registered: 29/08/2000 Posts: 14496 Loc: Canada	I use ext3 exclusively here, mostly for the reduced hassle: all distros and boot CDs support it "out of the box". But it sucks for performance compared with XFS / JFS. ReiserFS is actually two different file systems right now: Reiser3 and Reiser4. The latter is too new to be stable, and the former is inferior to XFS/JFS, but existed in Linux before any of its rivals. Cheers
Top

#275791 - 08/02/2006 01:18 Re: Which filesystem? [Re: mlord]
genixia Carpal Tunnel Registered: 08/02/2002 Posts: 3411	Forward-compatibility with distros is always a factor. In the past I've always brought up a new box on new disks, then added in the old disks to migrate data across. With only RAID1 this works well. Obviously this gets harder with RAID5 because you need all the disks in the array (you can't just mount a degraded half like you can with RAID1), and the number of disks is higher. I'm hoping that the recent mdadm tools and LVM will enable me to grow the RAID5 array as necessary in the future. That all being said, I don't see XFS, JFS or Reiser being dropped from mainline distros anytime soon and I'm only going to run mainline distros on a fileserver - I'll leave the esoterics for the desktop. I'm still not sure how much I care about performance versus convenience and reliability. For the RAID5 array, read performance is a bigger issue than write - mostly the data is static. Grabbing shows off of the replaytvs is severely bandwidth limited by the replaytvs themselves. Writing mp3s is going to be CPU limited. And it's not as if this box is going to be serving many people either. The smaller RAID1 system array is going to need better write performance. syslog. A month or two ago I saw an in-depth benchmarking of all of these filesystems, specifically with RAID. I wish I could find it now. _________________________ Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.
Top

#275792 - 08/02/2006 04:21 Re: Which filesystem? [Re: genixia]
tonyc carpal tunnel Registered: 27/06/1999 Posts: 7058 Loc: Pittsburgh, PA	As I type this, I'm going through pretty much the same thing with my main server. Here's how I'm setting mine up: 4 identical physical disks (4 x 320GB SATA) 3 RAID arrays, across all 4 disks: Small RAID1 (200MB) ... logical partition ...... /boot (ext3) Small-ish RAID0 (5GB) ... LVM volume group ...... swap ...... /tmp (reiserfs) Big-ass RAID5 (the rest of the space) ... LVM volume group ...... / (reiserfs) ...... /home (reiserfs) ...... /music (jfs) ...... /photo (jfs) ...... /video (jfs) (the latter 3 possibly on the same logical volume) So, the idea is, simple mirroring for the boot partition, simple striping for the "don't care if they die, but need the speed" partitions, and RAID5 for general use. Within the RAID5, partitions with smaller files will be reiserfs (reiser3), and larger (media) files will go on JFS. My reason for this was based on this benchmark test and one other I've lost the link to, where the conclusions were: reiser for small files / general use, xfs for the absolute fastest write speeds with larger files, and jfs for the fastest read speeds with larger files. The write performance of jfs isn't much lower than xfs, certainly not so low as to trade off read speed for my media. I recognize a couple anecdotal benchmarks don't prove one or the other is better, but it's enough for me to make my decision. Back to the adventure (on install CD 3 of 4 now...) _________________________ - Tony C my empeg stuff
Top

#275793 - 12/02/2006 11:47 Re: Which filesystem? [Re: genixia]
mlord carpal tunnel Registered: 29/08/2000 Posts: 14496 Loc: Canada	Quote: I'm rebuilding my home server. It will have 2x80GB IDE drives (previously existing RAID1 array) and 4x250GB SATA300 drives set up as RAID5. On top the RAID arrays I am going to run LVM. I like the flexibility that LVM affords, and I really like the snapshot feature which will be useful for server virtualisation. On top of that though... which filesystem? I have just received a box here, with 12 300GB SATA drives installed. I expect that I'll be setting up RAID5 over LVM on it within a few weeks, and it'll be interesting to see what kind of performance it is capable of.. Cheers
Top

#275794 - 12/02/2006 17:03

Re: Which filesystem? [Re: mlord]

genixia
Carpal Tunnel

Registered: 08/02/2002
Posts: 3411

Oh, I am so jealous. That should yield about 3.07TB in RAID5!

I'd be tempted to try RAID6 with that many drives though, or at least reserving one as a hot spare. You'd still get about 2.79TB.

I take it that this is work-related and on someone-elses dollar. Can you tell us which sata card you're using?

Here's my bonnie++ test using 10GB logical volumes.
Code:


Version  1.03                 ------Sequential Output------ --Sequential Input- --Random-
                              -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine                  Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
genixia-ext2-4k         2049M 34531  82 74377  16 17483   5 23804  60 57312  10 436.0   1
genixia-ext3-4k         2049M 31148  75 68173  20 21928   6 28116  68 54798  10 364.1   1
genixia-jfs-4k          2049M 33759  82 70475  16 19181   5 23839  49 50013   9 466.9   1
genixia-reiser-4k       2049M 34656  78 67953  23 21457   6 24017  60 52841  11 417.0   0
genixia-xfs-4k          2049M 35217  81 72467  22 19437   6 24457  56 56003  10 431.5   1
genixia-ext3-4k-noatime 2049M 32251  75 60789  18 22492   7 28932  61 56562  11 388.6   0
                           ------Sequential Create------ --------Random Create--------
                           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min               /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
genixia-ext2-4k         16  3815  99 +++++ +++ +++++ +++  3880  96 +++++ +++ 11239  99
genixia-ext3-4k         16  2809  98 +++++ +++ +++++ +++  2799  97 +++++ +++  6702  95
genixia-jfs-4k          16  1163   0 +++++ +++   546   0   511   0 +++++ +++   342   0
genixia-reiser-4k       16 23224  99 +++++ +++ 25527 100 22164 100 +++++ +++ 22071  98
genixia-xfs-4k          16  1353  14 +++++ +++  1381  11  1397  16 +++++ +++   579   6
genixia-ext3-4k-noatime 16  2791  99 +++++ +++ +++++ +++  2816  99 +++++ +++  6145  96

_________________________
Mk2a 60GB Blue. Serial 030102962 sig.mp3: File Format not Valid.

Top

#275795 - 12/02/2006 17:19 Re: Which filesystem? [Re: genixia]
mlord carpal tunnel Registered: 29/08/2000 Posts: 14496 Loc: Canada	Quote: Can you tell us which sata card you're using? Some custom motherboard thing, in a 2U rackmount. But on an other note, I expect to be taking on maintenance of the Linux Marvell SATA driver for a while. Apparently lots of people use Marvell controllers for large RAIDs, and I've heard varying reports about the (un)reliability of the driver (seems to vary by exact chipset version). Cheers
Top

Previous Topic

Index

Next Topic