I've got performance issues on a server I've just set up, it's a Fujitsu TX200 S6 quad core running 4 win servers (SQL server, Terminal server, Exchange and AD) over vmware. It came with a pre-installed LSI MegaRaid card (haven't managed to discern which model) with 512MB RAM, due to HDD availability issues I ended up with 3 x Seagate Barracuda XT 2TB 3.5 SATA Internal Hard Drive (64MB cache) in a raid 5 array.

We're having intermittent performance issues over the servers but according to the VM console RAM & CPU utilisation is fine, the only thing that's looking poor is write latency. Have spoken to Fujitsu who say that we should try enabling write cache on the controller, I've bought the BBU to protect the data and will be installing it on Sunday.

Am I looking in the right place? Does enabling write cache make much difference? Should I have hung on for SAS disks?

In the previous setup the SQL & Exchange machines were on a single core Dell (5 yrs old?) and the AD & terminal server on a dual core Fujitsu TX100 S5 (3 yrs old?). All on much less total RAM, and performance was acceptable, I expected performance to be better.

Any help appreciated.

Write cache on the RAID controller should help quite a bit. SQL and Exchange tend to issue lots of very small write commands. By using write cache on the RAID controller, it can batch together several smaller write commands into a better performing larger write command, along with the RAID 5 checksum.

If you look up pretty much any drive on storagereview.com, they all have pretty low 4K write performance compared to larger 2MB writes. Hard drives of any type suffer much more then SSDs, and SATA drives tend to be at the bottom due to 7,200 RPM still being common. Most SAS drives spin at 10k or 15k, helping to reduce the seek time of the drive, a major contributor for 4K write delays. Smaller frequent writes tend to involve more hard drive seeks compared to larger sustained writes.

Thanks Tom, looks like I'm on the right path then. As I said I'll be installing the card on Sunday, but I can't quite work out whether the array will be rebuilt when I enable the write cache, any ideas?

Mmmm.. interesting. Just how slow is it writing right now? In particular, how slow is a large, (mostly) sequential write?

This info is very relevant to something else I'm working on right now.

Thanks.

I have very little experience with LSI cards, but my past experiences with other controllers has been that changing/enabling caches don't require RAID rebuilds.

What kind of test could I do to determine this? I'll be bringing the server down to install the IBBU in an hour or so

I set up some test jobs, these are the results before cache write enable (over gigabit LAN), the last job (copying from one folder to another on the VM) is just shocking:

Job: Test_Copy0 - My home directory NAS to VM
Run Time: 11/03/2012 11:24:57 to 11/03/2012 11:30:53 (Duration: 5:56)
Status: Completed Successfully
Files Processed: 4,522
Files Copied: 4,519
Folders Processed: 346
Bytes Copied: 1,863,435,115
Bytes per Second: 5,229,785

Job: Test_Copy1 - Zip of my home directory NAS to VM
Run Time: 11/03/2012 11:30:53 to 11/03/2012 11:32:33 (Duration: 1:41)
Status: Completed Successfully
Files Processed: 1
Files Copied: 1
Folders Processed: 1
Bytes Copied: 1,523,321,591
Bytes per Second: 15,145,674

Job: Test_Copy2 - Zip of my home directory VM to VM
Run Time: 11/03/2012 11:32:33 to 11/03/2012 11:38:46 (Duration: 6:12)
Status: Completed Successfully
Files Processed: 1
Files Copied: 1
Folders Processed: 1
Bytes Copied: 1,523,321,591
Bytes per Second: 4,092,717

This is after enabling cache write, run just now with other people in work, yesterday it was just me, test_copy2 has improved hugely:

Job: Test_Copy0
Run Time: 12/03/2012 11:27:23 to 12/03/2012 11:33:32 (Duration: 6:10)
Status: Completed Successfully
Files Processed: 4,517
Files Copied: 4,514
Folders Processed: 344
Bytes Copied: 1,863,431,279
Bytes per Second: 5,039,910

Job: Test_Copy1
Run Time: 12/03/2012 11:33:32 to 12/03/2012 11:35:29 (Duration: 1:57)
Status: Completed Successfully
Files Processed: 1
Files Copied: 1
Folders Processed: 1
Bytes Copied: 1,523,321,591
Bytes per Second: 13,026,746

Job: Test_Copy2
Run Time: 12/03/2012 11:35:29 to 12/03/2012 11:36:43 (Duration: 1:14)
Status: Completed Successfully
Files Processed: 1
Files Copied: 1
Folders Processed: 1
Bytes Copied: 1,523,321,591
Bytes per Second: 20,637,579

Cool. Still very slow by modern standards, but much better than before.

Yes, so is that just down to the 7200 spin speed?

And the not many spindles. Three drives in RAID5 means that every write goes to two of the three, so it's scarcely faster than if they were all sharing just one drive (though of course it's much more resilient).

On Linux it's a big win to make sure that your filesystem blocks are aligned with the blocks of the underlying RAID, but I've no idea whether the same applies to Windows and/or VMWare, or how to frob it if it does.

Peter

So 4 drives would be faster? I didn't realise that.

Yes. I think some of the stuff in Wikipedia about RAID5 performance may be nonsense (the latency part -- RAID5 reads access just one disk, and writes just two, not all of them), but fairly clearly, the more drives, the greater the chance that any given read or write will go to one that isn't already busy.

Peter

No, even with 7200rpm drives (assuming modern ones), the basic RAID5 should be giving 50-70MBytes/sec. The GigE is capable of a raw 100MBytes/sec or so, but some protocols drag that down do around 45-50MBytes/sec.

You're not seeing anything close to those speeds.

The drives are very current 2TB Seagate Barracudas, so is there anything else I can do to improve performance? I guess any other solution is going to involve rebuilding the array but I can do that over a weekend.

Peter - I couldn't see (well not in a way that made sense to me) how an extra drive in the array would improve performance, this would be the cheapest option so I'm happy to do that if it's worthwhile.

Dunno. Perhaps the RAID5 overhead on that operating system is much higher than I'd expected. Dunno.

OK, it's a huge improvement so I'll maybe wait till I can invest in some quicker drives

Now that you mention OS I've just googled for RAID5 performance on VMWare which according to this link is pretty poor:

http://www.superjason.com/archive/2007/10/26/vmware-raid-5-vs-10-and-stripe-size.aspx

I didn't even consider that it might be an issue with VMWare doh!

More drives in a RAID speeds up the read and write performance due to less downtime from seeks. Let me give you an example:

Lets say you have a RAID 0 array with 4 drives, and want to write a 2MB file. For this example, we will say the RAID stripe size is 512KB (this is way above average, but helps for the example. RAID strips are another factor that can impact performance, and should be tuned for the primary use of the RAID).

To write out 2MB, the RAID controller will break up the data into 4 512KB chunks. Chunk A goes to drive 1, B to 2, C to 3 and D to 4. These writes can happen in parallel, so the 2MB operation completes in the same time as one drive writing 512KB in a standalone setup. This write was the absolute best the RAID could do performance wise.

Now, lets change the stripe size to 256KB. That 2MB data now requires 8 chunks (A-H). Drive 1 ends up with chunk A, and E. Drive 2 has B and F. And so on. Now the write takes a little longer, because the RAID controller has to wait for chunk A to be written before it can write out E to drive 1.

Keeping the stripe size at 256KB, and expanding the RAID from 4 to 8 drives would now write out that 2MB file quicker, since there are 8 drives to accept the 8 chunks of data going out.

RAID 5 works similar to RAID 0 in the striping aspect, with the addition of redundancy data that is written in the same manner. In the first example, Drives 1 2 and 3 would write data, while drive 4 would write 256KB of redundant info to allow data reconstruction. Next write around would potentially write data to drive 1, 2 and 4, with the redundant info going to drive 3. RAID 5 is called distributed parity due to that redundant parity info being written to any drive.

This is all just a high level overview showing why more hard drives end up running faster (up to a point). It's a matter of parallelizing the workload, and reducing the main bottleneck which is seek time of each individual drive. Small data read/write operations tend to be measured in nanoseconds. Seek time to get to that data is measured in microseconds. The layers of striping, cache, and reordering drive operations are all different ways to try and minimize how much time a drive spends seeking vs reading/writing.

Isn't the RAID work being done by the controller?



That article, from 2007, refers to VMware server, which could be referring to the hosted product that VMware used to offer, I think it can still be downloaded, but VMware have not made any changes to it in over 2 years. One of the links from the comments does clearly refer to server.
From your first post is sounds like you are using the ESXi, which can also be used for free. However, it does not do any RAID work, it leaves that up to the hardware.

Yes I am using ESXi

Get to know esxtop, it will tell you a lot more than the graphs in the UI
Duncan has a good intro to it: http://www.yellow-bricks.com/esxtop/ including how to export the data.

Thanks

Test_Copy2 was seeing 20MBytes/sec in a within-the-array copy, so that's 40MBytes/sec throughput, and not a fully streaming read/write either as it must flick from the read site to the write site. That's not far off IMO.

On the other hand, RAID5 is always going to suck for workloads with lots of small writes. That's not VMWare's fault, as it's just passing through the writes made by the operating systems inside the VMs, nor the controller's fault, as it's inherently how RAID5 works. If you don't mind buying a fourth disk, but you only need 2N amount of storage, then switching to RAID1+0 will probably be a win for the Exchange/SQL Server workload.

Peter