On 2/3/2012 5:54 AM, Brian Candler wrote:
> and the drives are Hitachi Deskstar 5K3000 HDS5C3030ALA630:
3TB, 32MB cache, 5940 RPM
> The files have an average size of 0.65MB
You stated you're writing 100,000 of these files across 1,000
directories, with Bonnie, then reading them back with dd in your custom
script. You state this is similar to your production workload.
You've hit the peak read rate of these Hitachi drives. As others
pointed out, if you need more read performance than the dozen of these
you plan to RAID stripe, then you'll need to swap them for units with a
or with SSDs, which will yield an order of magnitude increase. Your
stated need is 20M 500-800KB files, or 20GB if my math is correct. Four
of these enterprise Intel SLC SSDs in a layered mdRAID0 over mdRAID1
will give you ~375 file reads/sec at 800KB per file, again if my math is
for $480 USD total investment. You might get by with a mirrored pair
depending on your actual space needs, but performance would be half.
You're probably wondering why I didn't recommend an mdRAID10 instead, or
a 3 SSD RAID5. All of the mdRAID striped RAID codes serialize on a
single master thread, except for RAID0 and the linear concatenation
(--linear). With storage devices capable of 35K IOPS each, that single
thread, even running on a 3+GHz core, has trouble keeping up.
The LSI 9201-16 you have is based on the SAS2116 chip, which isn't fast
enough in RAID10 mode to keep up with the SSDs. In straight HBA mode it