Peter Grandi schrieb:
> Ralf> Hi, we have a new large raid array, the shelf has 48 disks,
> Ralf> the max. amount of disks in a single raid 5 set is 16.
>
> Too bad about that petty limitation ;-).
Yeah, I prefer 24x RAID 5 without spare. Why waste so much space ;)
After talking to the people that own the data and wanted to use as
much as possible space of the device, we'll start with four 12/11
disk RAID 6 volumes (47 disk + 1 spare). That's ~12% less space than
before with five RAID 5 volumes. I think this is a good compromise
between safety and max. usable disk space.
There's only one point left: will the RAID 6 during a rebuild be able
to deliver 2-3 streams of 17MB/s. Write performance is not the point
then, but the clients will be running simulations for up to 5 days and
need this (more or less) constant data rate. Now that I'm getting
~400 MB/s (which is limited by the FC controller) this should be
possible.
> Ralf> There will be one global spare disk, thus we have two raid 5
> Ralf> with 15 data disks and one with 14 data disk.
>
> Ahhh a positive-thinking, can-do, brave design ;-).
We have a 60 slot tape lib too (well, we'll have next week...I hope).
I know that raid != backup.
> [ ... ]
> Ralf> Each client then needs a data stream of about 17 MB/s
> Ralf> (max. 5 clients are expected to acces the data in parallel).
>
> Do the requirements include as features some (possibly several)
> hours of ''challenging'' read performance if any disk fails or
> total loss of data if another disks fails during that time? ;-)
The data then is still on USB disks and on tape. Maybe I'll pull out
a disk of one of the new RAID 6 volumes and see how much the read
performance drops. At the moment only one test bed is active, thus 17
MB/s would be ok. Later with 5 test beds 5 x 17 MB/s are needed (if
they are online at the same time).
> IIRC Google have reported 5% per year disk failure rates across a
> very wide mostly uncorrelated population, you have 48 disks,
> perhaps 2-3 disks per year will fail. Perhaps more and more often,
> because they will likely be all from the same manufacturer, model,
> batch and spinning in the same environment.
Hey, these are ENTERPRISE disks ;) As far as I know, we couldn't even
use other disks than the ones that the manufacturer provides (modified
firmware?).
> Ralf> [ ... ] I expect the fs, each will have a size of 10-11 TB,
> Ralf> to be filled > 90%. I know this is not ideal, but we need
> Ralf> every GB we can get.
>
> That "every GB we can get" is often the key in ''wide RAID5''
> stories. Cheap as well as fast and safe, you can have it all with
> wide RAID5 setups, so the salesmen would say ;-).
I think we now have found a reasonable solution.
> Ralf> [ ... ] Stripe Size : 960 KB (15 x 64 KM)
> Ralf> [ ... ] Stripe Size : 896 KB (14 x 64 KB)
>
> Pretty long stripes, I wonder what happens when a whole stripe
> cannot be written at once or it can but is not naturally aligned
> ;-).
I'm still confused bye the chunk/stripe and block size values. The
block size of the HW-RAID is fixed to 512 bytes, I think that a bit
small.
Also, I first thought about wasting disk space with larger
chunk/stripe sizes (HW RAID), but as the OS/FS doesn't necessarily
know about the values, it can't be true - unlike the FS block size
which defines the smallest possible file size.
> Ralf> [ ... ] about 150 MB/s in seq. writing
>
> Surprise surprise ;-).
>
> Ralf> (tiobench) and 160 MB/s in seq. reading.
>
> This is sort of low. If there something that RAID5 can do sort of
> OK is reads (if there are no faults). I'd look at the underlying
> storage system and the maximum performance that you can get out of
> a single disk.
/sbin/blockdev --setra 16384 /dev/sdc
was the key to ~400 MB/s read performance.
> I have seen a 45-drive 500GB storage subsystem where each drive
> can deliver at most 7-10MB/s (even if the same disk standalone in
> an ordinary PC can do 60-70MB/s), and the supplier actually claims
> so in their published literature (that RAID product is meant to
> compete *only* with tape backup subsystems). Your later comment
> that "The raid array is connect to the server by fibre channel"
> makes me suspect that it may be the same brand.
>
> Ralf> This is ok,
>
> As the total aggregate requirement is 5x17MB/s this is probably
> the case [as long as there are no drive failures ;-)].
>
> Ralf> but I'm curious what I could get with tuned xfs parameters.
>
> Looking at the archives of this mailing list the topic ''good mkfs
> parameters'' reappears frequently, even if usually for smaller
> arrays, as many have yet to discover the benefits of 15-wide RAID5
> setups ;-). Threads like these may help:
>
> http://OSS.SGI.com/archives/xfs/2007-01/msg00079.html
> http://OSS.SGI.com/archives/xfs/2007-05/msg00051.html
I've seen some of JP's postings before. I couldn't get much more
performace with the sw/su options, I got the best results with the
default values. But I haven't tried external logs yet.
Ralf
|