[ ... chunk sizes and relatively small random IO ... ]
> So the conclusion is: do you actually care about performance
> for this application? If you do, I'd say don't use RAID5.
That's a general argument :-). http://WWW.BAARF.com/
The argument you make about RMW for relatively small random
transactions becomes even more relevant when considering parity
rebuilding in case of a drive failure.
> If you absolutely must use parity RAID then go buy a Netapp
> ($$$) or experiment with btrfs (risky).
The Netapp WAFL or BTRFS don't "solve" the RMW problem, they
just do parity with COW (object based in the case of BTRFS).
The COW does not do in-place RMW, but something that has the
same cost overall (depending on balance of read/writes and duty
cycle and temporal vs spatial locality).
The presence of parity chunks that must be kept in sync with the
other blocks in the same stripe turns the stripe into a a block
"cluster" for write purposes, and that's inescapable.
If *multithreaded* performance were not important there would
instead be a case for RAID2/3 with synchronous disks (to nullify
the disk alignment times), but suitable components are probably
not easy to source.
> The cost of another 10 disks for a RAID10 array is going to be
> small in comparison.
More wise words, but this is a discussion about a choice to use
an 11+1 RAID5, which is something that looks good to "management"
by saving money upfront by delaying trouble (decreasing speed and
higher risk) to later :-).