> Which brings another subject: usually hw RAID host adapter have
> cache, and have firmware that cleverly rearranges writes.
> Looking at the specs of the P400:
> it seems to me that it has standard 256MB of cache, and only
> supports RAID6 with a battery backed write cache (wise!).
> Which means that your Linux-level seek graphs may be not so
> useful, because the host adapter may be drastically rearranging
> the seek patterns, and you may need to tweak the P400 elevator,
> rather than or in addition to the Linux elevator.
> Unless possibly barriers are enabled, and even with a BBWC the
> P400 writes through on receiving a barrier request. IIRC XFS is
> rather stricter in issuing barrier requests than 'ext4', and you
> may be seeing more the effect of that than the effect of aiming
> to splitting the access patterns between 4 AGs to improve the
> potential for multithreading (which you deny because you are
> using what is most likely a large RAID6 stripe size with a small
> IO intensive write workload, as previously noted).
Yes, it does have 256 MB BBWC, and it is enabled. When I disabled it,
the time needed would rise from 120 sec in the BBWC case to a whopping
IIRC, I did the benchmark with barrier=0, but changing this did not
make a big difference. Nothing did; that’s what frustrated me a bit
;). I also tried different Linux IO elevators, as you suggested in
your other response, without any measurable effect.
The stripe size is this, btw.: su=16k,sw=4