[ ... ]
> Which means that your Linux-level seek graphs may be not so
> useful, because the host adapter may be drastically rearranging
> the seek patterns, and you may need to tweak the P400 elevator,
> rather than or in addition to the Linux elevator.
> Unless possibly barriers are enabled, and even with a BBWC the
> P400 writes through on receiving a barrier request. IIRC XFS is
> rather stricter in issuing barrier requests than 'ext4', and you
> may be seeing more the effect of that than the effect of aiming
> to splitting the access patterns between 4 AGs [ ... ]
As to this, in theory even having split the files among 4 AGs,
the upload from system RAM to host adapter RAM and then to disk
could happen by writing first all the dirty blocks for one AG,
then a long seek to the next AG, and so on, and the additional
cost of 3 long seeks would be negligible.
That you report a significant slowdown indicates that this is
not happening, and that likely XFS flushing is happening not in
spacewise order but in timewise order.
The seeks graphs you have gathered indeed indicate that with
'ext4' there is a spacewise flush, while with XFS the flush
alternates constantly among the 4 AGs, instead of doing each AG
in turn. Which seems to indicate an elevator issue or a barrier
issue after the delayed allocator has assigned block addresses
to the various pages being flushed.