> from the iostat log:
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> .....
> dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00
> 22.28 839.01 1224.00 835.57 44.25 100.00
> dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00
> 2.82 1517.33 0.00 1517.33 833.33 100.00
> dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00
> 1727.51 4178.89 0.00 4178.89 5.12 100.00
> ...
> dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 1.00 0.00 0.00 0.00 0.00 100.00
> dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 1178.85 0.00 0.00 0.00 0.00 100.00
>
> dm-7 is showing almost a second for single IO wait times, when it is
> actually completing IO. dm-8 has a massive queue depth - I can only
> assume you've tuned sys/block/*/queue/nr_requests to something
> really large? But like dm-7, it's showing very long IO times, and
> that's likely the source of your latency problems.
I see that /sys/block/*/queue/nr_requests is set to 128 which is way
less than the queue depth shown in the iostat numbers. What gives?
One other observation we had was that xfs shows a large amount of
directory fragmentation. Directory fragmentation was shown at ~40%
whereas file fragmentation was very low at 0.1%.
|