Inode and dentry cache behavior

[Dave Chinner]

On Thu, Apr 23, 2015 at 04:48:51PM -0700, Shrinand Javadekar wrote:
> > from the iostat log:
> >
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> > .....
> > dm-6              0.00     0.00    0.20   22.40     0.00     0.09    8.00    22.28  839.01 1224.00  835.57  44.25 100.00
> > dm-7              0.00     0.00    0.00    1.20     0.00     0.00    8.00     2.82 1517.33    0.00 1517.33 833.33 100.00
> > dm-8              0.00     0.00    0.00  195.20     0.00     0.76    8.00  1727.51 4178.89    0.00 4178.89   5.12 100.00
> > ...
> > dm-7              0.00     0.00    0.00    0.00     0.00     0.00     0.00     1.00    0.00    0.00    0.00   0.00 100.00
> > dm-8              0.00     0.00    0.00    0.00     0.00     0.00     0.00  1178.85    0.00    0.00    0.00   0.00 100.00
> >
> > dm-7 is showing almost a second for single IO wait times, when it is
> > actually completing IO. dm-8 has a massive queue depth - I can only
> > assume you've tuned  sys/block/*/queue/nr_requests to something
> > really large? But like dm-7, it's showing very long IO times, and
> > that's likely the source of your latency problems.
> 
> I see that /sys/block/*/queue/nr_requests is set to 128 which is way
> less than the queue depth shown in the iostat numbers. What gives?

No idea, but it's indicative of a problem below XFS. Work out what
is happening with your storage hardware first, then work your way up
the stack...

> One other observation we had was that xfs shows a large amount of
> directory fragmentation. Directory fragmentation was shown at ~40%
> whereas file fragmentation was very low at 0.1%.

Pretty common. Directories are only accessed a single block at a
time, and sequential offset reads are pretty rare, so fragmentation
makes little difference to performance. You're seeing almost zero
read IO load, so the directory layout is not a concern for this
workload.

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com