Inode and dentry cache behavior

[Dave Chinner]

On Thu, Apr 23, 2015 at 12:50:15PM -0700, Shrinand Javadekar wrote:
> Hi,
> 
> I am running Openstack Swift on a single server with 8 disks. All
> these 8 disks are formatted with default XFS parameters. Each disk has
> a capacity of 3TB. The machine has 64GB of data.
> 
> Here's what Openstack Swift does:
....
> * We observe that the time for fsync remains pretty much constant throughout.
> * What seems to be causing the performance to nosedive, is that inode
> and dentry caching doesn't seem to be working.
> * For experiment sake, we set vfs_cache_pressure to 0 so there would
> be no reclaiming of inode and dentry cache entries. However, that does
> not seem to help.
> * We see openat() calls taking close to 1 second.
> 
> Any ideas, what might be causing this behavior? Are there other
> params, specifically, xfs params that can be tuned for this workload.
> The sequence of events above is the typical workload, at high
> concurrency.

Work out why you're disks are reporting 100% utilisation when they
have little or no IO being issued to them.

> See the attached files iostat_log and vmstat_log.

from the iostat log:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
.....
dm-6              0.00     0.00    0.20   22.40     0.00     0.09    8.00    22.28  839.01 1224.00  835.57  44.25 100.00                                              
dm-7              0.00     0.00    0.00    1.20     0.00     0.00    8.00     2.82 1517.33    0.00 1517.33 833.33 100.00                                              
dm-8              0.00     0.00    0.00  195.20     0.00     0.76    8.00  1727.51 4178.89    0.00 4178.89   5.12 100.00                                              
...
dm-7              0.00     0.00    0.00    0.00     0.00     0.00     0.00     1.00    0.00    0.00    0.00   0.00 100.00                                              
dm-8              0.00     0.00    0.00    0.00     0.00     0.00     0.00  1178.85    0.00    0.00    0.00   0.00 100.00                                              

dm-7 is showing almost a second for single IO wait times, when it is
actually completing IO. dm-8 has a massive queue depth - I can only
assume you've tuned  sys/block/*/queue/nr_requests to something
really large? But like dm-7, it's showing very long IO times, and
that's likely the source of your latency problems.

i.e. this looks like a storage problem, not an XFS problem.

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com