On Mon, Jun 18, 2007 at 02:33:29PM -0400, Mark Seger wrote:
> First of all I wasn't sure who to send this to so I copied all the
> addresses in the 'maintainers' document...
Just the mailing lists are sufficient.
> There appears to be a critical problem reading more than 1M xfs files in
> the same directory using nfs. Shortly after beginning the reads, the
> CPU load hits a fairly steady 25% system load and the nfs read also rate
> drops well below 100 read/sec and eventually falls below 10. I'm doing
> these tests with 4096 byte files on a two-socket dual-core operteron
> with 8GB ram. See the attachment named 'hosed.txt' which shows what the
> cpu, disk, network and nfs were doing during the timeframe that thing
> went bad.
Sounds like you are running out of memory to cache the workload in.
The readdir load indicates that you are probably running out of dentry/inode
cache space, and so every lookup is having to re-read the inodes
from disk. i.e. readdir and stat are necessary.
I'd suggest looking at /proc/slabinfo (slabtop helps here) and
/proc/meminfo to determine how much of your working set of inodes
are being held in cache and how quickly they are being recycled.
perhaps fiddling with /proc/sys/vm/vfs_cache_pressure will help
keep inodes/dentryies in memory over page cache pages...
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
|