On Fri, May 12, 2006 at 12:36:21PM -0700, Peter Broadwell wrote:
> I seem to be having the same problem as CHIKAMA Masaki was having in
> December 7, 2005,
> namely "chown -R" running very slowly when hitting lots of files (~17
> million in my case).
The problem is different because there's no OOM killer
being invoked, right? All you see is a slowdown? How much CPU
time is the chmod consuming?
> I'm most interested in anything to (safely) speed this up on a live file
> system as it
> has been running for nearly 24 hours so far... not hung or corrupted
> anything as far
> as I can tell.
Well, doing a chmod on a single file requires an inode read,
a log write, and eventually a inode write.
> xfs_chashlist 205900 385952 32 112 1 : tunables 120 60 8
> xfs_ili 273754 273760 192 20 1 : tunables 120 60 8
> xfs_inode 275317 275317 528 7 1 : tunables 54 27 8
> xfs_vnode 275316 275316 632 6 1 : tunables 54 27 8
> dentry_cache 252909 252909 224 17 1 : tunables 120 60 8
From the inode to cluster ratio (xfs_inode/xfs_chashlist), you've
got very sparse inode clusters, so each inode read and write will do
a disk I/O. So, two I/Os per file chmod() plus a log write every few
files plus directory reads. That makes it roughly 40 million I/Os
to do your recursive chmod.
On a single disk sustaining 200 I/Os per second, I'd expect it to
take more than a couple of days to complete the recursive chmod. Your
filesystem is going to be slow while this is going on as well.
> peter@cl1 /data $ df -k
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/md/1 449447808 338816792 110631016 76% /data
peter@cl1 /data $ xfs_info /data
data = bsize=4096 blocks=112394720, imaxpct=25
= sunit=16 swidth=64 blks, unwritten=1
So a 64k stripe unit and 4-unit wide stripe. What RAID level are you
using for your stripe? What's the spindle speed of the disks?
log =internal bsize=4096 blocks=32768, version=1
With a 128MB version 1 log.
If you were using version 2 logs, I'd suggest using a larger
log buffer size to reduce the number of log writes. That would
help quite a bit. Other than that, I can't think of much you
could tune to help here. When you need to do that many I/Os,
the only thing that speeds it up is to have lots of spindles...
R&D Software Enginner
SGI Australian Software Group