Improving XFS file system inode performance

[Jesse Stroik]

XFS community,

I have a couple of medium-sized file systems on an ftp server (10TB file 
system mounted within a 20TB file system).  The load on these file 
systems is getting pretty high because we have many users mirroring 
datasets from the server.  As far as I can tell, the main issue is with 
inode performance.  For example, an 'ls' on a directory may take 20 
seconds to complete.  At any given time, there is > 50 ftp STAT, LIST or 
NLST commands some of which list entire directories or wildcards.

Sadly, the file system was created with 32 bit inodes.  I've remounted 
it with the inode64 option, but I assume performance will be boosted 
primarily when old files are replaced with new files.  Is there anything 
I can do to improve performance now?

I'm also using noatime and logbufs=8.

Performance was fine before the file system was filled -- last week ~8TB 
showed up and filled the 20TB file system.  Since, it has been 
performing poorly.

I'd also be interested in inode cache tuning options specific to XFS. 
i've been having trouble finding documentation on this particular issue.

This is a production file system so please frame your suggestions with 
respect to that.  It is a RHEL 5.5 system running xfsprogs-2.9.4.1 
centos and redhat kernel version 2.6.18-194.17.1 which includes a 
variety of backported xfs fixes.

Best,
Jesse