Inode cache uses a lot of memory

["Sebastian Kun" <seb@consensys.com>]

Hi,

I've been doing some SPEC SFS97 (http://www.spec.org/osg/sfs97r1/) testing
for my company.  SFS is a benchmark
for evaluating NFS performance, measuring both throughput (NFS ops/sec), and
latency (ORT, overall response time).  In our case, the benchmark created a
50GB fileset, consisting of 2 million files and 65000 directories.  During
the test, approximately 10% of the fileset was accessed (read, write,
getattr, lookup, etc.).

I have some questions about some unusual behaviour I've noticed under XFS.
There seems to be a problem with freeing up memory used for the inode cache.
During the test run, I periodically checked the amount of free memory using
the top command:

Mem:    2063220K av, 2050000K used, 13220K free, 0K shrd, 640K buff 138420K
cached

There's a lot of memory unaccounted for (even taking into account userspace
apps).  I ran 'cat /proc/slabinfo' which came up with the following entries
of interest:

xfs_ili           505795 505848    136 18066 18066
xfs_inode         1433589 1501336    468 187667 187667
inode_cache       1188597 1279404    512 182772 182772

(The format of these entries is [name] [active_obj] [total_obj] [obj_size]
[active_pages] [total_pages])

As you can see, the xfs_inode cache takes up over 180,000 pages, or around
750MB of memory.  The inode_cache takes up another 700MB.  Even after doing
several gigabytes of I/O to another filesystem (reiserfs), the memory used
by the inode caches was still around 1GB.  This memory is only freed when
the filesystem is unmounted.

Question 1: Is this behaviour normal for XFS?
Question 2: Is there any way to limit the amount of memory used by the inode
cache?

System stats are:

Running on a dual 1GHz Intel STL2, 2GB RAM, 14-disk 1TB RAID0 array.
Kernel version: 2.4.16 SMP, 4GB Highmem, 2GB/2GB split
XFS patch: xfs-2.4.16-all-i386 (december 3), compiled into the kernel
mkfs.xfs -f -d agsize=1g,sunit=128,swidth=1792 /dev/rze1
mount /dev/rze1 /sfs -o defaults,noatime,sunit=128,swidth=1792,logbufs=8

Thank you,

Sebastian Kun
Consensys Corp.
http://www.consensys.com/