I've been doing some SPEC SFS97 (http://www.spec.org/osg/sfs97r1/) testing
for my company. SFS is a benchmark
for evaluating NFS performance, measuring both throughput (NFS ops/sec), and
latency (ORT, overall response time). In our case, the benchmark created a
50GB fileset, consisting of 2 million files and 65000 directories. During
the test, approximately 10% of the fileset was accessed (read, write,
getattr, lookup, etc.).
I have some questions about some unusual behaviour I've noticed under XFS.
There seems to be a problem with freeing up memory used for the inode cache.
During the test run, I periodically checked the amount of free memory using
the top command:
Mem: 2063220K av, 2050000K used, 13220K free, 0K shrd, 640K buff 138420K
There's a lot of memory unaccounted for (even taking into account userspace
apps). I ran 'cat /proc/slabinfo' which came up with the following entries
xfs_ili 505795 505848 136 18066 18066
xfs_inode 1433589 1501336 468 187667 187667
inode_cache 1188597 1279404 512 182772 182772
(The format of these entries is [name] [active_obj] [total_obj] [obj_size]
As you can see, the xfs_inode cache takes up over 180,000 pages, or around
750MB of memory. The inode_cache takes up another 700MB. Even after doing
several gigabytes of I/O to another filesystem (reiserfs), the memory used
by the inode caches was still around 1GB. This memory is only freed when
the filesystem is unmounted.
Question 1: Is this behaviour normal for XFS?
Question 2: Is there any way to limit the amount of memory used by the inode
System stats are:
Running on a dual 1GHz Intel STL2, 2GB RAM, 14-disk 1TB RAID0 array.
Kernel version: 2.4.16 SMP, 4GB Highmem, 2GB/2GB split
XFS patch: xfs-2.4.16-all-i386 (december 3), compiled into the kernel
mkfs.xfs -f -d agsize=1g,sunit=128,swidth=1792 /dev/rze1
mount /dev/rze1 /sfs -o defaults,noatime,sunit=128,swidth=1792,logbufs=8