On Wed, Sep 07, 2016 at 06:36:19PM +0800, Lin Feng wrote:
> Hi all nice xfs folks,
>
> I'm a rookie and really fresh new in xfs and currently I ran into an
> issue same as the following link described:
> http://oss.sgi.com/archives/xfs/2014-04/msg00058.html
>
> In my box(running cephfs osd using xfs kernel 2.6.32-358) and I sum
> all possible memory counter can be find but it seems that nearlly
> 26GB memory has gone and they are back after I echo 2 >
> /proc/sys/vm/drop_caches, so seems these memory can be reclaimed by
> slab.
It isn't "reclaimed by slab". The XFS metadata buffer cache is
reclaimed by a memory shrinker, which are for reclaiming objects
from caches that aren't the page cache. "echo 2 >
/proc/sys/vm/drop_caches" runs the memory shrinkers rather than page
cache reclaim. Many slab caches are backed by memory shrinkers,
which is why it is thought that "2" is "slab reclaim"....
> And according to what David said replying in the list:
..
> That's where your memory is - in metadata buffers. The xfs_buf slab
> entries are just the handles - the metadata pages in the buffers
> usually take much more space and it's not accounted to the slab
> cache nor the page cache.
That's exactly the case.
> Minimum / Average / Maximum Object : 0.02K / 0.33K / 4096.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 4383036 4383014 99% 1.00K 1095759 4 4383036K xfs_inode
> 5394610 5394544 99% 0.38K 539461 10 2157844K xfs_buf
So, you have *5.4 million* active metadata buffers. Each buffer will
hold 1 or 2 4k pages on your kernel, so simple math says 4M * 4k +
1.4M * 8k = 26G. There's no missing counter here....
Obviously your workload is doing something extremely metadata
intensive to have a cache footprint like this - you have more cached
buffers than inodes, dentries, etc. That in itself is very unusual -
can you describe what is stored on that filesystem and how large the
attributes being stored in each inode are?
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|