<div dir="ltr">I have checked icount and ifree, but I found there are about 11.8 percent free, so the free inode should not be too few.<div><br></div><div>Here's the detail log, any new clue? </div><div><br></div><div>
<div># mount /dev/sda4 /data1/</div><div># xfs_info /data1/</div><div>meta-data=/dev/sda4 isize=256 agcount=4, agsize=142272384 blks</div><div> = sectsz=512 attr=2, projid32bit=0</div>
<div>data = bsize=4096 blocks=569089536, imaxpct=5</div><div> = sunit=0 swidth=0 blks</div><div>naming =version 2 bsize=4096 ascii-ci=0</div>
<div>log =internal bsize=4096 blocks=277875, version=2</div><div> = sectsz=512 sunit=0 blks, lazy-count=1</div><div>realtime =none extsz=4096 blocks=0, rtextents=0</div>
<div># umount /dev/sda4</div><div># xfs_db /dev/sda4</div><div>xfs_db> sb 0</div><div>xfs_db> p</div><div>magicnum = 0x58465342</div><div>blocksize = 4096</div><div>dblocks = 569089536</div><div>rblocks = 0</div><div>
rextents = 0</div><div>uuid = 13ecf47b-52cf-4944-9a71-885bddc5e008</div><div>logstart = 536870916</div><div>rootino = 128</div><div>rbmino = 129</div><div>rsumino = 130</div><div>rextsize = 1</div><div>agblocks = 142272384</div>
<div>agcount = 4</div><div>rbmblocks = 0</div><div>logblocks = 277875</div><div>versionnum = 0xb4a4</div><div>sectsize = 512</div><div>inodesize = 256</div><div>inopblock = 16</div><div>fname = "\000\000\000\000\000\000\000\000\000\000\000\000"</div>
<div>blocklog = 12</div><div>sectlog = 9</div><div>inodelog = 8</div><div>inopblog = 4</div><div>agblklog = 28</div><div>rextslog = 0</div><div>inprogress = 0</div><div>imax_pct = 5</div><div>icount = 220619904</div><div>
ifree = 26202919</div><div>fdblocks = 147805479</div><div>frextents = 0</div><div>uquotino = 0</div><div>gquotino = 0</div><div>qflags = 0</div><div>flags = 0</div><div>shared_vn = 0</div><div>inoalignmt = 2</div><div>unit = 0</div>
<div>width = 0</div><div>dirblklog = 0</div><div>logsectlog = 0</div><div>logsectsize = 0</div><div>logsunit = 1</div><div>features2 = 0xa</div><div>bad_features2 = 0xa</div><div>xfs_db> sb 1</div><div>xfs_db> p</div>
<div>magicnum = 0x58465342</div><div>blocksize = 4096</div><div>dblocks = 569089536</div><div>rblocks = 0</div><div>rextents = 0</div><div>uuid = 13ecf47b-52cf-4944-9a71-885bddc5e008</div><div>logstart = 536870916</div><div>
rootino = 128</div><div>rbmino = null</div><div>rsumino = null</div><div>rextsize = 1</div><div>agblocks = 142272384</div><div>agcount = 4</div><div>rbmblocks = 0</div><div>logblocks = 277875</div><div>versionnum = 0xb4a4</div>
<div>sectsize = 512</div><div>inodesize = 256</div><div>inopblock = 16</div><div>fname = "\000\000\000\000\000\000\000\000\000\000\000\000"</div><div>blocklog = 12</div><div>sectlog = 9</div><div>inodelog = 8</div>
<div>inopblog = 4</div><div>agblklog = 28</div><div>rextslog = 0</div><div>inprogress = 1</div><div>imax_pct = 5</div><div>icount = 0</div><div>ifree = 0</div><div>fdblocks = 568811645</div><div>frextents = 0</div><div>uquotino = 0</div>
<div>gquotino = 0</div><div>qflags = 0</div><div>flags = 0</div><div>shared_vn = 0</div><div>inoalignmt = 2</div><div>unit = 0</div><div>width = 0</div><div>dirblklog = 0</div><div>logsectlog = 0</div><div>logsectsize = 0</div>
<div>logsunit = 1</div><div>features2 = 0xa</div><div>bad_features2 = 0xa</div><div>xfs_db> sb 2</div><div>xfs_db> p</div><div>magicnum = 0x58465342</div><div>blocksize = 4096</div><div>dblocks = 569089536</div><div>
rblocks = 0</div><div>rextents = 0</div><div>uuid = 13ecf47b-52cf-4944-9a71-885bddc5e008</div><div>logstart = 536870916</div><div>rootino = null</div><div>rbmino = null</div><div>rsumino = null</div><div>rextsize = 1</div>
<div>agblocks = 142272384</div><div>agcount = 4</div><div>rbmblocks = 0</div><div>logblocks = 277875</div><div>versionnum = 0xb4a4</div><div>sectsize = 512</div><div>inodesize = 256</div><div>inopblock = 16</div><div>fname = "\000\000\000\000\000\000\000\000\000\000\000\000"</div>
<div>blocklog = 12</div><div>sectlog = 9</div><div>inodelog = 8</div><div>inopblog = 4</div><div>agblklog = 28</div><div>rextslog = 0</div><div>inprogress = 1</div><div>imax_pct = 5</div><div>icount = 0</div><div>ifree = 0</div>
<div>fdblocks = 568811645</div><div>frextents = 0</div><div>uquotino = 0</div><div>gquotino = 0</div><div>qflags = 0</div><div>flags = 0</div><div>shared_vn = 0</div><div>inoalignmt = 2</div><div>unit = 0</div><div>width = 0</div>
<div>dirblklog = 0</div><div>logsectlog = 0</div><div>logsectsize = 0</div><div>logsunit = 1</div><div>features2 = 0xa</div><div>bad_features2 = 0xa</div><div>xfs_db> sb 3</div><div>xfs_db> p</div><div>magicnum = 0x58465342</div>
<div>blocksize = 4096</div><div>dblocks = 569089536</div><div>rblocks = 0</div><div>rextents = 0</div><div>uuid = 13ecf47b-52cf-4944-9a71-885bddc5e008</div><div>logstart = 536870916</div><div>rootino = 128</div><div>rbmino = null</div>
<div>rsumino = null</div><div>rextsize = 1</div><div>agblocks = 142272384</div><div>agcount = 4</div><div>rbmblocks = 0</div><div>logblocks = 277875</div><div>versionnum = 0xb4a4</div><div>sectsize = 512</div><div>inodesize = 256</div>
<div>inopblock = 16</div><div>fname = "\000\000\000\000\000\000\000\000\000\000\000\000"</div><div>blocklog = 12</div><div>sectlog = 9</div><div>inodelog = 8</div><div>inopblog = 4</div><div>agblklog = 28</div><div>
rextslog = 0</div><div>inprogress = 1</div><div>imax_pct = 5</div><div>icount = 0</div><div>ifree = 0</div><div>fdblocks = 568811645</div><div>frextents = 0</div><div>uquotino = 0</div><div>gquotino = 0</div><div>qflags = 0</div>
<div>flags = 0</div><div>shared_vn = 0</div><div>inoalignmt = 2</div><div>unit = 0</div><div>width = 0</div><div>dirblklog = 0</div><div>logsectlog = 0</div><div>logsectsize = 0</div><div>logsunit = 1</div><div>features2 = 0xa</div>
<div>bad_features2 = 0xa</div></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-08-25 13:18 GMT+08:00 Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Mon, Aug 25, 2014 at 11:34:34AM +0800, Zhang Qiang wrote:<br>
> Dear XFS community & developers,<br>
><br>
> I am using CentOS 6.3 and xfs as base file system and use RAID5 as hardware<br>
> storage.<br>
><br>
> Detail environment as follow:<br>
> OS: CentOS 6.3<br>
> Kernel: kernel-2.6.32-279.el6.x86_64<br>
> XFS option info(df output): /dev/sdb1 on /data type xfs<br>
> (rw,noatime,nodiratime,nobarrier)<br>
><br>
> Detail phenomenon:<br>
><br>
> # df<br>
> Filesystem Size Used Avail Use% Mounted on<br>
> /dev/sda1 29G 17G 11G 61% /<br>
> /dev/sdb1 893G 803G 91G 90% /data<br>
> /dev/sda4 2.2T 1.6T 564G 75% /data1<br>
><br>
> # time touch /data1/1111<br>
> real 0m23.043s<br>
> user 0m0.001s<br>
> sys 0m0.349s<br>
><br>
> # perf top<br>
> Events: 6K cycles<br>
> 16.96% [xfs] [k] xfs_inobt_get_rec<br>
> 11.95% [xfs] [k] xfs_btree_increment<br>
> 11.16% [xfs] [k] xfs_btree_get_rec<br>
> 7.39% [xfs] [k] xfs_btree_get_block<br>
> 5.02% [xfs] [k] xfs_dialloc<br>
> 4.87% [xfs] [k] xfs_btree_rec_offset<br>
> 4.33% [xfs] [k] xfs_btree_readahead<br>
> 4.13% [xfs] [k] _xfs_buf_find<br>
> 4.05% [kernel] [k] intel_idle<br>
> 2.89% [xfs] [k] xfs_btree_rec_addr<br>
> 1.04% [kernel] [k] kmem_cache_free<br>
><br>
><br>
> It seems that some xfs kernel function spend much time (xfs_inobt_get_rec,<br>
> xfs_btree_increment, etc.)<br>
><br>
> I found a bug in bugzilla [1], is that is the same issue like this?<br>
<br>
</div></div>No.<br>
<div class=""><br>
> It's very greatly appreciated if you can give constructive suggestion about<br>
> this issue, as It's really hard to reproduce from another system and it's<br>
> not possible to do upgrade on that online machine.<br>
<br>
</div>You've got very few free inodes, widely distributed in the allocated<br>
inode btree. The CPU time above is the btree search for the next<br>
free inode.<br>
<br>
This is the issue solved by this series of recent commits to add a<br>
new on-disk free inode btree index:<br>
<br>
53801fd xfs: enable the finobt feature on v5 superblocks<br>
0c153c1 xfs: report finobt status in fs geometry<br>
a3fa516 xfs: add finobt support to growfs<br>
3efa4ff xfs: update the finobt on inode free<br>
2b64ee5 xfs: refactor xfs_difree() inobt bits into xfs_difree_inobt() helper<br>
6dd8638 xfs: use and update the finobt on inode allocation<br>
0aa0a75 xfs: insert newly allocated inode chunks into the finobt<br>
9d43b18 xfs: update inode allocation/free transaction reservations for finobt<br>
aafc3c2 xfs: support the XFS_BTNUM_FINOBT free inode btree type<br>
8e2c84d xfs: reserve v5 superblock read-only compat. feature bit for finobt<br>
57bd3db xfs: refactor xfs_ialloc_btree.c to support multiple inobt numbers<br>
<br>
Which is of no help to you, however, because it's not available in<br>
any CentOS kernel.<br>
<br>
There's really not much you can do to avoid the problem once you've<br>
punched random freespace holes in the allocated inode btree. IT<br>
generally doesn't affect many people; those that it does affect are<br>
normally using XFS as an object store indexed by a hard link farm<br>
(e.g. various backup programs do this).<br>
<br>
If you dump the superblock via xfs_db, the difference between icount<br>
and ifree will give you idea of how much "needle in a haystack"<br>
searching is going on. You can probably narrow it down to a specific<br>
AG by dumping the AGI headers and checking the same thing. filling<br>
in all the holes (by creating a bunch of zero length files in the<br>
appropriate AGs) might take some time, but it should make the<br>
problem go away until you remove more filesystem and create random<br>
free inode holes again...<br>
<br>
Cheers,<br>
<br>
Dave.<br>
<span class="HOEnZb"><font color="#888888">--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com">david@fromorbit.com</a><br>
</font></span></blockquote></div><br></div>