Question about continous blocks for inode due to 'no space left on device' problem

[Michel Verbraak <info1stsetup@xxxxxxxxx>]

All,

Recently we ran into a problem where our filesystem (300GB in size) reported 'no space left on device' (ENOSPC) but when we looked at disk space usage and inode usage it was around 52% for disk space and 11% for inode. (sorry do not have a save of the output of df command).

Second we looked at the xfs_info and xfs_db -c freesp for each AG. Sadly we do not have a copy anymore of the xfs_info from that time but I do have that of the freesp (at end of this mail).

As we did not fully understand the output of the 'xfs_db -c freesp' we simply added another 100GB to the filesystem and our problem was solved for now. The AG count went from 4 to 6.

For now we are looking into how can we detect (monitor) this filesystem and see the same problem creeping up to us before it happens.
After searching online, reading a lot of mail threads and info pages it looks like we hit the problem of not having any continuous blocks of at least 16k to hold a new inode (http://oss.sgi.com/pipermail/xfs/2014-September/038301.html).

Currently the xfs_info (after adding 100GB) looks like this:

meta-data="" isize=256ÂÂÂ agcount=6, agsize=19660800 blks
ÂÂÂÂÂÂÂÂ =ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ sectsz=512ÂÂ attr=2
dataÂÂÂÂ =ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ bsize=4096 blocks=104857600, imaxpct=25
ÂÂÂÂÂÂÂÂ =ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ sunit=0ÂÂÂÂÂ swidth=0 blks
namingÂÂ =version 2ÂÂÂÂÂÂÂÂÂÂÂÂÂ bsize=4096ÂÂ ascii-ci=0
logÂÂÂÂÂ =internalÂÂÂÂÂÂÂÂÂÂÂÂÂÂ bsize=4096ÂÂ blocks=38400, version=2
ÂÂÂÂÂÂÂÂ =ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ sectsz=512ÂÂ sunit=0 blks, lazy-count=1
realtime =noneÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ extsz=4096ÂÂ blocks=0, rtextents=0

The output of 'xfs_db -c freesp' shows a historgram. We first thought the "from" column shows the minimal number of continous blocks. And as each block is 4K (4096) the row with 'from 4 to 7' contains at least 4 continuous blocks with a minimal size of 4x4k=16k. So our filesystem has enough blocks and extents which are 16k for new inodes, atleast what we think.

- So why did we still get the ENOSPC error?
- Where can I find a better explanation of the 'xfs_db -c freesp' output?
- What is the best way to monitor/detect this problem before it happens.

Some extra info about the system and files on the system:
- Ubuntu 12.04.5 LTS
- Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- 32 million files of which about 75% is smaller than 1k. Files are separated over different folders to keep the number of files per folder low.

- mount options: defaults,noatime,inode64,nobarrier

Regards,

Michel Verbraak.

Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
 from to extents blocks pct
ÂÂÂÂÂ 1ÂÂÂÂÂÂ 1 3282633 3282633ÂÂ 9.03
ÂÂÂÂÂ 2ÂÂÂÂÂÂ 3 3416223 8372325Â 23.03
ÂÂÂÂÂ 4ÂÂÂÂÂÂ 7 6175009 24700036Â 67.94

Store 03 (root@server):~# xfs_db -c "freesp -a 0" /dev/sdb
 from to extents blocks pct
ÂÂÂÂÂ 1ÂÂÂÂÂÂ 1Â 813491Â 813491ÂÂ 8.95
ÂÂÂÂÂ 2ÂÂÂÂÂÂ 3Â 806828 1953270Â 21.50
ÂÂÂÂÂ 4ÂÂÂÂÂÂ 7 1579585 6318340Â 69.55

Store 03 (root@server):~# xfs_db -c "freesp -a 1" /dev/sdb
 from to extents blocks pct
ÂÂÂÂÂ 1ÂÂÂÂÂÂ 1Â 835174Â 835174ÂÂ 9.20
ÂÂÂÂÂ 2ÂÂÂÂÂÂ 3Â 953626 2381033Â 26.22
ÂÂÂÂÂ 4ÂÂÂÂÂÂ 7 1465787 5863148Â 64.58

Store 03 (root@server):~# xfs_db -c "freesp -a 2" /dev/sdb
 from to extents blocks pct
ÂÂÂÂÂ 1ÂÂÂÂÂÂ 1Â 819098Â 819098ÂÂ 9.05
ÂÂÂÂÂ 2ÂÂÂÂÂÂ 3Â 846158 2074235Â 22.93
ÂÂÂÂÂ 4ÂÂÂÂÂÂ 7 1538593 6154372Â 68.02

Store 03 (root@server):~# xfs_db -c "freesp -a 3" /dev/sdb
 from to extents blocks pct
ÂÂÂÂÂ 1ÂÂÂÂÂÂ 1Â 814870Â 814870ÂÂ 8.91
ÂÂÂÂÂ 2ÂÂÂÂÂÂ 3Â 809611 1963787Â 21.48
ÂÂÂÂÂ 4ÂÂÂÂÂÂ 7 1591044 6364176Â 69.61