"No space left on device" from hell

[Arkadiusz MiÅkiewicz <arekm@xxxxxxxx>]

Hello.

3.10.40 kernel and creating files causes "No space left on device". After 
deleting some files I can create new files again until the problem happens 
again.

Any ideas what is going on and, what is more important, how to fix the 
problem?

# df -h
/dev/mapper/vgsys-home  600G  368G  232G  62% /home

# df -i
/dev/mapper/vgsys-home 629145600 7982895 621162705    2% /home

# xfs_info /home
meta-data=/dev/mapper/vgsys-home isize=256    agcount=8, agsize=19660800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=157286400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=38400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=

imaxpct=25% but current inodes take only about 2GB, much less that 25% of 
600G, so that shouldn't be problem

/dev/mapper/vgsys-home /home xfs 
rw,nosuid,nodev,relatime,attr2,inode64,usrquota,prjquota 0 0

so inode64 is not a problem

# xfs_quota -c  "quota -p 0" /home
Disk quotas for Project #0 (0)
Filesystem              Blocks      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                      27848804          0          0   00 [--------] /home
# xfs_quota -c  "quota -u 0" /home
Disk quotas for User root (0)
Filesystem              Blocks      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                        103484          0          0   00 [--------] /home
# xfs_quota -c  "quota -i 0" /home
Disk quotas for User root (0)
Filesystem               Files      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                          8336          0          0   00 [--------] /home

so quota is not a problem, too.

# xfs_db -r "-c freesp -s"  /dev/mapper/vgsys-home
   from      to extents  blocks    pct
      1       1    3426    3426   0.01
      2       3    3319    8026   0.01
      4       7    3694   19027   0.03
      8      15   23225  330378   0.54
     16      31     396    8180   0.01
     32      63      75    3256   0.01
     64     127      44    3443   0.01
    128     255      26    4892   0.01
    256     511      32   10737   0.02
    512    1023       5    3867   0.01
   1024    2047       8   13576   0.02
   2048    4095       5   13893   0.02
   4096    8191       3   16896   0.03
   8192   16383       1   15532   0.03
  16384   32767       1   17731   0.03
1048576 2097151       1 1414952   2.33
16777216 19660800       3 58913965  96.90
total free extents 34264
total free blocks 60801777
average free extent size 1774.51

unfortunately no idea how to interpret this. Man page isn't helpful.


http://sprunge.us/IVjE
(trace-cmd of touch /home/x ... but looks to be partial only)

-- 
Arkadiusz MiÅkiewicz, arekm / maven.pl