ENOSPC but df and df -i show free space

[Dave Chinner]

On Sun, Jun 19, 2011 at 02:50:39PM -0700, Andy Isaacson wrote:
> % touch /d1/tmp/foo
> touch: cannot touch `/d1/tmp/foo': No space left on device
> % df /d1
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/mapper/vg0-d1   943616000 904690332  38925668  96% /d1

Problems like this will occur if you run your filesystem at > 85-90%
full for extented periods....

> % df -i /d1
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/mapper/vg0-d1   167509008 11806336 155702672    8% /d1
> % sudo xfs_growfs -n /d1
> meta-data=/dev/mapper/vg0-d1     isize=256    agcount=18, agsize=13107200 blks
>          =                       sectsz=512   attr=2
> data     =                       bsize=4096   blocks=235929600, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=25600, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> % grep d1 /proc/mounts
> /dev/mapper/vg0-d1 /d1 xfs rw,relatime,attr2,noquota 0 0
> 
> Obviously I'm missing something, but what?

Most likely is that you have no contiguous free space large enough
to create a new inode chunk.  using xfs_db to dump the freespace
size histogram will tell you if this is the case or not.

> Nothing relevant in dmesg that I can see.  The filesystem started out at
> 200 GB and has been xfs_growfs'd in 100GB increments up to its current
> size of 900 GB.

Ugh. Sounds like you've been running the filesystem near full for
it's entire life....

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com