No space left on device on xfs filesystem with 1.6TB free

[Dave Chinner]

On Wed, Jan 21, 2009 at 10:33:10AM +0000, Owen Dunn wrote:
> I'm running SuSE Linux Enterprise Server 10 SP2, on a 32-bit i386 Linux 
> machine with kernel version 2.6.16.60-0.33-smp.  I appear to have version 
> 2.9.4 of xfsprogs.
> 
> I have a 6TB xfs filesystem which has been grown twice from 2TB, and which 
> appears to have 1.6TB and plenty of inodes free.  However, attempts to 
> create a new file fail with ENOSPC:

You've run out of inode space below 1TB or free space is
sufficiently fragmented below 1TB that new inode chunks cannot
be allocated.

> sf3:/EXPORT/space # touch splat
> touch: cannot touch `splat': No space left on device
> sf3:/EXPORT/space # df -h .
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/bigvg-lv1
>                        6.0T  4.5T  1.6T  75% /EXPORT/space
> sf3:/EXPORT/space # df -ih .
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/mapper/bigvg-lv1
>                          4.0G    887K    4.0G    1% /EXPORT/space

df -i really doesn't tell you anything useful on filesystems that
dynamically allocate inodes like XFS.

> Any idea what's going on and how to fix it? 

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com