Date: Sat, 9 Oct 2010 09:51:46 +1100
From: Dave Chinner <david@fromorbit.com>
To: James Braid <jamesb@loreland.org>
Cc: xfs@oss.sgi.com
Subject: Re: ENOSPC at 90% with plenty of inodes
Message-ID: <20101008225146.GJ4681@dastard>
References: <AANLkTinwnOm0V=9WwbXiUGFnyjCwT3GUHo0hiQahQNUV@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <AANLkTinwnOm0V=9WwbXiUGFnyjCwT3GUHo0hiQahQNUV@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)

On Fri, Oct 08, 2010 at 06:17:52PM +0100, James Braid wrote:
> Hi,
> 
> On a fairly busy mail server hosting about 1000 cyrus mailboxes (each
> message is stored as a separate file), we are observing ENOSPC errors
> from the cyrus pop/imap daemons when attempting to create (tiny)
> files. e.g.
> 
> Oct  8 12:18:49 xxx cyrus/pop3[14119]: IOERROR: creating
> /user/imap/proc/14119: No space left on device
> Oct  8 12:18:49 xxx cyrus/lmtpunix[14361]: IOERROR: creating quota
> file /user/imap/quota/m/user.xx.NEW: No space left on device
> Oct  8 12:18:49 xxx cyrus/imap[12804]: IOERROR: creating quota file
> /user/imap/quota/d/user.xxx.NEW: No space left on device
> 
> The filesystem appears to have plenty of space available:
> 
> # df -i /user
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sdb1            102294688 10201313 92093375   10% /user
> # df /user
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdb1            213014148 190023408  22990740  90% /user
> 
> Any ideas for how can we go about debugging where ENOSPC is coming
> from? Unfortunately it's running a vintage 2.6.24 kernel (Ubuntu 8.04)
> - we're trying to get it upgraded to 10.04 (2.6.32)

Sounds like fragmented free space. What is the output of:

# xfs_db -r -c "freesp -s" <device>

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com