xfs
[Top] [All Lists]

Re: ENOSPC at 90% with plenty of inodes

To: xfs@xxxxxxxxxxx
Subject: Re: ENOSPC at 90% with plenty of inodes
From: Stan Hoeppner <stan@xxxxxxxxxxxxxxxxx>
Date: Tue, 12 Oct 2010 18:21:42 -0500
In-reply-to: <20101012102954.GQ4681@dastard>
References: <AANLkTinwnOm0V=9WwbXiUGFnyjCwT3GUHo0hiQahQNUV@xxxxxxxxxxxxxx> <20101008225146.GJ4681@dastard> <AANLkTikLyEuSoFRVwirKLws97Vemz4ScNHOqu2Jc5sWR@xxxxxxxxxxxxxx> <20101011223507.GB32255@dastard> <4CB3B964.40109@xxxxxxxxxxxxxxxxx> <20101012102954.GQ4681@dastard>
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.2.9) Gecko/20100915 Thunderbird/3.1.4
Dave Chinner put forth on 10/12/2010 5:29 AM:
> On Mon, Oct 11, 2010 at 08:27:00PM -0500, Stan Hoeppner wrote:
>> Dave Chinner put forth on 10/11/2010 5:35 PM:
>>> On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
>>>> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
>>>>> Sounds like fragmented free space. What is the output of:
>>>>>
>>>>> # xfs_db -r -c "freesp -s" <device>
>>>>
>>>> # xfs_db -r -c "freesp -s" /dev/sdb
>>>>    from      to extents  blocks    pct
>>>>       1       1 2298052 2298052  40.52
>>>>       2       3 1568338 3337017  58.84
>>>>       4       7    8432   35716   0.63
>>>>       8      15      50     423   0.01
>>>> total free extents 3874872
>>>> total free blocks 5671208
>>>> average free extent size 1.46359
>>>>
>>>> Which seems to say there are a few tiny pieces of free space
>>>> available? The files that were failing to be written were a few
>>>> hundred bytes in size.
>>>
>>> The error has nothing to do with the size of the files, but
>>> everything to do with being able to allocate more inodes. Inode
>>> allocation requires 4 contiguous blocks (for 256 byte inodes, more
>>> for larger inodes) with alignment constraints. That means when you
>>> run out of 8 block or larger free extents, inode allocation will
>>> start failing and you'll get ENOSPC being reported.
>>>
>>>> We haven't seen any errors so far today, but xfs_fsr ran over the
>>>> weekend, so perhaps I guess it's reorganized the filesystem.
>>>
>>> Only a little. xfs_fsr will not improve fragmented free space
>>> conditions (indeed, it normally fragments free space more). The only
>>> way to reduce the fragmentation of free space is to remove a
>>> significant amount of data and inodes from the filesystem...
>>
>> Hay Dave, would a "backup/reformat/restore" help with free space
>> fragmentation in this case?
> 
> Of course. But that's the last resort....
> 
>> If so, could/should the OP specify anything
>> during the mkfs.xfs reformat that may help alleviate or mitigate his
>> problem in the future?
> 
> No. These problems usually appear in filesystems that have run at
> greater than 85-90% full for extended periods of time without being
> emptied at all. Once you start to free up space, it naturally
> defragments itself, but if you never free up any significant amount
> of space in the filesytesm, this cannot occur and so fragmentation
> just keeps getting worse....

So, given that this problem is on a production IMAP server, and the OP
likely can't just willy nilly start deleting user files, would adding
more disk (and assuming he's using LVM or somesuch) and growing the
filesystem alleviate this inode issue?

Or would he be better off adding more disk, creating a new filesystem,
and moving half or so of his mailboxen over to the new filesystem at the
Cyrus (application) level?  I've never used Cyrus, though IIRC Dovecot
can do this "split mail store" setup.

-- 
Stan

<Prev in Thread] Current Thread [Next in Thread>