[Top] [All Lists]

Re: Premature "No Space left on device" on XFS

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: Premature "No Space left on device" on XFS
From: Bernhard Schmidt <berni@xxxxxxxxxxxxx>
Date: Sat, 08 Oct 2011 14:29:34 +0200
Cc: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/simple; d=birkenwald.de; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:from:from :date:date:message-id:received; s=mailout; t=1318076976; bh=NZik f/v0V1F8rJ6LCtPYjpB8gTH5/4YCsFrFjYwHzOI=; b=h055flyLYOBUFd8xXiws 47tuPAzPWGwlf01mjV52CHvGI/zt8wsYkp34EmVpoXQZ6SGsEmzgrVwObzaBnATG hqS5+zVDJRP/oLDAmoo/sSUUEotOhv0mYH0LOdUdJbl4A7PNbr/8lpR0Jj3yMoEh cgJDmRAk4kll3ohmo+K3WtY=
In-reply-to: <20111007231410.GG3159@dastard>
References: <4E8E079B.4040103@xxxxxxxxxxxxx> <20111007013711.GW3159@dastard> <4E8F0385.7060906@xxxxxxxxxxxxx> <20111007231410.GG3159@dastard>
User-agent: Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1

lxmhs45:~ # xfs_info /dev/sdb
meta-data=/dev/sdb               isize=256    agcount=4, agsize=655360 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2621440, imaxpct=50

And there lies the reason you are getting the filesystem into this
situation - you're allowing a very large number of inodes to be created
in the filesystem.

Ah, sorry, I changed that to 50% _after_ the first fuckup due to a suggestion on the postfix ml, it used to be the default 25% before.

I'd suggest that for your workload, you need to allow at least 10GB
of disk space per million inodes. Because of the number of small
files, XFS is going to need a much larger amount of free space
available to prevent aging related freespace fragmentation problems.
The above ratio results in a maximum space usage of about 50%, which
will avoid such issues. If you need to hold 2 million files, use a
20GB filesystem...

I don't need to hold 2 million files, 1 million might be enough, I have to make sure I cannot run out of inodes way before I run out of free space.

Generally speaking I have the following problem:

External nodes are submitting data (mails) to this system as fast as they can. The mails can be between 800 bytes and several megabytes. There are 50 receiver that write those mails as single files flat in a single directory.

There are 4 worker threads that process a _random_ file out of this directory. To process it they need to be able to create a temporary file on the same filesystem. Together they are slower than the 50 receivers (they can process maybe 20% of the incoming rate), which means that this incoming directory is going to fill. For the sake of the argument lets assume that the amount of mails to be sent is unlimited.

The only knob the software knows to prevent this from going over is free disk space. When free disk space is lower than 2 Gigabyte, the acceptance of new mails is blocked gracefully until there is free space again.

It has, however, no way to deal with ENOSPC before that. When it cannot create new files due no free inodes (ext4 with default settings) or fragmentation in XFS, it breaks quite horribly and cannot recover by itself.

Can I avoid XFS giving ENOSPC due to inode shortage even in worst case situations? I would be fine preallocating 1 GB for inode storage if that would fix the problem. ext4 with bytes-per-inode = blocksize does this fine.

You mentioned an aging problem with XFS. I guess you mean that an XFS filesystem will get slower/more fragmented by time with abuse like this. These mail submission above will happen in bursts, during normal times it will go down to << 1000 files on the entire filesystem (empty incoming directory). Is this enough for XFS to "fix itself"?

BTW, the software can hash the incoming directory in 16 or 16x16 subdirectories. Would that help XFS in any way with those filesizes? At first glance I would have said yes, but due to the random access in those directories it would still have the entire spool as workload.


<Prev in Thread] Current Thread [Next in Thread>