Premature "No Space left on device" on XFS

[Bernhard Schmidt]

Hi,

>> lxmhs45:~ # xfs_info /dev/sdb
>> meta-data=/dev/sdb               isize=256    agcount=4, agsize=655360 blks
>>           =                       sectsz=512   attr=2
>> data     =                       bsize=4096   blocks=2621440, imaxpct=50
>                                                                  ^^^^^^^^^^
>
> And there lies the reason you are getting the filesystem into this
> situation - you're allowing a very large number of inodes to be created
> in the filesystem.

Ah, sorry, I changed that to 50% _after_ the first fuckup due to a 
suggestion on the postfix ml, it used to be the default 25% before.
>
> I'd suggest that for your workload, you need to allow at least 10GB
> of disk space per million inodes. Because of the number of small
> files, XFS is going to need a much larger amount of free space
> available to prevent aging related freespace fragmentation problems.
> The above ratio results in a maximum space usage of about 50%, which
> will avoid such issues. If you need to hold 2 million files, use a
> 20GB filesystem...

I don't need to hold 2 million files, 1 million might be enough, I have 
to make sure I cannot run out of inodes way before I run out of free space.

Generally speaking I have the following problem:

External nodes are submitting data (mails) to this system as fast as 
they can. The mails can be between 800 bytes and several megabytes. 
There are 50 receiver that write those mails as single files flat in a 
single directory.

There are 4 worker threads that process a _random_ file out of this 
directory. To process it they need to be able to create a temporary file 
on the same filesystem. Together they are slower than the 50 receivers 
(they can process maybe 20% of the incoming rate), which means that this 
incoming directory is going to fill. For the sake of the argument lets 
assume that the amount of mails to be sent is unlimited.

The only knob the software knows to prevent this from going over is free 
disk space. When free disk space is lower than 2 Gigabyte, the 
acceptance of new mails is blocked gracefully until there is free space 
again.

It has, however, no way to deal with ENOSPC before that. When it cannot 
create new files due no free inodes (ext4 with default settings) or 
fragmentation in XFS, it breaks quite horribly and cannot recover by itself.

Can I avoid XFS giving ENOSPC due to inode shortage even in worst case 
situations? I would be fine preallocating 1 GB for inode storage if that 
would fix the problem. ext4 with bytes-per-inode = blocksize does this fine.

You mentioned an aging problem with XFS. I guess you mean that an XFS 
filesystem will get slower/more fragmented by time with abuse like this. 
These mail submission above will happen in bursts, during normal times 
it will go down to << 1000 files on the entire filesystem (empty 
incoming directory). Is this enough for XFS to "fix itself"?

BTW, the software can hash the incoming directory in 16 or 16x16 
subdirectories. Would that help XFS in any way with those filesizes? At 
first glance I would have said yes, but due to the random access in 
those directories it would still have the entire spool as workload.

Bernhard