xfs
[Top] [All Lists]

Re: XFS unlink still slow on 3.1.9 kernel ?

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: XFS unlink still slow on 3.1.9 kernel ?
From: Richard Ems <richard.ems@xxxxxxxxxxxxxxxxx>
Date: Tue, 14 Feb 2012 13:32:00 +0100
Cc: xfs@xxxxxxxxxxx
In-reply-to: <20120214000924.GF14132@dastard>
References: <4F394116.8080200@xxxxxxxxxxxxxxxxx> <20120214000924.GF14132@dastard>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120129 Thunderbird/10.0.1
Hi Dave, hi list,

first thanks for the very detailed reply. Please find below my comments
and questions.

On 02/14/2012 01:09 AM, Dave Chinner wrote:
> On Mon, Feb 13, 2012 at 05:57:58PM +0100, Richard Ems wrote:
>> I am running openSUSE 12.1, kernel 3.1.9-1.4-default. The 20 TB XFS
>> partition is 100% full
> 
> Running filesystems to 100% full is always a bad idea - it causes
> significant increases in fragementation of both data and metadata
> compared to a filesystem that doesn't get past ~90% full.

Yes, true, I know. But I have no other free space for this backups. I am
waiting for a new already ordered system and will have 4 times this
space. So later I will open a new thread asking if my thoughts for
creating this new 80 TB XFS partition are right.



>> I am asking because I am seeing very long times while removing big
>> directory trees. I thought on kernels above 3.0 removing dirs and files
>> had improved a lot, but I don't see that improvement.
> 
> You won't if the directory traversal is seek bound and that is the
> limiting factor for performance.

*Seek bound*? *When* is the directory traversal *seek bound*?


>> This is a backup system running dirvish, so most files in the dirs I am
>> removing are hard links. Almost all of the files do have ACLs set.
> 
> The unlink will have an extra IO to read per inode - the out-of-line
> attribute block, so you've just added 11 million IOs to the 800,000
> the traversal already takes to the unlink overhead. So it's going to
> take roughly ten hours because the unlink is gong to be read IO seek
> bound....

It took 110 minutes and not 10 hours. All files and dirs there had ACLs set.


> Christophs suggestions to use larger inodes to keep the attribute
> data inline is a very good one - whenever you have a workload that
> is attribute heavy you should use larger inodes to try to keep the
> attributes in-line if possible. The down side is that increasing the
> inode size increases the amount of IO required to read/write inodes,
> though this typically isn't a huge penalty compared to the penalty
> of out-of-line attributes.

I will use larger inodes always from now on, since we largely use ACLs
on our XFS partitions.


> Also, for large directories like this (millions of entries) you
> should also consider using a larger directory block size (mkfs -n
> size=xxxx option) as that can be scaled independently to the
> filesystem block size. This will significantly decrease the amount
> of IO and fragmentation large directories cause. Peak modification
> performance of small directories will be reduced because larger
> block size directories consume more CPU to process, but for large
> directories performance will be significantly better as they will
> spend much less time waiting for IO.

This was not ONE directory with that many files, but a directory
containing 834591 subdirectories (deeply nested, not all in the same
dir!) and 10539154 files.

Many thanks,
Richard


-- 
Richard Ems       mail: Richard.Ems@xxxxxxxxxxxxxxxxx

Cape Horn Engineering S.L.
C/ Dr. J.J. Dómine 1, 5º piso
46011 Valencia
Tel : +34 96 3242923 / Fax 924
http://www.cape-horn-eng.com

<Prev in Thread] Current Thread [Next in Thread>