[Top] [All Lists]

Re: 1B files, slow file creation, only AG0 used

To: Michael Spiegle <mike@xxxxxxxxxxxxxxxx>
Subject: Re: 1B files, slow file creation, only AG0 used
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Mon, 12 Mar 2012 11:56:32 +1100
Cc: xfs@xxxxxxxxxxx
In-reply-to: <CAEm1Pvny7Q2rrsCLURvo5kQM3vt+yMg17WxoSYGKVWm7Lgp8MA@xxxxxxxxxxxxxx>
References: <CAEm1Pvny7Q2rrsCLURvo5kQM3vt+yMg17WxoSYGKVWm7Lgp8MA@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Fri, Mar 09, 2012 at 06:13:27PM -0800, Michael Spiegle wrote:
> We're seeing some very strange behavior with XFS on the default kernel
> for CentOS 5.6 (note, I have also 3.2.9 and witnessed the same issue).
>  The dataset on this server is about 1B small files (anywhere from 1KB
> to 50KB).  We first noticed it when creating files in a directory.  A
> simple 'touch' would take over 300ms on a completely idle system.  If
> I simply create a different directory, touching files is 1ms or
> faster.  Example:
> # time touch 0
> real    0m0.323s
> user    0m0.000s
> sys     0m0.323s
> # mkdir tmp2
> # time touch tmp2/0
> real    0m0.001s
> user    0m0.000s
> sys     0m0.000s

Entirely normal. some operations require Io to complete (e.g.
reading directory blocks to find where to insert the new entry),
while adding the first file to a directory generally requires zero
IO. You're seeing the difference between cold cache and hot cache

> We've done quite a bit of testing and debugging, and while we don't
> have an answer yet, we've noticed that our filesystem was created with
> the default of 32 AGs.  When using xfs_db, we notice that all
> allocations appear to be in AG0 only.

Go look up what the inode32 and inode64 mount options do. The
default is inode32....


Dave Chinner

<Prev in Thread] Current Thread [Next in Thread>