[Top] [All Lists]

Re: 1B files, slow file creation, only AG0 used

To: Eric Sandeen <sandeen@xxxxxxxxxxx>
Subject: Re: 1B files, slow file creation, only AG0 used
From: Michael Spiegle <mike@xxxxxxxxxxxxxxxx>
Date: Fri, 9 Mar 2012 21:25:06 -0800
Cc: xfs@xxxxxxxxxxx
In-reply-to: <4F5ADFCB.9010602@xxxxxxxxxxx>
References: <CAEm1Pvny7Q2rrsCLURvo5kQM3vt+yMg17WxoSYGKVWm7Lgp8MA@xxxxxxxxxxxxxx> <4F5ADFCB.9010602@xxxxxxxxxxx>
Reply-to: mike@xxxxxxxxxxxxxxxx
On Fri, Mar 9, 2012 at 8:59 PM, Eric Sandeen <sandeen@xxxxxxxxxxx> wrote:
> on centos please be sure you're not using the old xfs kmod package, just FYI.
> The module shipped with the kernel is what you should use.

We're definitely using the XFS module that came with the kernel.  In
any case, we're seeing problems with the 3.2.9 kernel as well, but
maybe this is due to running mkfs.xfs from CentOS 5.6

> If anything this is testament to xfs scalability, if it can make the 
> billion-and-first inode in a single dir in "only" 300ms ;)
> You might want to read up on the available docs, i.e.
> http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/xfs-allocators.html
> probably covers a lot of what you are wondering about.
> When you make a new dir, in general xfs will put inodes & files in that dir 
> into a new
> AG.
> But another thing you are likely running into is the inode32 allocation 
> behavior, also explained in the doc above.
> In that case inodes are kept in the lower ags, and data is sprinkled around 
> the higher AGs.

Our "1B" files are spread out evenly into a tree of 65536 directories.
 I've read the docs, and they seem explicit about new directories
being created in new AGs, however we are not seeing that on our
system.  All 1B files (despite being spread out across more than 64K
dirs) are in the first AG.  I have tried remounting the filesystem
with inode64 (and on 3.2.9), but this behavior does not seem to change
even if I add more files afterwards.

> If you mount with -o inode64, inodes may be allocated anywhere on the fs, in 
> any AG. New subdirs go to new AGs, activity will be distributed across the 
> filesystem. As long as your applications can properly handle 64-bit inode 
> numbers, this is probably the way to go.
> You would be better off not creating all billion in a single dir, as well.

As mentioned above, the inode64 mount option doesn't seem to affect
anything.  Can you think of anything else I should check that would
prevent this from working?

> I wonder why you have attr=0 and 32 ags; pretty old xfsprogs maybe.

Yep, the filesystem was created with the xfsprogs that came with
CentOS 5.6.  Unfortunately, it took us a long time to copy all of the
files to this filesystem, so we're looking for a way to fix this
without having to reformat.

<Prev in Thread] Current Thread [Next in Thread>