xfs
[Top] [All Lists]

Re: Issues and new to the group

To: xfs@xxxxxxxxxxx
Subject: Re: Issues and new to the group
From: Joe Landman <joe.landman@xxxxxxxxx>
Date: Thu, 26 Sep 2013 10:59:41 -0400
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=U7n78Kqh/Rnfi194enfnyyWOkfGS6Lq/RBnQvZcYk3A=; b=qWWsJ9vLbmQMeSTnZg3fwQ/Dof0zs+YC9T6EXD/FguMRHFtXQd7AuzWWlqEXsi2wL/ iSa6ItAa3qu57mfc4hD1elFiv7oEZeE6wN3SUnH9iwGSQADeSqbZBT12xPwxGtiE/ZqX Otj6MjYOONnTQuzMaFdLw0wD89OYqxjiVTpzdMPwgpmBR9ktD8Q2mEeMJokxw5yESAAZ ws4WHmlBAFrYbqQ7PtSh+7q1Th+GlgDoovUVQbx/4n1nu4NGCarigEdSZolwxOaBM9Jr HmFmJX3gdenF/XJJW6FHkCWl4Asi9q10X8T9ra29x3a696yJUuU/RceMkZDiDI6F0bNY vl6w==
In-reply-to: <100f01cebaba$0ae84280$20b8c780$@host2max.com>
References: <0e4201cebaae$24873680$6d95a380$@host2max.com> <5244234D.1010603@xxxxxxxxxxxxxxxxx> <100f01cebaba$0ae84280$20b8c780$@host2max.com>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0
On 09/26/2013 09:12 AM, Ronnie Tartar wrote:
Stan,

Thanks for the reply.

My fragmentation is:

[root@AP-FS1 ~]# xfs_db -c frag -r /dev/xvdb1
actual 10470159, ideal 10409782, fragmentation factor 0.58%

This was never likely the cause ...

[...]

This is running virtualized, definitely not a rust bucket.  It's x5570 cpus

... well, this is likely the cause (virtualized)

with MD3200 Array with light I/O.

Seems like i/o wait is not problem, system% is problem.  Is this the OS
trying to find spot for these files?

From your previous description

takes.  The folders are image folders that have anywhere between 5 to
10 million images in each folder.

The combination of very large folders, and virtualization is working against you. Couple that with an old (ancient by Linux standards) xfs in the virtual CentOS 5.9 system, and you aren't going to have much joy with this without changing a few things.

First and foremost:

Can you change from one single large folder to a heirarchical set of folders? The single large folder means any metadata operation (ls, stat, open, close) has a huge set of lists to traverse. It will work, albiet slowly. As a rule of thumb, we try to make sure our users don't go much beyond 10k files/folder. If they need to, building a heirarchy of folders slightly increases management complexity, but keeps the lists that are needed to be traversed much smaller.

A strategy for doing this: If your files are named "aaaa0001" "aaaa0002" ... "zzzz9999" or similar, then you can chop off the first letter, and make a directory of it, and then put all files starting with that letter in that directory. Then within each of those directories, do the same thing with the second letter. This gets you 676 directories and about 15k files per directory. Much faster directory operations. Much smaller lists to traverse.

If you can't change the layout, this is a much harder problem to solve, though you could do it by using one very large file, maintaining your own start/end and file metadata in other files, and writing your files at specific offsets (start and end above). This isn't a good solution unless you know how to write file systems.

<Prev in Thread] Current Thread [Next in Thread>