[Top] [All Lists]

Re: creating a new 80 TB XFS

To: xfs@xxxxxxxxxxx
Subject: Re: creating a new 80 TB XFS
From: Joe Landman <landman@xxxxxxxxxxxxxxxxxxxxxxx>
Date: Sun, 26 Feb 2012 11:55:23 -0500
In-reply-to: <20120226170820.45730357@xxxxxxxxxxxxxx>
Organization: Scalable Informatics
References: <4F478818.4050803@xxxxxxxxxxxxxxxxx> <20120224150805.243e4906@xxxxxxxxxxxxxxxxxxxx> <4F47B020.4000202@xxxxxxxxxxxxxxxxx> <20297.22833.759182.360340@xxxxxxxxxxxxxxxxxx> <4F499F81.7080305@xxxxxxxxxxxxxxxxx> <20120226170820.45730357@xxxxxxxxxxxxxx>
Reply-to: landman@xxxxxxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
On 02/26/2012 11:08 AM, Emmanuel Florac wrote:
Le Sat, 25 Feb 2012 20:57:05 -0600 vous écriviez:

As others mentioned, an xfs_[check|repair] can take many hours or even
days on a multi-terabyte huge metadata filesystem.

Just nitpicking, but I never had such a problem. I've run quite a lot
of xfs_repair on 40TB+ filesystems, and it rarely was longer than 10 to
20 minutes. The important part is having enough RAM if the system hits
swap it makes the check much slower).

We've found that adding the -m X and -P options seem to fix many of the longer running issues for large nearly full many TB file systems. Biggest one we've repaired has been 108TB and its taken a few hours, with ~80% utilization of the underlying file system.

I don't know if the sparse file bit we reported last year (with more data reported to the list in January this year) has had much attention (hard to reproduce I would imagine). But apart from this, repair seems to work reasonably quickly. I've not seen an instance after using the -m X -P options, or "days" for repair, even on heavily fragmented file systems. Possibly Peter has seen this, and he might describe his observations in this regard.

Repair time is important. There's no doubt of that. To some degree, repair performance will be related to the speed of accessing the data on the drives, so if your best case IO speeds are low, performance on repair won't be terribly good. Memory size is also important ... we've had some repairs start swapping (not good) during repair. Hence the -m X option (for suitable values of X).


Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@xxxxxxxxxxxxxxxxxxxxxxx
web  : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

<Prev in Thread] Current Thread [Next in Thread>