Date: Wed, 10 Aug 2005 18:58:49 +1000
From: David Chinner <dgc@sgi.com>
To: Steve Wray <stevew-lists@catalyst.net.nz>
Cc: linux-xfs@oss.sgi.com
Subject: Re: XFS repair problem
Message-ID: <20050810185849.A27595508@melbourne.sgi.com>
References: <001001c59bfb$1fe7aae0$0400a8c0@LocalHost> <42F74B6D.8060002@gmx.net> <00d801c59c20$e0354080$0400a8c0@LocalHost> <42F798AF.5080505@gmx.net> <20050809074858.B25981667@melbourne.sgi.com> <005301c59c7d$2eedab20$0400a8c0@LocalHost> <20050809120823.E13484145@melbourne.sgi.com> <42F90DC5.9070702@catalyst.net.nz>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5.1i
In-Reply-To: <42F90DC5.9070702@catalyst.net.nz>; from stevew-lists@catalyst.net.nz on Wed, Aug 10, 2005 at 08:10:45AM +1200
Sender: linux-xfs-bounce@oss.sgi.com
Errors-to: linux-xfs-bounce@oss.sgi.com
Precedence: bulk
Status: O
Content-Length: 3222
Lines: 83

On Wed, Aug 10, 2005 at 08:10:45AM +1200, Steve Wray wrote:
> David Chinner wrote:
> > On Tue, Aug 09, 2005 at 02:56:28AM +0200, djani22@dynamicweb.hu wrote:
> > 
> >>More info:
> >>
> >>I try xfs_check and xfs_ncheck (and more progs) with +200GB swap, but no
> >>different!
> >>less than 1 second and get : out of memory.
> > 
> [snip]
> > Your filesystem (8TiB) may simply bee too large for your system to
> > be able to repair. Try mounting it on a 64bit system with more RAM
> > in it and repairing it from there.
> 
> Sorry, but is this a joke?

A joke? Absolutely not.

Acheivable XFS filesystem sizes outgrew the capability of 32 bit
Irix systems to repair them several years ago. Now that linux
supports larger than 2TiB filesystems on 32 bit systems, this is
true for Linux as well.

FWIW, look at the irix man pages for xfs_check and xfs_repair:

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_check.z&srch=xfs_check

http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/xfs_repair.z&srch=xfs_repair

A quote so you don't have to follow the links:

     xfs_repair is an n32 binary and will run on all Irix platforms.  However,
     when repairing a multi-terabyte filesystem, the memory requirements
     exceed what is available to n32 binaries.  For those filesystems,
     xfs_repair64, 64-bit binary, should be used.

This has also been mentioned before on this list in similar circumstances:

http://marc.theaimsgroup.com/?l=linux-xfs&m=109890988924676&w=2

As for moving disks to other machines to repair them - that's not
unusual, either. e.g:

http://marc.theaimsgroup.com/?l=linux-xfs&m=107435532601553&w=2

> Surely xfs could/should have a repair mode that actually works on the
> hardware that the filesystem is installed on?

Surely it could. When can we expect a patch? ;)

> Alternatively, so that others can avoid the situation of having to go
> and get their hands on a 64 bit machine to repair their xfs filesystems,
> is there a cutoff point heuristic? Ie: how big does an xfs filesystem
> have to be for it to require a 64 bit architecture to fix?

From some quick tests I just ran, for 32bit binaries xfs_check needs
around 1GiB RAM per TiB of filesystem plus about 100MiB RAM per
1million inodes in the filesystem (more if you have lots of
fragmented files). Double this for 64bit binaries.  e.g. it took
1.5GiB RAM for 32bit xfs_check and 2.7GiB RAM for a 64bit
xfs_check on a 1.1TiB filesystem with 3million inodes in it.

For xfs_repair, there is no one-size fits all formula as memory
consumption depends not only on the size of the filesystem but what
is in the filesystem, how it is laid out, what is corrupted in the
filesystem, etc. For example, the filesystem I checked above only
required ~150MiB for repair to run but that is a consistent
filesystem. I've seen equivalently size filesystems (~1TiB)
take close to 1GiB of RAM to repair when they've been significantly
corrupted.

Sorry I can't be more precise than this, but it should give you
some idea of what to expect....

Cheers,

Dave.
-- 
Dave Chinner
R&D Software Enginner
SGI Australian Software Group