Re: xfs_check problems on 3.6TB fs

[Frank Hellmann <frank@xxxxxxxxxxxxx>]

Hi Michal!

Michal Szymanski wrote:
> On Tue, Oct 26, 2004 at 12:05:30PM +0200, Frank Hellmann wrote:
>
>
> > Similar effects here. Basically we have the same setup (Hardware RAID5 
> > with Software RAID0 striped together). I ran into some problems during a 
> > xfs_repair. See these articles:
> >
> > http://marc.theaimsgroup.com/?l=linux-xfs&m=109714077512194&w=2
> >
> > http://marc.theaimsgroup.com/?l=linux-xfs&m=109719162327429&w=2
> >
> > Not to good news, I am afraid. We are currently working on a scheme to 
> > cache our files into multiple directories instead of the filesystem 
> > root. That as far as I can tell helps to avoid this issue. So you 
> > shouldn't run into these during xfs_repair.
> >
> > Concering xfs_check... anyone?
>
>
> Well, I am even more confused now. I got "attracted" to XFS by reading a
> comparison between XFS and EXT3 which stated that EXT3 on a huge
> partition is a memory and time hog.

Don't be confused. XFS is a giant leap forward, at least from my view.

Remember the Large Block Device (2+ TB devices) support is working 
"correctly" since Kernel 2.6.6. So there might be a few issues that have 
not showed up until recently.

> Indeed, I have several hardware RAID
> arrays sliced into sub-2TB partitions with EXT3 FS, and it takes a few
> hours to complete a filesystem check after a crash. Well, but it
> completes. Now the XFS, advertized as a true 64-bit FS, capable of
> dealing with Petabytes, seems to be unable to check an almost empty
> partition and/or repair a partition with several hundred thousand files.

We have two production servers running a similar setup. We work with 
film images that are usually 10MB-60MB in size and about 130.000 files 
minimum. Except for that one issue (putting everything into the 
filesystem root) this works really well, has good performance and is 
stable. We have way more issues with the nvidia drivers, than anything else.

It seems to me, that xfs_check is running into some wrap-around bug at 
the 2TB limit and just spits out the "out of memory" error. I don't have 
sufficent knowledge to fix this (IMHO there is a bigger chance of me 
breaking it completly).

> On the other hand, I've read a page "Who's using XFS" with some
> important projects (like Sloan Digital Sky Survey) generating huge
> amounts of data. Do they also have partitions that they cannot check/repair?

I am guessing that they use IRIX and that is known to support larger 
devices since quiet some time. I guess the user-tools will just catch up 
with the new kernel features soon. As I said LBD is known to be working 
since only a _few_ month.

> The oss.sgi.com/projects/xfs pages content suggests that actually 'fcsk'
> is not needed anymore on a journalling FS like XFS. So maybe we can just
> live without it?

No. There will be times, you'll need it. Powerloss is never going to 
give you predictable results.

> To resume, could someone tell me whether I can safely put my data on a
> 3.6TB XFS filesystem or not? The machine it is currently attached to
> has 4GB RAM + 2GB Swap. If this is not enough for XFS to do a
> check/repair, I would say it is not a solution for me.

As I said, in a very unlikely event (about 500.000 files in the fs-root, 
+ Nvidia driver crashing the machine) there is an issue repairing it.

I all other cases we never had an issue with xfs and the xfs_repair. And 
we have to run xfs_repair at least once a week due to the nvidia crashes 
and hard resets.

> regards, Michal.
>
> PS. I've just found, on http://oss.sgi.com/projects/xfs/irix-linux.html
>
>
> > Linux XFS filesystems are limited to 2 Terabytes in size due to
> > limitations in the Linux block device I/O layers.
>
>
> Is it just an out-of-date page? Or, maybe, it is just the true reason of
> our problems?

A bit outdated I would say. But maybe someone else would like to comment 
on the LBD stability and sanity?

                Cheers,
                        Frank...
--
--------------------------------------------------------------------------
Frank Hellmann          Optical Art GmbH           Waterloohain 7a
DI Supervisor           http://www.opticalart.de   22769 Hamburg
frank@xxxxxxxxxxxxx     Tel: ++49 40 5111051       Fax: ++49 40 43169199