bug: xfs_repair becomes very slow when file system has a large sparse file

[Joe Landman]

(If you prefer we file this on a bug reporting system, please let me 
know where and I'll do this).

Scenario:  xfs_repair being run against an about 17TB volume, containing 
1 large sparse file.  Logical size of 7 PB, actual size, a few hundred GB.

Metadata:  Kernel = 2.6.32.41, 2.6.39.4, and others. Xfstools 3.1.5. 
Hardware RAID ~17TB LUN.  Base OS: Centos 5.6 + updates + updated xfs 
tools + our kernels.  Using external journal on a different device

What we observe:

Running xfs_repair

	xfs_repair -l /dev/md2 -vv /dev/sdd2

the system gets to stage 3 and the first ag.  Then it appears to stop. 
After an hour or so, we strace it, and we see

	pread(...) = 4096

occurring about 2-3 per second.  An hour later, its down to 1 per 
second.   An hour after that, its once every 2 seconds.

Also, somewhere on this disk, someone has created an unfortunately large 
file

[root at jr4-2 ~]# ls -alF /data/brick-sdd2/dht/scratch/xyzpdq
total 4652823496
d---------   2 1232 1000               86 Jun 27 20:31 ./
drwx------ 104 1232 1000            65536 Aug 17 23:53 ../
-rw-------   1 1232 1000               21 Jun 27 09:57 Default.Route
-rw-------   1 1232 1000              250 Jun 27 09:57 Gau-00000.inp
-rw-------   1 1232 1000                0 Jun 27 09:57 Gau-00000.d2e
-rw-------   1 1232 1000 7800416534233088 Jun 27 20:18 Gau-00000.rwf

[root at jr4-2 ~]# ls -ahlF /data/brick-sdd2/dht/scratch/xyzpdq
total 4.4T
d---------   2 1232 1000   86 Jun 27 20:31 ./
drwx------ 104 1232 1000  64K Aug 17 23:53 ../
-rw-------   1 1232 1000   21 Jun 27 09:57 Default.Route
-rw-------   1 1232 1000  250 Jun 27 09:57 Gau-00000.inp
-rw-------   1 1232 1000    0 Jun 27 09:57 Gau-00000.d2e
-rw-------   1 1232 1000 7.0P Jun 27 20:18 Gau-00000.rwf

This isn't a 7PB file system, its a 100TB file system across 3 machines, 
roughly 17TB per brick or OSS.  The Gau-00000.rwf is obviously a sparse 
file, as could be seen with an ls -alsF

Upon removing that file, the xfs_repair completes within ~10 minutes. 
Leaving that file on there, the xfs_repair does not terminate, it just 
gets asymptotically slower.

I suspect it is looking for extents which are not there as part of the 
repair.

Please let me know if you need more information, or if you would like me 
to file this somewhere else for official reportage.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615