xfsdump __alloc_pages failures and (maybe) kernel panics on debian stable/vanilla 2.4.29

[Robert Buels <rmb32@xxxxxxxxxxx>]

Hi all,

Summary:

On a pretty fast server running debian stable and vanilla linux 2.4.29, 
xfsdump is very slow to write media files and produces "__alloc_pages: 
0-order allocation failed (gfp=0x0/0)" kernel error messages when 
backing up a 1/3 full 915GB partition, and may be causing the machine to 
hard lock.

Any ideas what the problem could be here?


Extended info:

I have a new server, dual 2.4GHz xeon, 2GB RAM, with an LSI SATA 
hardware raid 5 ('megaraid' kernel driver), running debian stable, with 
a 2.4.29 vanilla kernel, connected to an exabyte LTO2 tape changer 
through a SCSI card using the sym53c8xx.  On the big raid 5 (appearing 
to the OS as /dev/sda), there is a large (915GB) XFS partition mounted 
at /data.  I am running backups of this large partition onto the tape 
changer using xfsdump.

During testing of this new setup, I had about 300GB on the partition, 
with a handful of directories and mostly in one large 250GB file that 
was a ext3 dump of one of our old fileservers.  xfsdump worked just 
fine, backing up the 250 gigs overnight with no problem, writing files 
to the tapes at about 16MB/sec, which is near spec for the tape drive.

Following the successful testing of the xfsdump-based backup system, I 
have now started transitioning this machine into production use, filling 
the /data partition with about 384 GB in 2,129,850 files and about 9,000 
directories.

Now, xfsdump seems to be having some significant difficulties backing up 
this partition.  It now seems to take upwards of 3 minutes to assemble a 
single media file of about 300MB, while the tape drive sits idly, then 
writes the media file to the tape drive at what seems to be near-spec 
speed.  This may be normal for xfsdump, I don't know, since this is my 
first experience with it.  HOWEVER, while this is going on, error 
messages from the kernel pop up, 2 or 3 at a time, saying:

__alloc_pages: 0-order allocation failed (gfp=0x0/0)

The gfp number there what it actually says is real, and is always 
0x0/0.  These are the only kernel error messages I have seen.

Worse, the machine hard-locked sometime last night, and I was unable to 
recover any console output and there was nothing in the system or kernel 
logs, it just stopped and hung.  I started a level 0 dump four days ago 
that I expect probably finished last night, and I have a suspicion that 
xfsdump may have caused the hard lock, but I have no evidence to confirm 
this, but it seems to me like a pretty good bet.

What could be the problem here?


More Info:

Operating System: Linux Debian stable (woody)

solanine:/data/shared# uname -a
Linux solanine 2.4.29 #7 SMP Fri Feb 18 17:33:29 EST 2005 i686 unknown

solanine:/data/shared# apt-cache show xfsdump
Package: xfsdump
Priority: optional
Section: admin
Installed-Size: 636
Maintainer: Nathan Scott <nathans@xxxxxxxxxx>
Architecture: i386
Version: 2.0.1-2
Depends: attr (>= 2.0.0), dmapi (>= 2.0.0), libc6 (>= 2.2.4-4), xfsprogs 
(>= 2.0.0)
Filename: pool/main/x/xfsdump/xfsdump_2.0.1-2_i386.deb
Size: 250276
MD5sum: 360961a6f9d54222c6c08385592e8a33

example xfsdump output on a root console:

<snip>
xfsdump: media file size 376438784 bytes
xfsdump: creating dump session media file 1156 (media 1, file 229)
xfsdump: dumping ino map
xfsdump: dumping directories
__alloc_pages: 0-order allocation failed (gfp=0x0/0)
__alloc_pages: 0-order allocation failed (gfp=0x0/0)
__alloc_pages: 0-order allocation failed (gfp=0x0/0)
xfsdump: dumping non-directory files
xfsdump: ending media file
xfsdump: media file size 376438784 bytes
xfsdump: creating dump session media file 1157 (media 1, file 230)
xfsdump: dumping ino map
xfsdump: dumping directories



--
Robert Buels
SGN Bioinformatics Analyst
252A Emerson Hall, Cornell University
Ithaca, NY  14853
Tel: 607-255-2360
rmb32@xxxxxxxxxxx
http://www.sgn.cornell.edu