xfs
[Top] [All Lists]

XFS (sdd1): Internal error xfs_da_do_buf(2) at line 2097 of file /usr/sr

To: xfs@xxxxxxxxxxx
Subject: XFS (sdd1): Internal error xfs_da_do_buf(2) at line 2097 of file /usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.
From: Richard Ems <richard.ems@xxxxxxxxxxxxxxxxx>
Date: Mon, 17 Sep 2012 16:56:19 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20120825 Thunderbird/15.0
Hi all,

saturday morning one hard disc on our RAID6 failed. About one hour later,
 the XFS running on that device reported the following error:

XFS (sdd1): Internal error xfs_da_do_buf(2) at line 2097 of file 
/usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.

We run openSUSE 12.1 64 bit, kernel 3.3.6, on a DELL R410 server, with
 4 MD1200 storage systems attached.
The failed disc was on one of these MD1200, configured with
 11 x 3 TB discs on a RAID 6 and one hot standby disc. 
A rebuild started on the standby disc immediately and finished
 several hours later.

But, and as mentioned above, about one hour after the disc failure
 the above error was reported and the mountpoint was set to read-only mode.

This error got reported 228664 times in 23 hours:

fs1:/var/log # grep -c -i "XFS .* Internal error xfs_da_do_buf(2)" 
messages-20120916
228664

fs1:/var/log # grep -i "XFS .* Internal error xfs_da_do_buf(2)" 
messages-20120916 | head -n 1
Sep 15 07:30:51 fs1 kernel: [7369085.790742] XFS (sdd1): Internal error 
xfs_da_do_buf(2) at line 2097 of file 
/usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.  
Caller 0xffffffffa038a09a

fs1:/var/log # grep -i "XFS .* Internal error xfs_da_do_buf(2)" 
messages-20120916 | tail -n 1
Sep 16 06:30:00 fs1 kernel: [7451668.512765] XFS (sdd1): Internal error 
xfs_da_do_buf(2) at line 2097 of file 
/usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.  
Caller 0xffffffffa038a09a


The complete stack trace reported is:

Sep 15 07:30:51 fs1 kernel: [7369085.790735] ffff8801fd805000: 6f 86 00 0f 9d 
38 00 15 26 1c 00 aa e3 17 00 a7  o....8..&.......
Sep 15 07:30:51 fs1 kernel: [7369085.790742] XFS (sdd1): Internal error 
xfs_da_do_buf(2) at line 2097 of file 
/usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.  
Caller 0xffffffffa038a09a
Sep 15 07:30:51 fs1 kernel: [7369085.790745] 
Sep 15 07:30:51 fs1 kernel: [7369085.790749] Pid: 16081, comm: rsync Not 
tainted 3.3.6-1-default #1
Sep 15 07:30:51 fs1 kernel: [7369085.790752] Call Trace:
Sep 15 07:30:51 fs1 kernel: [7369085.790766]  [<ffffffff81004618>] 
dump_trace+0x78/0x320
Sep 15 07:30:51 fs1 kernel: [7369085.790775]  [<ffffffff81579939>] 
dump_stack+0x69/0x6f
Sep 15 07:30:51 fs1 kernel: [7369085.790804]  [<ffffffffa0357202>] 
xfs_corruption_error+0x62/0x90 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.790858]  [<ffffffffa0389fba>] 
xfs_da_do_buf+0x5fa/0x680 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.790961]  [<ffffffffa038a09a>] 
xfs_da_read_buf+0x1a/0x20 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791061]  [<ffffffffa03903a0>] 
xfs_dir2_leaf_lookup_int+0x60/0x300 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791169]  [<ffffffffa0390b05>] 
xfs_dir2_leaf_lookup+0x25/0x100 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791275]  [<ffffffffa038c937>] 
xfs_dir_lookup+0x187/0x1a0 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791376]  [<ffffffffa03693d0>] 
xfs_lookup+0xb0/0x110 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791442]  [<ffffffffa035f492>] 
xfs_vn_lookup+0x42/0x90 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791496]  [<ffffffff81169b5c>] 
d_alloc_and_lookup+0x3c/0x90
Sep 15 07:30:51 fs1 kernel: [7369085.791505]  [<ffffffff8116beb3>] 
do_lookup+0x2a3/0x3a0
Sep 15 07:30:51 fs1 kernel: [7369085.791514]  [<ffffffff8116e1ff>] 
do_last+0xef/0x7d0
Sep 15 07:30:51 fs1 kernel: [7369085.791524]  [<ffffffff8116ea0c>] 
path_openat+0xdc/0x400
Sep 15 07:30:51 fs1 kernel: [7369085.791532]  [<ffffffff8116ee5d>] 
do_filp_open+0x4d/0xc0
Sep 15 07:30:51 fs1 kernel: [7369085.791542]  [<ffffffff8115e952>] 
do_sys_open+0x102/0x1e0
Sep 15 07:30:51 fs1 kernel: [7369085.791552]  [<ffffffff815848b9>] 
system_call_fastpath+0x16/0x1b
Sep 15 07:30:51 fs1 kernel: [7369085.791569]  [<00007f2e079aadf0>] 
0x7f2e079aadef
Sep 15 07:30:51 fs1 kernel: [7369085.791574] XFS (sdd1): Corruption detected. 
Unmount and run xfs_repair
Sep 15 07:30:51 fs1 kernel: [7369085.791836] ffff8801fd805000: 6f 86 00 0f 9d 
38 00 15 26 1c 00 aa e3 17 00 a7  o....8..&.......

Sep 15 07:30:51 fs1 kernel: [7369085.791842] XFS (sdd1): Internal error 
xfs_da_do_buf(2) at line 2097 of file 
/usr/src/packages/BUILD/kernel-default-3.3.6/linux-3.3/fs/xfs/xfs_da_btree.c.  
Caller 0xffffffffa038a09a
Sep 15 07:30:51 fs1 kernel: [7369085.791849] 
Sep 15 07:30:51 fs1 kernel: [7369085.791851] Pid: 16081, comm: rsync Not 
tainted 3.3.6-1-default #1
Sep 15 07:30:51 fs1 kernel: [7369085.791854] Call Trace:
Sep 15 07:30:51 fs1 kernel: [7369085.791864]  [<ffffffff81004618>] 
dump_trace+0x78/0x320
Sep 15 07:30:51 fs1 kernel: [7369085.791869]  [<ffffffff81579939>] 
dump_stack+0x69/0x6f
Sep 15 07:30:51 fs1 kernel: [7369085.791887]  [<ffffffffa0357202>] 
xfs_corruption_error+0x62/0x90 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.791937]  [<ffffffffa0389fba>] 
xfs_da_do_buf+0x5fa/0x680 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792039]  [<ffffffffa038a09a>] 
xfs_da_read_buf+0x1a/0x20 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792140]  [<ffffffffa03903a0>] 
xfs_dir2_leaf_lookup_int+0x60/0x300 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792247]  [<ffffffffa0390b05>] 
xfs_dir2_leaf_lookup+0x25/0x100 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792354]  [<ffffffffa038c937>] 
xfs_dir_lookup+0x187/0x1a0 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792455]  [<ffffffffa03693d0>] 
xfs_lookup+0xb0/0x110 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792522]  [<ffffffffa035f492>] 
xfs_vn_lookup+0x42/0x90 [xfs]
Sep 15 07:30:51 fs1 kernel: [7369085.792562]  [<ffffffff81169b5c>] 
d_alloc_and_lookup+0x3c/0x90
Sep 15 07:30:51 fs1 kernel: [7369085.792569]  [<ffffffff8116beb3>] 
do_lookup+0x2a3/0x3a0
Sep 15 07:30:51 fs1 kernel: [7369085.792577]  [<ffffffff8116cc15>] 
path_lookupat+0x135/0x730
Sep 15 07:30:51 fs1 kernel: [7369085.792585]  [<ffffffff8116d23c>] 
do_path_lookup+0x2c/0xd0
Sep 15 07:30:51 fs1 kernel: [7369085.792592]  [<ffffffff8116edac>] 
user_path_at_empty+0x5c/0xb0
Sep 15 07:30:51 fs1 kernel: [7369085.792597]  [<ffffffff81163efe>] 
vfs_fstatat+0x3e/0x90
Sep 15 07:30:51 fs1 kernel: [7369085.792603]  [<ffffffff8116414f>] 
sys_newlstat+0x1f/0x50
Sep 15 07:30:51 fs1 kernel: [7369085.792609]  [<ffffffff815848b9>] 
system_call_fastpath+0x16/0x1b
Sep 15 07:30:51 fs1 kernel: [7369085.792616]  [<00007f2e079aa4c5>] 
0x7f2e079aa4c4
Sep 15 07:30:51 fs1 kernel: [7369085.792619] XFS (sdd1): Corruption detected. 
Unmount and run xfs_repair


And this repeating again and again ...

This system has been running fine for 87 days, no power outages or such.
It's connected to an UPS, and the H800 Raid Controller has a BBU installed.

The XFS is mounted with these options:
/dev/sdd1 on /data_1 type xfs 
(rw,noatime,attr2,nobarrier,inode64,logbufs=8,logbsize=256k,sunit=128,swidth=1152,noquota,_netdev)


After running xfs_repair on it, the XFS could be mounted again, 
BUT several dirs and plain files were put into lost+found:

fs1:/data_1/lost+found # find . -type d | wc -l
115

fs1:/data_1/lost+found # find . -type f | wc -l
479

I have the full xfs_repair output. It's 56 MB uncompressed, and 1.1 MB gzipped.
Should I attach/send it here?
Why could this have happened?
What more info can I provide to understand this issue and avoid this to happen 
again?

Many thanks,
Richard





-- 
Richard Ems       mail: Richard.Ems@xxxxxxxxxxxxxxxxx

Cape Horn Engineering S.L.
C/ Dr. J.J. Dómine 1, 5º piso
46011 Valencia
Tel : +34 96 3242923 / Fax 924
http://www.cape-horn-eng.com

<Prev in Thread] Current Thread [Next in Thread>