xfs
[Top] [All Lists]

Re: [BUG report]xfs_btree_make_block_unfull generated an OOPS

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: [BUG report]xfs_btree_make_block_unfull generated an OOPS
From: hank peng <pengxihan@xxxxxxxxx>
Date: Wed, 13 Jan 2010 09:11:02 +0800
Cc: Eric Sandeen <sandeen@xxxxxxxxxxx>, xfs-oss <xfs@xxxxxxxxxxx>
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=rm2qV/HKYnj8+jCPC/VMYFBsHSMyyOt2sZj2GE8dsLc=; b=qGYhHTDreSCC+IrXt5rQ18Z7/aLt+pi41RpBfMV4WXbFtvOv2L+f/HEhhG6xIjQN/c ICVRFnPN0yDKohpIPwYxd6zrI+WBzSZkDRlQ5+mlDrUCOAzes4NHdFBEh52hw33A79KZ 6ITHAdgewC1LrqcIvgLZ0HcdCgMcC9+VOTV3o=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=LezC/oIt3CDgjJ+xfiRaV/Qb+fNBYFQggAgKNfTgCz43rmXoeHSZFD5sJBw2cX1cXC 89JO1himX8gSGTGZi27Em0SdV8pWpZ3NI9Pa9CXNrXxvofT6IqWiafMwLIW+HoNc1g37 FKXhUprGQOpvl3+WNzc14XVRNrWIKznxHu9e8=
In-reply-to: <20091215012640.GA4850@xxxxxxxxxxxxxxxx>
References: <4B1F1211.90607@xxxxxxxxxxx> <4B1F18C4.3060704@xxxxxxxxxxx> <389deec70912082053v4310057dg479f6d4b6c4b46f7@xxxxxxxxxxxxxx> <4B1F31FD.3020705@xxxxxxxxxxx> <389deec70912082220pcb3b5d1q516ac197d31502c5@xxxxxxxxxxxxxx> <389deec70912082230g38987576pc48d7699f23844c5@xxxxxxxxxxxxxx> <389deec70912140119q40ed91cao62fe9c9ebdf13601@xxxxxxxxxxxxxx> <4B26604B.3060901@xxxxxxxxxxx> <389deec70912141649g767a1540hdeae66707c4c68fd@xxxxxxxxxxxxxx> <20091215012640.GA4850@xxxxxxxxxxxxxxxx>
2009/12/15 Dave Chinner <david@xxxxxxxxxxxxx>:
> On Tue, Dec 15, 2009 at 08:49:37AM +0800, hank peng wrote:
>> Hi, Eric:
>> I add some code like this:
>> if (*stat) {
>>                 printk("*stat = 0x%08x, oindex = %p, index = %p\n",
>>                                 *stat, oindex, index);
>>                 if (oindex == NULL || index == NULL) {
>
> This won't catch bad non-NULL pointers like you are seeing.
>
>>                         printk("BUG occured!\n");
>>                         printk("oindex = %p, index = %p\n", oindex, index);
>>                         BUG();
>>                 }
>>                 *oindex = *index = cur->bc_ptrs[level];
>>                 return 0;
>>         }
>>
>> And the same OOPS happened again but a little different, kernel messages are:
>>
>> <snip>
>> *stat = 0x00000001, oindex = e87d7bf8, index = e87d7bfc
>> *stat = 0x00000001, oindex = e87d7bf8, index = e87d7bfc
>> *stat = 0x00000001, oindex = e87d7bf8, index = e87d7bfc
>> *stat = 0x00000001, oindex = e87d7bf8, index = e87d7bfc
>> *stat = 0x00000001, oindex = 00000501, index = 22008424
>> Unable to handle kernel paging request for data at address 0x22008424
>
> Given that oindex and index are stack varibles, this indicates some
> thing is probably smashing the stack. Possibly a buffer overrun. To
> narrow down the possible cause, can you add the debug:
>
>        printk("%s:%s: oindex = %p, index = %p\n",
>                        __func__, __LINE__, oindex, index);
>
> throughout the xfs_btree_make_block_unfull() function? i.e. at
> first entry, before the xfs_btree_rshift() call, before the
> xfs_btree_lshift() call, etc, to see if any of the parameters
> are being modified during execution of the function?
>
> If the variables being passed into xfs_btree_make_block_unfull() are
> already bad, then do the same thing for the caller
> xfs_btree_insert(). This may help narrow down where the problem
> is coming from....
>
I added the following debug code as you said:
<code>
printk("%s: before xfs_btree_rshift, oindex = %p, index = %p\n",
                        __func__, oindex, index);
        error = xfs_btree_rshift(cur, level, stat);
        if (error || *stat)
                return error;

        /* Next, try shifting an entry to the left neighbor. */
        printk("%s: before xfs_btree_lshift, oindex = %p, index = %p\n",
                        __func__, oindex, index);
        error = xfs_btree_lshift(cur, level, stat);
        if (error)
                return error;

        if (*stat) {
                printk("%s: oindex = %p, index = %p, *stat = %d\n",
__func__, oindex, index, *stat);
                *oindex = *index = cur->bc_ptrs[level];
                return 0;
        }
</code>

It has been working fine for about 36 hours without problem, but in
today's morning, odd OOPS appeared:

xfs_btree_make_block_unfull: before xfs_btree_rshift, oindex =
d3a27bd8, index = d3a27bdc
xfs_btree_make_block_unfull: before xfs_btree_lshift, oindex =
d3a27bd8, index = d3a27bdc
xfs_btree_make_block_unfull: oindex = d3a27bd8, index = d3a27bdc, *stat = 1
xfs_btree_make_block_unfull: before xfs_btree_rshift, oindex =
d3a27bd8, index = d3a27bdc
Unable to handle kernel paging request for data at address 0x00000501
Faulting instruction address: 0xc019f4f0
Oops: Kernel access of bad area, sig: 11 [#2]
MPC85xx CDS
Modules linked in:
NIP: c019f4f0 LR: c019f4e8 CTR: c023fabc
REGS: d3a27ad0 TRAP: 0300   Tainted: G      D     (2.6.31.6-svn45)
MSR: 00029000 <EE,ME,CE>  CR: 22008424  XER: 20000000
DEAR: 00000501, ESR: 00000000
TASK = efb46a30[20273] 'cp' THREAD: d3a26000
GPR00: c019f4e8 d3a27b80 efb46a30 00000000 d3a27b38 d3a27b38 00000010 007f0f26
GPR08: c04a7c40 ffffffff e8517850 d3a27b80 20008422 100eb39c 3fff5400 100a0000
GPR16: 100d5ac8 00000000 016d30f3 e8517850 c019d08c 00029000 d3a27bf0 c023fabc
GPR24: c019d068 00000000 22008424 d3a27bdc 00000501 d3a27bd8 00000000 e8517850
NIP [c019f4f0] xfs_btree_make_block_unfull+0x8c/0x1f8
LR [c019f4e8] xfs_btree_make_block_unfull+0x84/0x1f8
Call Trace:
[d3a27b80] [c019f4e8] xfs_btree_make_block_unfull+0x84/0x1f8 (unreliable)
[d3a27bc0] [c019f9d0] xfs_btree_insrec+0x374/0x4b0
[d3a27c30] [c019fb88] xfs_btree_insert+0x7c/0x1c0
[d3a27c90] [c01865d0] xfs_free_ag_extent+0x34c/0x810
[d3a27d00] [c0187168] xfs_free_extent+0xdc/0x104
[d3a27d90] [c018fe50] xfs_bmap_finish+0x154/0x1a0
[d3a27dc0] [c01b697c] xfs_itruncate_finish+0x254/0x3b8
[d3a27e40] [c01d2134] xfs_inactive+0x2c4/0x450
[d3a27e80] [c01e193c] xfs_fs_clear_inode+0x40/0x50
[d3a27e90] [c00a84bc] clear_inode+0x6c/0x108
[d3a27ea0] [c00a87d0] generic_delete_inode+0x114/0x118
[d3a27eb0] [c00a7ff8] iput+0x74/0x94
[d3a27ec0] [c00a003c] do_unlinkat+0x114/0x198
[d3a27f40] [c000f7ac] ret_from_syscall+0x0/0x3c
Instruction dump:
7f66db78 7f44d378 7fa5eb78 3863eca4 4cc63182 4be97ef5 7fe3fb78 7fc4f378
7f85e378 4bffdb15 7c791b79 40820010 <801c0000> 2f800000 419e001c 80010044
---[ end trace 95e2c49eb5a34f9a ]---

(gdb) list *(xfs_btree_make_block_unfull+0x8c)
0xc019f4f0 is in xfs_btree_make_block_unfull (fs/xfs/xfs_btree.c:2636).
2631
2632            /* First, try shifting an entry to the right neighbor. */
2633            printk("%s: before xfs_btree_rshift, oindex = %p, index = %p\n",
2634                            __func__, oindex, index);
2635            error = xfs_btree_rshift(cur, level, stat);
2636            if (error || *stat)
2637                    return error;
2638
2639            /* Next, try shifting an entry to the left neighbor. */
2640            printk("%s: before xfs_btree_lshift, oindex = %p, index = %p\n",

It seems that after call xfs_btree_rshift, the value of 'stat' has
been changed, how could it be possible since it is local variable?

# uname -a
Linux Storage 2.6.31.6-svn45 #87 Mon Jan 11 13:22:14 CST 2010 ppc unknown
# mount
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw,relatime,errors=continue)
/dev/mtdblock2 on /mnt/sys_data type jffs2 (rw,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /opt/upgrade type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
/dev/vg_log/lv_log on /var/log type reiserfs (rw,relatime)
/dev/Pool_md1/SS1 on /mnt/Pool_md1/SS1 type xfs
(rw,relatime,attr2,inode64,noquota)
/dev/Pool_md2/SS2 on /mnt/Pool_md2/SS2 type xfs
(rw,relatime,attr2,inode64,noquota)
root@Storage:/var/log# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               124.0M     72.6M     51.4M  59% /
/dev/mtdblock2            1.0M    408.0K    616.0K  40% /mnt/sys_data
tmpfs                   505.3M         0    505.3M   0% /opt/upgrade
/dev/vg_log/lv_log       10.0G     32.4M     10.0G   0% /var/log
/dev/Pool_md1/SS1         2.7T    270.2G      2.5T  10% /mnt/Pool_md1/SS1
/dev/Pool_md2/SS2         2.7T    344.0G      2.4T  12% /mnt/Pool_md2/SS2

>From assembly code, I noticed that the local variable 'stat' didn't
have real space in stack. It is optimised to be a register(r28).
According to powerpc ABI, before call xfs_btree_rshift, some registers
will be saved at stack and before return from xfs_btree_rshift, these
registers will be restored. Is it possible that a smash occured at
this time?

BTW, I noticed that my cross-compiler "powerpc-linux-gnuspe-gcc"
didn't have default 4 bytes alignment but 8 bytes alignment.



> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@xxxxxxxxxxxxx
>



-- 
The simplest is not all best but the best is surely the simplest!

<Prev in Thread] Current Thread [Next in Thread>
  • Re: [BUG report]xfs_btree_make_block_unfull generated an OOPS, hank peng <=