RE: Re-occurance of NFS server panics

[Steve Lord <lord@xxxxxxx>]

On Wed, 2002-09-18 at 13:31, Ian D. Hardy wrote:
> Steve,
> 
> >On Mon, 2002-09-16 at 11:56, Ian D. Hardy wrote:
> >> Steve,
> >> 
> >> Thanks for the quick response. I don't always get a Oops output 
> >> (sometimes the server just hangs and requires a reboot). However as
> it 
> >> happens the server has just crashed again with the following Oops 
> >> (through 'ksymoops'):
> >
> >This one suggests heap corruption more than anything else. 
> >
> >Steve
> 
> I upgraded the kernel to the current CVS (2.4.19-xfs) tree (as of Monday
> 16th Sept.) today and got a very similar looking Ooops to the one I
> reported on Monday, see below. I guess it is very difficult to know what
> would have caused any heap corruption. As I understand it there's
> nothing in these latest panics to directly link them with XFS? I need to
> do some more thinking on this. Any pointers would be very welcome.
> 
> Regards Ian Hardy
> 
> 
>  kernel BUG at slab.c:1439!


OK, so you have slab debugging turned on, which I was going to ask
you to do. Looks like someone walked off the end of an allocation
here, that is progress.

So, the question is what did it, thats the really hard part! Is
this machine just running XFS via NFS, or is it doing anything
else? Also, which options do you have turned on in XFS, in fact,
sending the whole kernel config might be an idea.

Steve


> invalid operand: 0000
> CPU:    0
> EIP:    0010:[<c0133a64>]    Tainted: P 
> EFLAGS: 00010016
> eax: 5a2cf071   ebx: 006f5b20   ecx: f7edf00c   edx: c1c0f060
> esi: e51e6000   edi: e51e6c3c   ebp: e51e6c18   esp: f7eddf48
> ds: 0018   es: 0018   ss: 0018
> Process kswapd (pid: 5, stackpage=f7edd000)
> Stack: 0000293b 00000002 f7edf00c f7edf000 00000000 00000007 00000000
> 00000000 
>        00000000 c1c0f060 00000020 000001d0 00000006 00000000 c0134e59
> c036fa08 
>        00000006 000001d0 c036fa08 00000000 c0134f0c 00000020 c036fa08
> 00000002 
> Call Trace:    [<c0134e59>] [<c0134f0c>] [<c0134fb1>] [<c0135026>]
> [<c013515d>]
>   [<c0105000>] [<c0107296>] [<c01350c0>]
> 
> Code: 0f 0b 9f 05 a0 54 2b c0 8b 44 24 24 89 ea 8b 48 18 b8 71 f0 
>  
> Entering kdb (current=0xf7edc000, pid 5) on processor 0 Oops: invalid
> operand
> due to oops @ 0xc0133a64
> eax = 0x5a2cf071 ebx = 0x006f5b20 ecx = 0xf7edf00c edx = 0xc1c0f060 
> esi = 0xe51e6000 edi = 0xe51e6c3c esp = 0xf7eddf48 eip = 0xc0133a64 
> ebp = 0xe51e6c18 xss = 0x00000018 xcs = 0x00000010 eflags = 0x00010016 
> xds = 0xe51e0018 xes = 0x00000018 origeax = 0xffffffff &regs =
> 0xf7eddf14
> [0]kdb> 
>  kernel BUG at slab.c:1439!
> invalid operand: 0000
> CPU:    0
> EIP:    0010:[<c0133a64>]    Tainted: P 
> Using defaults from ksymoops -t elf32-i386 -a i386
> EFLAGS: 00010016
> eax: 5a2cf071   ebx: 006f5b20   ecx: f7edf00c   edx: c1c0f060
> esi: e51e6000   edi: e51e6c3c   ebp: e51e6c18   esp: f7eddf48
> ds: 0018   es: 0018   ss: 0018
> Process kswapd (pid: 5, stackpage=f7edd000)
> Stack: 0000293b 00000002 f7edf00c f7edf000 00000000 00000007 00000000
> 00000000 
>        00000000 c1c0f060 00000020 000001d0 00000006 00000000 c0134e59
> c036fa08 
>        00000006 000001d0 c036fa08 00000000 c0134f0c 00000020 c036fa08
> 00000002 
> Call Trace:    [<c0134e59>] [<c0134f0c>] [<c0134fb1>] [<c0135026>]
> [<c013515d>]
>   [<c0105000>] [<c0107296>] [<c01350c0>]
> Code: 0f 0b 9f 05 a0 54 2b c0 8b 44 24 24 89 ea 8b 48 18 b8 71 f0 
> 
> >>EIP; c0133a64 <kmem_cache_reap+1c4/490>   <=====
> Trace; c0134e58 <shrink_caches+18/90>
> Trace; c0134f0c <try_to_free_pages+3c/60>
> Trace; c0134fb0 <kswapd_balance_pgdat+50/a0>
> Trace; c0135026 <kswapd_balance+26/40>
> Trace; c013515c <kswapd+9c/b6>
> Trace; c0105000 <_stext+0/0>
> Trace; c0107296 <kernel_thread+26/30>
> Trace; c01350c0 <kswapd+0/b6>
> Code;  c0133a64 <kmem_cache_reap+1c4/490>
> 00000000 <_EIP>:
> Code;  c0133a64 <kmem_cache_reap+1c4/490>   <=====
>    0:   0f 0b                     ud2a      <=====
> Code;  c0133a66 <kmem_cache_reap+1c6/490>
>    2:   9f                        lahf   
> Code;  c0133a66 <kmem_cache_reap+1c6/490>
>    3:   05 a0 54 2b c0            add    $0xc02b54a0,%eax
> Code;  c0133a6c <kmem_cache_reap+1cc/490>
>    8:   8b 44 24 24               mov    0x24(%esp,1),%eax
> Code;  c0133a70 <kmem_cache_reap+1d0/490>
>    c:   89 ea                     mov    %ebp,%edx
> Code;  c0133a72 <kmem_cache_reap+1d2/490>
>    e:   8b 48 18                  mov    0x18(%eax),%ecx
> Code;  c0133a74 <kmem_cache_reap+1d4/490>
>   11:   b8 71 f0 00 00            mov    $0xf071,%eax
> 
> Entering kdb (current=0xf7edc000, pid 5) on processor 0 Oops: invalid
> operand
> eax = 0x5a2cf071 ebx = 0x006f5b20 ecx = 0xf7edf00c edx = 0xc1c0f060 
> esi = 0xe51e6000 edi = 0xe51e6c3c esp = 0xf7eddf48 eip = 0xc0133a64 
> ebp = 0xe51e6c18 xss = 0x00000018 xcs = 0x00000010 eflags = 0x00010016 
-- 

Steve Lord                                      voice: +1-651-683-3511
Principal Engineer, Filesystem Software         email: lord@xxxxxxx