From: "Richard Smith" <rgsmith72@xxxxxxxxxxxxx>
Reply-To: "Richard Smith" <rgsmith72@xxxxxxxxxxxxx>
To: "Russell Cattelan" <cattelan@xxxxxxx>,"Rick Smith"
<rgsmith72@xxxxxxxxxxx>
CC: <sandeen@xxxxxxx>,<linux-xfs@xxxxxxxxxxx>,<willy@xxxxxxxxxx>
Subject: Re: rm hanging intermittently
Date: Fri, 17 Jan 2003 21:52:29 -0800
Russell,
Below are the bdflush, kswapd, kupdated and pagebufd backtraces from
the
lastest hanging process. They all seem to be paused in the same place. If I
kill the hung rm process, everyone wakes back up and the system continues
normally.
Where do I create a new bugzilla bug? Thanks for your help.
Rick
[2]kdb> btp 8
0xf7cd0000 00000008 00000001 0 002 stop 0xf7cd0370 bdflush
ESP EIP Function (args)
0xf7cd1f84 0xc0116063 schedule+0x493 (0x0, 0xf7cd0000, 0xc040d538,
0xc040d538, 0x1f4)
kernel .text 0xc0100000 0xc0115bd0
0xc0116120
0xf7cd1fc4 0xc01164ca interruptible_sleep_on+0x4a (0x10f00, 0xf7ffbfb8)
kernel .text 0xc0100000 0xc0116480
0xc0116500
0xf7cd1fe4 0xc013e6b7 bdflush+0xc7
kernel .text 0xc0100000 0xc013e5f0
0xc013e6c0
0xf7cd1ff4 0xc0107296 kernel_thread+0x26
kernel .text 0xc0100000 0xc0107270
0xc01072a0
[2]kdb> go
[0]kdb> btp 7
0xf7cd4000 00000007 00000001 0 000 stop 0xf7cd4370 kswapd
ESP EIP Function (args)
0xf7cd5f94 0xc0116063 schedule+0x493 (0x0, 0xf7cd4000, 0xc040cad4,
0xc040cad4, 0x10f00)
kernel .text 0xc0100000 0xc0115bd0
0xc0116120
0xf7cd5fd4 0xc0132d66 kswapd+0x86
kernel .text 0xc0100000 0xc0132ce0
0xc0132d96
0xf7cd5ff4 0xc0107296 kernel_thread+0x26
kernel .text 0xc0100000 0xc0107270
0xc01072a0
[0]kdb> go
[3]kdb> btp 9
0xf7cce000 00000009 00000001 0 003 stop 0xf7cce370 kupdated
ESP EIP Function (args)
0xf7ccff68 0xc0116063 schedule+0x493 (0xf7ccffac, 0xc04abd04, 0xc044a7fc,
0x1bc3472, 0xf7cce000)
kernel .text 0xc0100000 0xc0115bd0
0xc0116120
0xf7ccffa8 0xc0115b1e schedule_timeout+0x7e (0x0, 0x10f00, 0xf7ffbfac,
0xc0105000)
kernel .text 0xc0100000 0xc0115aa0
0xc0115b40
0xf7ccffdc 0xc013e764 kupdate+0xa4
kernel .text 0xc0100000 0xc013e6c0
0xc013e800
0xf7ccfff4 0xc0107296 kernel_thread+0x26
kernel .text 0xc0100000 0xc0107270
0xc01072a0
[3]kdb> go
[2]kdb> btp 10
0xf7c98000 00000010 00000001 0 002 stop 0xf7c98370 pagebufd
ESP EIP Function (args)
0xf7c99f48 0xc0116063 schedule+0x493 (0x0, 0xf7c98000, 0xc04130e8,
0xc04130e8, 0xf7c99fc0)
kernel .text 0xc0100000 0xc0115bd0
0xc0116120
0xf7c99f88 0xc01164ca interruptible_sleep_on+0x4a (0xf7c99fc0, 0xf7c99fc0,
0xf7c99fc0, 0xf7c98000, 0xf7c99fb8)
kernel .text 0xc0100000 0xc0116480
0xc0116500
0xf7c99fa8 0xc02044a3 pagebuf_daemon+0xd3
kernel .text 0xc0100000 0xc02043d0
0xc0204640
0xf7c99ff4 0xc0107296 kernel_thread+0x26
kernel .text 0xc0100000 0xc0107270
0xc01072a0
[2]kdb> go
----- Original Message -----
From: "Russell Cattelan" <cattelan@xxxxxxx>
To: "Rick Smith" <rgsmith72@xxxxxxxxxxx>
Cc: <sandeen@xxxxxxx>; <linux-xfs@xxxxxxxxxxx>; <rgsmith72@xxxxxxxxxxxxx>;
<willy@xxxxxxxxxx>
Sent: Wednesday, January 15, 2003 1:46 PM
Subject: RE: rm hanging intermittently
> Before this gets lost can you open a bug in bugzilla and
> add this BT to it.
>
> And actually a bta (back trace all) would probably be more helpful
> since it usually informative to see what the kernel threads bdflush
> kswapd, kupdated, and the pagebuf daemons are doing.
>
> On Wed, 2003-01-15 at 14:14, Rick Smith wrote:
> > Eric,
> > I was able to successfully capture a backtrace of the hanging rm
> > problem today. It looks like it (and several other processes) are
stuck
in
> > the schedule() function. After searching the mailing list archive,
there
> > were several other similar problems concerning deadlock, but the
closest
was
> > posted by Matthew Wilcox at debian on 10-30-02 with the subject
"unlink
> > deadlock". The backtrace that he posted is quite similar to mine and
we
are
> > both using raid 0. I, however, am not using IA-64 architecture.
> > For me the problem seems to happen when there are multiple writes
(to
> > two different XFS partitions) very close to each other. Has this
deadlock
> > been addressed in kernels later than 2.4.20-rc1-xfs? Thanks for your
help.
> > The backtrace follows:
> >
> > [2]kdb> btp 18758
> > 0xd5812000 00018758 00018739 0 002 stop 0xd5812370 rm
> > ESP EIP Function (args)
> > 0xd5813d38 0xc0116063 schedule+0x493 (0x1, 0xd5812000, 0xf7c3ad8c,
> > 0xf7c3ad8c, 0xf7580700)
> > kernel .text 0xc0100000 0xc0115bd0
0xc0116120
> > 0xd5813d78 0xc0107828 __down+0x68
> > kernel .text 0xc0100000 0xc01077c0
0xc0107890
> > 0xd5813d94 0xc01079d4 __down_failed+0x8 (0x33c, 0xd5813df8,
0xf711f3cc,
> > 0xd5813dfc, 0xc01ec543)
> > kernel .text 0xc0100000 0xc01079cc
0xc01079d8
> > 0xd5813da4 0xc01ee0eb .text.lock.xfs_log+0xdb
> > kernel .text 0xc0100000 0xc01ee010
0xc01ee250
> > 0xd5813da4 0xc01eccb2 xlog_state_get_iclog_space+0x62 (0xf7c3ad80,
0x33c,
> > 0xd5813df8, 0xf711f3cc, 0xd5813dfc)
> > kernel .text 0xc0100000 0xc01ecc50
0xc01ecda0
> > 0xd5813db8 0xc01ec543 xlog_write+0x153 (0xf6ff8c00, 0xd5813e68, 0xc,
> > 0xf711f3cc, 0xdf8b2c5c)
> > kernel .text 0xc0100000 0xc01ec3f0
0xc01ec800
> > 0xd5813e18 0xc01eb51c xfs_log_write+0x3c (0xf6ff8c00, 0xd5813e68, 0xc,
> > 0xf711f3cc, 0xdf8b2c5c)
> > kernel .text 0xc0100000 0xc01eb4e0
0xc01eb550
> > 0xd5813e3c 0xc01f7b24 xfs_trans_commit+0x184 (0xdf8b2c10, 0x4, 0x0,
> > 0xd5813f2c, 0x11)
> > kernel .text 0xc0100000 0xc01f79a0
0xc01f7c50
> > 0xd5813efc 0xc01fe6b8 xfs_remove+0x398 (0xd8c876d8, 0xd8c85580, 0x0)
> > kernel .text 0xc0100000 0xc01fe320
0xc01fe790
> > 0xd5813f54 0xc0209fbe linvfs_unlink+0x1e (0xd8c86da0, 0xd8c85580)
> > kernel .text 0xc0100000 0xc0209fa0
0xc020a000
> > 0xd5813f70 0xc0145d35 vfs_unlink+0x135 (0xd8c86da0, 0xd8c85580,
0xd8c8e180,
> > 0xf7cd2e40, 0xf2df9000)
> > kernel .text 0xc0100000 0xc0145c00
0xc0145da0
> > 0xd5813f8c 0xc0145e29 sys_unlink+0x89 (0x8053dab, 0x1, 0x0, 0x8053dab,
0x0)
> > kernel .text 0xc0100000 0xc0145da0
0xc0145e90
> > 0xd5813fc4 0xc0108c2b system_call+0x33
> > kernel .text 0xc0100000 0xc0108bf8
0xc0108c30
> > [2]kdb> go
> >
> > Rick Smith
> >
> > _________________________________________________________________
> > MSN 8 with e-mail virus protection service: 2 months FREE*
> > http://join.msn.com/?page=features/virus
> --
> Russell Cattelan <cattelan@xxxxxxx>