Date: Wed, 4 Aug 2010 16:53:47 +1000
From: Dave Chinner <david@fromorbit.com>
To: Ilia Mirkin <imirkin@alum.mit.edu>
Cc: xfs@oss.sgi.com
Subject: Re: XFS hung on 2.6.33.3 kernel
Message-ID: <20100804065347.GZ7362@dastard>
References: <AANLkTilX3l8TbUztLStj_u9OqOZnBrsNQxmeV4DuBmYJ@mail.gmail.com>
 <20100718012033.GA18888@dastard>
 <AANLkTikEv75KRyRTs4awmG894NSKMnBkJNJPYsypMdWf@mail.gmail.com>
 <20100718235036.GC32635@dastard>
 <AANLkTi=EzePfBRdEFQo2BRtKy=464Ci6zPG=UyYJiwNp@mail.gmail.com>
 <20100804004746.GT7362@dastard>
 <AANLkTikfK6z7b=0Vpijyz-3m3+_MTtcrbeQk-B9LkK61@mail.gmail.com>
 <20100804042725.GX7362@dastard>
 <AANLkTikkwW48u_y=Tzkh=ob-YanbOpGmCaAH7q=8PpFx@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <AANLkTikkwW48u_y=Tzkh=ob-YanbOpGmCaAH7q=8PpFx@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)

On Wed, Aug 04, 2010 at 12:39:08AM -0400, Ilia Mirkin wrote:
> On Wed, Aug 4, 2010 at 12:27 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Tue, Aug 03, 2010 at 09:15:53PM -0400, Ilia Mirkin wrote:
> >> On Tue, Aug 3, 2010 at 8:47 PM, Dave Chinner <david@fromorbit.com> wrote:
> >> > Ilia,
> >> >
> >> > Can you send me the output of this for your kernel that the
> >> > traces came from:
> >> >
> >> > $ gdb <path/to/vmlinux>
> >> > (gdb) l *( xfs_write+0x2cc)
> >> >
> >> > You can run it against the vmlinux file in the kernel build
> >> > directory.  Basically I need to know which xfs_ilock() call in
> >> > xfs_write() one of the mysqld-test processes is stuck on.
> >>
> >> No problem - BTW, I'm running this on a 2.6.33.3 kernel (same as the
> >> one before, although diff hardware). If you want (and are fine with me
> >> "destroying" the current state), I can upgrade it to a kernel of your
> >> choice and repeat the test overnight.
> >>
> >> Naturally I didn't have CONFIG_DEBUG_INFO in there... just changed
> >> that to Y and recompiled. I'm not entirely sure that this preserves
> >> all the offsets, but at least the BUG-HUNTING doc makes allusions that
> >> it would.
> >>
> >> (gdb) l *( xfs_write+0x2cc)
> >> 0xffffffff8124342d is in xfs_write (fs/xfs/linux-2.6/xfs_lrw.c:597).
> >> 592                     if (!need_i_mutex && (mapping->nrpages || pos
> >> > xip->i_size)) {
> >> 593                             xfs_iunlock(xip, XFS_ILOCK_EXCL|iolock);
> >> 594                             iolock = XFS_IOLOCK_EXCL;
> >> 595                             need_i_mutex = 1;
> >> 596                             mutex_lock(&inode->i_mutex);
> >> 597                             xfs_ilock(xip, XFS_ILOCK_EXCL|iolock);
> >
> > Make sense. Can you run 'l *(xfs_ilock+0x2c)' as well? I just need to
> > confirm which lock it has blocked on.
> 
> (gdb) l *(xfs_ilock+0x2c)
> 0xffffffff81221001 is in xfs_ilock (fs/xfs/linux-2.6/mrlock.h:48).
> 43              down_read_nested(&mrp->mr_lock, subclass);
> 44      }
> 45
> 46      static inline void mrupdate_nested(mrlock_t *mrp, int subclass)
> 47      {
> 48              down_write_nested(&mrp->mr_lock, subclass);
> 49      #ifdef DEBUG
> 50              mrp->mr_writer = 1;
> 51      #endif
> 52      }

OK, that doesn't help - it followed into the inline function
rather than telling me which of the two calls in the function it
was. I guess I'll need the disassembly output to work it out.
Can you send the output of "disass xfs_ilock" instead? Thanks.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com