xfs
[Top] [All Lists]

Re: XFS crash?

To: Austin Schuh <austin@xxxxxxxxxxxxxxxx>
Subject: Re: XFS crash?
From: Dave Chinner <david@xxxxxxxxxxxxx>
Date: Tue, 13 May 2014 16:39:43 +1000
Cc: xfs <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CANGgnMZ0q9uE3NHj2i0SBK1d0vdKLx7QBJeFNb+YwP-5EAmejQ@xxxxxxxxxxxxxx>
References: <CANGgnMYPLF+8616Rs9eQOXUc9He2NSgFnNrvHvepV-x+pWS6oQ@xxxxxxxxxxxxxx> <20140305233551.GK6851@dastard> <CANGgnMb=2dYGQO4K36pQ9LEb8E4rT6S_VskLF+n=ndd0_kJr_g@xxxxxxxxxxxxxx> <CANGgnMa80WwQ8zSkL52yYegmQURVQeZiBFv41=FQXMZJ_NaEDw@xxxxxxxxxxxxxx> <20140513034647.GA5421@dastard> <CANGgnMZ0q9uE3NHj2i0SBK1d0vdKLx7QBJeFNb+YwP-5EAmejQ@xxxxxxxxxxxxxx>
User-agent: Mutt/1.5.21 (2010-09-15)
On Mon, May 12, 2014 at 09:03:48PM -0700, Austin Schuh wrote:
> On Mon, May 12, 2014 at 8:46 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> > On Mon, May 12, 2014 at 06:29:28PM -0700, Austin Schuh wrote:
> >> On Wed, Mar 5, 2014 at 4:53 PM, Austin Schuh <austin@xxxxxxxxxxxxxxxx> 
> >> wrote:
> >> > Hi Dave,
> >> >
> >> > On Wed, Mar 5, 2014 at 3:35 PM, Dave Chinner <david@xxxxxxxxxxxxx> wrote:
> >> >> On Wed, Mar 05, 2014 at 03:08:16PM -0800, Austin Schuh wrote:
> >> >>> Howdy,
> >> >>>
> >> >>> I'm running a config_preempt_rt patched version of the 3.10.11 kernel,
> >> >>> and I'm seeing a couple lockups and crashes which I think are related
> >> >>> to XFS.
> >> >>
> >> >> I think they ar emore likely related to RT issues....
> >> >>
> >> >
> >> > That very well may be true.
> >> >
> >> >> Your usb device has disconnected and gone down the device
> >> >> removal/invalidate partition route. and it's trying to flush the
> >> >> device, which is stuck on IO completion which is stuck waiting for
> >> >> the device error handling to error them out.
> >> >>
> >> >> So, this is a block device problem error handling problem caused by
> >> >> device unplug getting stuck because it's decided to ask the
> >> >> filesystem to complete operations that can't be completed until the
> >> >> device error handling progress far enough to error out the IOs that
> >> >> the filesystem is waiting for completion on.
> >> >>
> >> >> Cheers,
> >> >>
> >> >> Dave.
> >> >> --
> >> >> Dave Chinner
> >> >> david@xxxxxxxxxxxxx
> >>
> >> I had the issue reproduce itself today with just the main SSD
> >> installed.  This was on a new machine that was built this morning.
> >> There is a lot less going on in this trace than the previous one.
> >
> > The three blocked threads:
> >
> >         1. kworker running IO completion waiting on an inode lock,
> >            holding locked pages.
> >         2. kworker running writeback flusher work waiting for a page lock
> >         3. direct flush work waiting for allocation, holding page
> >            locks and the inode lock.
> >
> > What's the kworker thread running the allocation work doing?
> >
> > You might need to run `echo w > proc-sysrq-trigger` to get this
> > information...
> 
> I was able to reproduce the lockup.  I ran `echo w >
> /proc/sysrq-trigger` per your suggestion.  I don't know how to figure
> out what the kworker thread is doing, but I'll happily do it if you
> can give me some guidance.

There isn't a worker thread blocked doing an allocation in that
dump, so it doesn't shed any light on the problem at all. try
`echo l > /proc/sysrq-trigger`, followed by `echo t >
/proc/sysrq-trigger` so we can see all the processes running on CPUs
and all the processes in the system...

Cheers,

Dave.
-- 
Dave Chinner
david@xxxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>