XFS hung task in xfs_ail_push_all_sync() when unmounting FS after disk failure/recovery

[Carlos Maiolino]

On Fri, Mar 25, 2016 at 08:56:03AM +1100, Dave Chinner wrote:
> On Thu, Mar 24, 2016 at 05:52:44PM +0100, Carlos Maiolino wrote:
> > I can now reproduce it, or at least part of the problem.
> > 
> > Regarding your question Dave, yes, it can be unmounted after I issue xfs_io shutdown
> > command. But, if a umount is issued before that, then we can't find the
> > mountpoint anymore.
> > 
> > I'm not sure if I'm correct, but, what it looks like to me, as you already
> > mentioned, is that we keep getting IO errors but we never actually shutdown
> > the filesystem while doing async metadata writes.
> 
> *nod*
> 
> > I believe I've found the problem. So, I will try to explain it, so you guys
> > can review and let me know if I'm right or not
> > 
> > I was looking the code, and for me, looks like async retries are designed to
> > keep retrying forever, and rely on some other part of the filesystem to actually
> > shutdown it.
> 
> *nod*
> 
> [snip description of metadata IO error behaviour]
> 
> Yes, that is exactly how the code is expected to behave - in fact,
> that's how it was originally designed to behave.
> 
> > Looks like, somebody already noticed it:
> > 
> >         /*
> >         ¦* If the write was asynchronous then no one will be looking for the
> >         ¦* error.  Clear the error state and write the buffer out again.
> >         ¦*
> >         ¦* XXX: This helps against transient write errors, but we need to find
> >         ¦* a way to shut the filesystem down if the writes keep failing.
> >         ¦*
> >         ¦* In practice we'll shut the filesystem down soon as non-transient
> >         ¦* errors tend to affect the whole device and a failing log write
> >         ¦* will make us give up.  But we really ought to do better here.
> >         ¦*/
> > 
> > 
> > So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
> > setting limits to IO errors can be slightly modified to fix this issue too, but,
> 
> The patchset I have doesn't need modification to fix this issue - it
> has a patch specifically to address this, and it changes the default
> behaviour to "fail async writes at unmount":
> 
> http://oss.sgi.com/archives/xfs/2015-08/msg00092.html
> 
> > the problem is that the user must set it BEFORE he tries to unmount the
> > filesystem, otherwise it will get stuck here.
> 
> Yes, but that doesn't answer the big question: why don't the
> periodic log forces that are failing with EIO cause a filesystem
> shutdown? We issue a log force every 30s even during unmount, and a
> failed log IO must cause the filesystem to shut down. So why aren't
> these causing the filesystem to shutdown as we'd expect when the
> device has been pulled?
> 

Right, good point, I'll take a look on it

> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david at fromorbit.com
> 
> _______________________________________________
> xfs mailing list
> xfs at oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Carlos