On Thu, Mar 24, 2016 at 05:52:44PM +0100, Carlos Maiolino wrote:
> I can now reproduce it, or at least part of the problem.
>
> Regarding your question Dave, yes, it can be unmounted after I issue xfs_io
> shutdown
> command. But, if a umount is issued before that, then we can't find the
> mountpoint anymore.
>
> I'm not sure if I'm correct, but, what it looks like to me, as you already
> mentioned, is that we keep getting IO errors but we never actually shutdown
> the filesystem while doing async metadata writes.
*nod*
> I believe I've found the problem. So, I will try to explain it, so you guys
> can review and let me know if I'm right or not
>
> I was looking the code, and for me, looks like async retries are designed to
> keep retrying forever, and rely on some other part of the filesystem to
> actually
> shutdown it.
*nod*
[snip description of metadata IO error behaviour]
Yes, that is exactly how the code is expected to behave - in fact,
that's how it was originally designed to behave.
> Looks like, somebody already noticed it:
>
> /*
> ¦* If the write was asynchronous then no one will be looking for the
> ¦* error. Clear the error state and write the buffer out again.
> ¦*
> ¦* XXX: This helps against transient write errors, but we need to find
> ¦* a way to shut the filesystem down if the writes keep failing.
> ¦*
> ¦* In practice we'll shut the filesystem down soon as non-transient
> ¦* errors tend to affect the whole device and a failing log write
> ¦* will make us give up. But we really ought to do better here.
> ¦*/
>
>
> So, if I'm write in how we hit this problem, and IIRC, Dave's patchset for
> setting limits to IO errors can be slightly modified to fix this issue too,
> but,
The patchset I have doesn't need modification to fix this issue - it
has a patch specifically to address this, and it changes the default
behaviour to "fail async writes at unmount":
http://oss.sgi.com/archives/xfs/2015-08/msg00092.html
> the problem is that the user must set it BEFORE he tries to unmount the
> filesystem, otherwise it will get stuck here.
Yes, but that doesn't answer the big question: why don't the
periodic log forces that are failing with EIO cause a filesystem
shutdown? We issue a log force every 30s even during unmount, and a
failed log IO must cause the filesystem to shut down. So why aren't
these causing the filesystem to shutdown as we'd expect when the
device has been pulled?
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|