<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, May 2, 2014 at 6:05 PM, Dave Chinner <span dir="ltr"><<a href="mailto:david@fromorbit.com" target="_blank">david@fromorbit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On Fri, May 02, 2014 at 11:32:19AM -0600, Bob Mastors wrote:<br>
> On Tue, Apr 29, 2014 at 9:22 PM, Dave Chinner <<a href="mailto:david@fromorbit.com">david@fromorbit.com</a>> wrote:<br>
</div><div class="">> > Well, I guess I should been more specific. ftrace is pretty much<br>
> > useless here - we need the information taht is recorded by the<br>
> > events, not the timing of the events.<br>
> ><br>
> Heh, I have been trying to bluff my way through this.<br>
> I have hacked on file systems but am new to linux and xfs.<br>
<br>
</div>I wouldn't have guessed that. Most of the time the response is<br>
"trace events? what are they?". :)<br>
<div class=""><br>
> Output file attached but it all looks like this:<br>
><br>
> xfsaild/sdb-<a href="tel:3613%20%20%5B001%5D%20%20%20445" value="+13613001445">3613 [001] 445</a>.757616: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88001d3017b8 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
> xfsaild/sdb-3613 [001] 445.757617: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88001d301850 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
> xfsaild/sdb-3613 [001] 445.757618: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88001de63ed8 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
> xfsaild/sdb-3613 [001] 445.757618: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88001cd24720 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
> xfsaild/sdb-3613 [001] 445.757618: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88001cd248e8 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
> xfsaild/sdb-3613 [001] 445.757619: xfs_ail_flushing: dev 8:16<br>
> lip 0x0xffff88002c435ab0 lsn 32/3573 type XFS_LI_INODE flags IN_AIL<br>
<br>
</div>That's very enlightening - a bunch of inodes returning<br>
XFS_ITEM_FLUSHING from attempts to push them.<br>
<div class=""><br>
> The addresses repeat.<br>
> I started the record as soon as I noticed the umount hanging, probably<br>
> within 20 seconds or so.<br>
> Also attached is the console output.<br>
><br>
> This was done with your patch applied.<br>
><br>
> Seems like xfsaild_push is missing a check that the file system is<br>
> XFS_FORCED_SHUTDOWN.<br>
<br>
</div>It might look that way, but what is supposed to happen is that<br>
everything else is supposed to handle shutdown situations and empty<br>
the AIL. i.e. the act of issuing IO by the AIL is supposed to<br>
trigger shutdown processing on the objects, and so having the AIL<br>
shut down itself won't help.<br>
<br>
So, what's the problem? Well, the inodes have been flushed,<br>
triggering the inode buffer to be written. However, then inode<br>
buffer has had an IO error (i.e. the async write error), is retried,<br>
and then we stop retrying the IO. The idea is that the AIL will push<br>
it again soon and the IO will be retried.<br>
<br>
What is happening is that the inodes in the buffer are already<br>
marked as "being flushed" and hence the AIL does not push the inode<br>
buffer again, and hence the IO completion callbacks on the buffer<br>
that will remove the inodes from the AIL are never called. Not sure<br>
how to fix this one yet....<br></blockquote><div>Thanks for the explanation.</div><div>It clears up some questions I had.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=""><br>
> Let me know if there is any other info I can capture that would be<br>
> interesting.<br>
<br>
</div>I know what the problem is now, so testing a fix will be the next<br>
step. But I need to think about how to fix this, and that can wait<br>
till Monday. ;)<br></blockquote><div>Just an fyi next week is busy for me with some other stuff so I</div><div>won't be able to try out a fix until the following week.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks for following up with the trace information, Bob!<br></blockquote><div>Not a problem.</div><div><br></div><div>Thanks,</div><div>Bob</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
Cheers,<br>
<br>
Dave.<br>
--<br>
Dave Chinner<br>
<a href="mailto:david@fromorbit.com">david@fromorbit.com</a><br>
</div></div></blockquote></div><br></div></div>