<div dir="ltr"><div><div>Hi Brain and Eric,<br></div>If the problem is the agno can't be found in the unlinked list. Can we just bypass it instead of passing ino=0xffffffff to xfs_inotobp?<br></div>Thank you.<br></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/18 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div>Hi Eric,<br></div>The shutdown issue is still not reproduced yet. But I get the following error today during test.<br><br>Apr 18 07:42:51 10 kernel: Call Trace:<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d91ef>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]<br>
Apr 18 07:42:51 10 kernel: [<ffffffff814fe6a5>] schedule_timeout+0x215/0x2e0<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d5f07>] ? kmem_zone_alloc+0x77/0xf0 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffff814ff5c2>] __down+0x72/0xb0<br>
Apr 18 07:42:51 10 kernel: [<ffffffffa02da652>] ? _xfs_buf_find+0x102/0x280 [xfs]<br>Apr 18 07:42:51 10 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>Apr 18 07:42:51 10 kernel: glusterfsd D ffffffff8160b3c0 0 14522 1 0x00000083<br>
Apr 18 07:42:51 10 kernel: ffff882015a63a28 0000000000000082 0000000000000000 0000000000000000<br>Apr 18 07:42:51 10 kernel: ffff882015a639b8 ffffffffa02d91ef ffff882015a639d8 0000000000000246<br>Apr 18 07:42:51 10 kernel: ffff880e70491af8 ffff882015a63fd8 000000000000fb88 ffff880e70491af8<br>
Apr 18 07:42:51 10 kernel: Call Trace:<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d91ef>] ? xfs_buf_cond_lock+0x2f/0xc0 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffff814fe6a5>] schedule_timeout+0x215/0x2e0<br>
Apr 18 07:42:51 10 kernel: [<ffffffffa02d5f07>] ? kmem_zone_alloc+0x77/0xf0 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffff814ff5c2>] __down+0x72/0xb0<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02da652>] ? _xfs_buf_find+0x102/0x280 [xfs]<br>
Apr 18 07:42:51 10 kernel: [<ffffffff81097ef1>] down+0x41/0x50<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02da493>] xfs_buf_lock+0x53/0x110 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02da652>] _xfs_buf_find+0x102/0x280 [xfs]<br>
Apr 18 07:42:51 10 kernel: [<ffffffffa02da83b>] xfs_buf_get+0x6b/0x1a0 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02daeac>] xfs_buf_read+0x2c/0x100 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d0af8>] xfs_trans_read_buf+0x1f8/0x400 [xfs]<br>
Apr 18 07:42:51 10 kernel: [<ffffffffa02b3444>] xfs_read_agi+0x74/0x100 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02b967b>] xfs_iunlink+0x5b/0x180 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffff810724c7>] ? current_fs_time+0x27/0x30<br>
Apr 18 07:42:51 10 kernel: [<ffffffffa02d12a7>] ? xfs_trans_ichgtime+0x27/0xa0 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d15fb>] xfs_droplink+0x5b/0x70 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02d2f9e>] xfs_remove+0x27e/0x3a0 [xfs]<br>
Apr 18 07:42:51 10 kernel: [<ffffffff81186fd3>] ? generic_permission+0x23/0xb0<br>Apr 18 07:42:51 10 kernel: [<ffffffffa02e0968>] xfs_vn_unlink+0x48/0x90 [xfs]<br>Apr 18 07:42:51 10 kernel: [<ffffffff81188c0f>] vfs_unlink+0x9f/0xe0<br>
Apr 18 07:42:51 10 kernel: [<ffffffff8118795a>] ? lookup_hash+0x3a/0x50<br>Apr 18 07:42:51 10 kernel: [<ffffffff8118b143>] do_unlinkat+0x183/0x1c0<br>Apr 18 07:42:51 10 kernel: [<ffffffff81017938>] ? syscall_trace_enter+0x1d8/0x1e0<br>
Apr 18 07:42:51 10 kernel: [<ffffffff8118b196>] sys_unlink+0x16/0x20<br>Apr 18 07:42:51 10 kernel: [<ffffffff8100b308>] tracesys+0xd9/0xde<br><br></div>Thank you.<br></div><div class="gmail_extra"><div><div class="h5">
<br><br><div class="gmail_quote">
2013/4/17 Eric Sandeen <span dir="ltr"><<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto"><div><div>On Apr 16, 2013, at 8:48 PM, ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div>
<div>Hi Brain,<br></div>Can I change as following?<br></div></div></div></blockquote><div><br></div></div>ASSERTS are no-ops in a non-debug kernel, so this won't change any behavior. I hope we'll know more if we get new traces from your patched kernel....<div>
<br></div><div>Eric</div><div><div><div><br><blockquote type="cite"><div><div dir="ltr"><div>--- a/xfs_inode.c<br>+++ b/xfs_inode.c<br>@@ -1773,6 +1773,8 @@ xfs_iunlink_remove(<br> if (last_ibp != NULL) {<br>
xfs_trans_brelse(tp, last_ibp);<br>
}<br>+ ASSERT(next_agino != NULLAGINO);<br>+ ASSERT(next_agino != 0);<br> next_ino = XFS_AGINO_TO_INO(mp, agno, next_agino);<br>
error = xfs_inotobp(mp, tp, next_ino, &last_dip,<br> &last_ibp, &last_offset, 0);<br>@@ -1783,8 +1785,6 @@ xfs_iunlink_remove(<br> return error;<br>
}<br> next_agino = be32_to_cpu(last_dip->di_next_unlinked);<br>- ASSERT(next_agino != NULLAGINO);<br>- ASSERT(next_agino != 0);<br>
}<br> /*<br> * Now last_ibp points to the buffer previous to us on<br><br></div>Thank you.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/17 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Brain,<br></div>If it is because NULLAGINO is passed in to xfs_inotobp().<br></div>Can I move the following two lines before xfs_inotobp?<br>
<br>For example:<br><br>1767 while (next_agino != agino) {<br>
1768 /*<br>1769 * If the last inode wasn't the one pointing to<br>1770 * us, then release its buffer since we're not<br>1771 * going to do anything with it.<br>
1772 */<br>1773 if (last_ibp != NULL) {<br>1774 xfs_trans_brelse(tp, last_ibp);<br>1775 }<br>1776 next_ino = XFS_AGINO_TO_INO(mp, agno, next_agino);<br>
+ ASSERT(next_agino != NULLAGINO);<br>+ ASSERT(next_agino != 0);<br>1777 error = xfs_inotobp(mp, tp, next_ino, &last_dip,<br>1778 &last_ibp, &last_offset, 0);<br>
1779 if (error) {<br>1780 xfs_warn(mp,<br>1781 "%s: xfs_inotobp() returned error %d.",<br>1782 __func__, error);<br>
1783 return error;<br>1784 }<br>1785 next_agino = be32_to_cpu(last_dip->di_next_unlinked);<br>- //ASSERT(next_agino != NULLAGINO);<br>
- //ASSERT(next_agino != 0);<br>1788 }<br></div>I don't understand xfs well and correct me if I'm totally wrong.<br><div><div><div>Thank you very much.<br></div></div>
</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/17 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Brain,<div>I want to ask a question, according to the shutdown trace. The ino in<font face="arial, sans-serif"><span style="font-size:14px"> </span></font><span style="font-family:arial,sans-serif;font-size:14px"> </span><span style="font-family:arial,sans-serif;font-size:14px">xfs_iunlink_remove is </span><span style="font-family:arial,sans-serif;font-size:14px">0x113, why </span><span style="font-family:arial,sans-serif;font-size:14px">xfs_imap got </span><span style="font-family:arial,sans-serif;font-size:14px">ino=0xffffffff</span><span style="font-family:arial,sans-serif;font-size:14px"> ? </span></div>
<div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:14px">--- xfs_imap -- module("xfs").function("xfs_</span><span style="font-family:arial,sans-serif;font-size:14px">imap@fs/xfs/xfs_ialloc.c:1257"</span><span style="font-family:arial,sans-serif;font-size:14px">).return -- return=0x16</span><br style="font-family:arial,sans-serif;font-size:14px">
<span style="font-family:arial,sans-serif;font-size:14px">vars: mp=0xffff882017a50800 tp=0xffff881c81797c70 ino=0xffffffff</span><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:14px">--- xfs_iunlink_remove -- module("xfs").function("xfs_</span><span style="font-family:arial,sans-serif;font-size:14px">iunlink_remove@fs/xfs/xfs_</span><span style="font-family:arial,sans-serif;font-size:14px">inode.c:1680").return -- return=0x16</span><br style="font-family:arial,sans-serif;font-size:14px">
<span style="font-family:arial,sans-serif;font-size:14px">vars: tp=0xffff881c81797c70 ip=0xffff881003c13c00 next_ino=? mp=? agi=? dip=? agibp=0xffff880109b47e20 ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff882000000000 bucket_index=? offset=? last_offset=0xffffffffffff8810 error=? __func__=[...]</span><br style="font-family:arial,sans-serif;font-size:14px">
<span style="font-family:arial,sans-serif;font-size:14px">ip: i_ino = 0x113, i_flags = 0x0</span><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br>
</span></div><div><span style="font-family:arial,sans-serif;font-size:14px">Thank you.</span></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div></div><div class="gmail_extra"><div><div>
<br><br><div class="gmail_quote">2013/4/17 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div>Hi Brain,<br></div>Thank you for your update, and I have applied your last kernel patch. However it is not easy to reproduce especially in out test environment. Till now is not happens again. I'll update the kernel patch now. BTW is there any findings in the logs of previous thread?<br>
<a href="http://oss.sgi.com/archives/xfs/2013-04/msg00327.html" target="_blank">http://oss.sgi.com/archives/xfs/2013-04/msg00327.html</a><br>I guess it tend to happen during glusterfs rebalance because glusterfs moves a lot of file from one server to another and then unlink it.<br>
<br></div>Thank you.<br></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">2013/4/17 Brian Foster <span dir="ltr"><<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 04/16/2013 12:24 PM, Dave Chinner wrote:<br>
> On Mon, Apr 15, 2013 at 07:14:39PM -0400, Brian Foster wrote:<br>
>> Hi,<br>
>><br>
>> Thanks for the data in the previous thread:<br>
>><br>
>> <a href="http://oss.sgi.com/archives/xfs/2013-04/msg00327.html" target="_blank">http://oss.sgi.com/archives/xfs/2013-04/msg00327.html</a><br>
>><br>
</div>...<br>
<div>>><br>
>> echo 1 > /sys/kernel/debug/tracing/events/xfs/xfs_iunlink/enable<br>
>> echo 1 > /sys/kernel/debug/tracing/events/xfs/xfs_iunlink_remove/enable<br>
>> ... reproduce ...<br>
>> cat /sys/kernel/debug/tracing/trace > trace.output<br>
><br>
> It's better to use trace-cmd for this. it will result in less<br>
> dropped events. i.e.:<br>
><br>
> $ trace-cmd record -e xfs_iunlink\*<br>
> ... reproduce ...<br>
> ^C<br>
> $ trace-cmd report > trace.output<br>
><br>
>> --- a/fs/xfs/linux-2.6/xfs_trace.h<br>
>> +++ b/fs/xfs/linux-2.6/xfs_trace.h<br>
>> @@ -581,6 +581,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);<br>
</div>...<br>
<div>><br>
> I would suggest that the the tracing shoul dbe at entry of the<br>
> function, otherwise we won't get a tracepoint for the operation that<br>
> triggers the shutdown. (That's the reason most tracepoints in XFS<br>
> are at function entry...)<br>
><br>
<br>
</div>Good points, thanks Dave. A v2 that pulls up the tracepoints towards<br>
function entry is appended.<br>
<br>
Brian<br>
<br>
>From 280943e78ebe0b97a774cba51e7815c42f044b55 Mon Sep 17 00:00:00 2001<br>
From: Brian Foster <<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>><br>
Date: Mon, 15 Apr 2013 18:16:24 -0400<br>
Subject: [PATCH v2] xfs: add tracepoints for xfs_iunlink and<br>
xfs_iunlink_remove<br>
<div><br>
---<br>
fs/xfs/linux-2.6/xfs_trace.h | 2 ++<br>
fs/xfs/xfs_inode.c | 4 ++++<br>
2 files changed, 6 insertions(+), 0 deletions(-)<br>
<br>
diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h<br>
</div>index adc6ec4..338a0f9 100644<br>
--- a/fs/xfs/linux-2.6/xfs_trace.h<br>
+++ b/fs/xfs/linux-2.6/xfs_trace.h<br>
@@ -583,6 +583,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);<br>
DEFINE_INODE_EVENT(xfs_destroy_inode);<br>
DEFINE_INODE_EVENT(xfs_dirty_inode);<br>
<div> DEFINE_INODE_EVENT(xfs_clear_inode);<br>
+DEFINE_INODE_EVENT(xfs_iunlink);<br>
+DEFINE_INODE_EVENT(xfs_iunlink_remove);<br>
<br>
DEFINE_INODE_EVENT(xfs_dquot_dqalloc);<br>
DEFINE_INODE_EVENT(xfs_dquot_dqdetach);<br>
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c<br>
</div>index 19900f0..d705c77 100644<br>
--- a/fs/xfs/xfs_inode.c<br>
+++ b/fs/xfs/xfs_inode.c<br>
@@ -1615,6 +1615,8 @@ xfs_iunlink(<br>
<br>
mp = tp->t_mountp;<br>
<br>
+ trace_xfs_iunlink(ip);<br>
+<br>
/*<br>
* Get the agi buffer first. It ensures lock ordering<br>
* on the list.<br>
@@ -1694,6 +1696,8 @@ xfs_iunlink_remove(<br>
mp = tp->t_mountp;<br>
agno = XFS_INO_TO_AGNO(mp, ip->i_ino);<br>
<br>
+ trace_xfs_iunlink_remove(ip);<br>
+<br>
/*<br>
* Get the agi buffer first. It ensures lock ordering<br>
* on the list.<br>
<span><font color="#888888">--<br>
1.7.7.6<br>
<br>
</font></span></blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>
</div></blockquote></div></div><blockquote type="cite"><div><span>_______________________________________________</span><br><span>xfs mailing list</span><br><span><a href="mailto:xfs@oss.sgi.com" target="_blank">xfs@oss.sgi.com</a></span><br>
<span><a href="http://oss.sgi.com/mailman/listinfo/xfs" target="_blank">http://oss.sgi.com/mailman/listinfo/xfs</a></span><br></div></blockquote></div></div></blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>
·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>