<div dir="ltr"><div><div><div><div>Dear Eric,<br></div>I have applied your latest patch and collected the following log:<br><br></div>/var/log/message<br>Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22 for inode 0x1b20b ag 0 agino 1b20b<br>
Apr 20 17:28:23 10 kernel:<br>Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22<br>Apr 20 17:28:23 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02d4d0a<br>
Apr 20 17:28:23 10 kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem<br>Apr 20 17:28:23 10 kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s)<br>Apr 20 17:28:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
Apr 20 17:29:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>Apr 20 17:29:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>Apr 20 17:30:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
<br></div>debugfs trace:<br><a href="https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing">https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing</a><br><br></div>Thank you.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/20 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>Hi Eric,<br></div>The xfs module is loaded from system kernel, it happens on our production server too (I did not touch that till now) and if the xfs module is mess up the systemstap may also not working but now it works. As you have mentioned, strange thing is xfs shutdown always happens when glusterfs rebalance completes.<br>
</div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">2013/4/20 Eric Sandeen <span dir="ltr"><<a href="mailto:sandeen@sandeen.net" target="_blank">sandeen@sandeen.net</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 4/19/13 9:03 PM, ·ûÓÀÌÎ wrote:<br>
> Hi Eric,<br>
> I will enable them and run test again. I can only reproduce it with<br>
> glusterfs rebalance. Glusterfs uses a mechanism it called syncop to<br>
> unlink file. For rebalance it uses<br>
> syncop_unlink(glusterfs/libglusterfs/src/syncop.c). In the glusterfs<br>
> sync_task framework(glusterfs/libglusterfs/src/syncop.c) it uses<br>
> "makecontext/swapcontext"<br>
</div>> <<a href="http://www.opengroup.org/onlinepubs/009695399/functions/makecontext.html" target="_blank">http://www.opengroup.org/onlinepubs/009695399/functions/makecontext.html</a>>.<br>
<div>> Does it leads to racing unlink from different CPU core?<br>
<br>
</div>Yep, I understand that it's rebalance. It dies when rebalance finishes because an<br>
open but unlinked file trips over the corrupted list from earlier, it seems.<br>
<br>
I don't know why makecontext would matter...<br>
<br>
Just to be sure, you are definitely loading the xfs module from the kernel you built, right, and you don't have a "priority" module getting loaded from elsewhere? Seems unlikely, but just to be sure.<br>
<br>
> Thank you.<br>
<br>
You could also add this patch to the xfs tracepoints to print more information about the inodes - the mode & flags.<br>
<br>
-Eric<br>
<br>
<br>
diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h<br>
index e8ce644..c314b87 100644<br>
--- a/fs/xfs/linux-2.6/xfs_trace.h<br>
+++ b/fs/xfs/linux-2.6/xfs_trace.h<br>
@@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,<br>
TP_STRUCT__entry(<br>
__field(dev_t, dev)<br>
__field(xfs_ino_t, ino)<br>
+ __field(__u16, mode)<br>
+ __field(unsigned long, flags)<br>
),<br>
TP_fast_assign(<br>
__entry->dev = VFS_I(ip)->i_sb->s_dev;<br>
__entry->ino = ip->i_ino;<br>
+ __entry->mode = VFS_I(ip)->i_mode;<br>
+ __entry->flags = ip->i_flags;<br>
),<br>
- TP_printk("dev %d:%d ino 0x%llx",<br>
+ TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",<br>
MAJOR(__entry->dev), MINOR(__entry->dev),<br>
- __entry->ino)<br>
+ __entry->ino, __entry->mode, __entry->flags)<br>
)<br>
<br>
#define DEFINE_INODE_EVENT(name) \<br>
<br>
<br>
<br>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>