xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: Eric Sandeen <sandeen@xxxxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Sat, 20 Apr 2013 12:27:39 +0800
Cc: Brian Foster <bfoster@xxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=UVmX7Z2rVY5KX1Y4hEwSajTZhm2NL3RM/W2WUN+xEK0=; b=Y30xRkvVjwPmE9NEXwBIBhZaboPMm1PIzdSqBKjL1Ownafujd7Qh8O3AqBtZVarqYE 8nzvdqlvFHBcW4SAQKAZ8pmnQ7aKiQE0GVXIi2AgYgjxfJUp3uL4PH/w5cyMj2QeyDOY YIur+pYfMGvo8z0/YsERQxZj6JZv7aKRKbWUC67CPg7smMtGFXhmxGQvoeJesZvh6o5M 32op0xFmaxPQtOriPnhJf3MzBeEKt33PNwUtqEeHInZ6+DYgZ6NxrOP3uuJJu1Ber12y Ic6xLVCAkiLosK1sf4HhM6md/d9aiUKHSJmN9xipFVoNLM2Ix8S8sGuCg3jFJPRZPNEf un7A==
In-reply-to: <51721798.1060501@xxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <CADFMGu+hPV9RanG7298TAYY4p9gMiBOk0+mq5gf5rhQUWXf4TQ@xxxxxxxxxxxxxx> <CADFMGuJYDp-YrPDqsz2KKx6_2RCkP37ZNGPLzdTVOpEgKDMsjA@xxxxxxxxxxxxxx> <51715BD4.8080501@xxxxxxxxxxx> <CADFMGuLjsNBeWE8wTDBgophhpixm3p+wY=9QWwk5u483zL0C4g@xxxxxxxxxxxxxx> <CADFMGuKuL8=B_NY=pKq5gj3aOK0kW0xuPWA=rSCDyziUgWGX6w@xxxxxxxxxxxxxx> <51716DCB.4060407@xxxxxxxxxxx> <CADFMGuJH106wg7zVQrt604DxvDWB_bnor==NEGpJ1Xcr9b+C8A@xxxxxxxxxxxxxx> <CADFMGuLcve0a5uiOzZYoVze8tm1UXTPxhEqForMWYsvCyuh0sg@xxxxxxxxxxxxxx> <5171790C.70400@xxxxxxxxxxx> <CADFMGuKfyw-mCsRn1Y5H5ek+z_nRMHDmW4bG-Ez9ANJm7_ec5A@xxxxxxxxxxxxxx> <CADFMGuL4+vSH9ZpWODXWbHVz9ndMcg2aZY9b0ccq74SJp3XzEw@xxxxxxxxxxxxxx> <CADFMGuK7FEbWibRrctK7B=XXAfAKtpjRej3NVB2k7JXhhYFLLg@xxxxxxxxxxxxxx> <CADFMGuJozkBQdp5o_BK7HbrPdv6iKUie=jHyz5LrtBBvHY1b4w@xxxxxxxxxxxxxx> <CADFMGuL05J+b=bv5jAneLT451eQFNNz2RNHQHccBOjqWsE68Kw@xxxxxxxxxxxxxx> <51720E49.9020001@xxxxxxxxxxx> <CADFMGu+9SGHJEPtUzuR3eObNwfs6VE0TmJNhJh_vxQ3+KBwocA@xxxxxxxxxxxxxx> <51721798.1060501@xxxxxxxxxxx>
Hi Eric,
The xfs module is loaded from system kernel, it happens on our production server too (I did not touch that till now) and if the xfs module is mess up the systemstap may also not working but now it works. As you have mentioned, strange thing is xfs shutdown always happens when glusterfs rebalance completes.


2013/4/20 Eric Sandeen <sandeen@xxxxxxxxxxx>
On 4/19/13 9:03 PM, 符永涛 wrote:
> Hi Eric,
> I will enable them and run test again. I can only reproduce it with
> glusterfs rebalance. Glusterfs uses a mechanism it called syncop to
> unlink file. For rebalance it uses
> syncop_unlink(glusterfs/libglusterfs/src/syncop.c). In the glusterfs
> sync_task framework(glusterfs/libglusterfs/src/syncop.c) it uses
> "makecontext/swapcontext"
> <http://www.opengroup.org/onlinepubs/009695399/functions/makecontext.html>.
> Does it leads to racing unlink from different CPU core?

Yep, I understand that it's rebalance.  It dies when rebalance finishes because an
open but unlinked file trips over the corrupted list from earlier, it seems.

I don't know why makecontext would matter...

Just to be sure, you are definitely loading the xfs module from the kernel you built, right, and you don't have a "priority" module getting loaded from elsewhere?  Seems unlikely, but just to be sure.

> Thank you.

You could also add this patch to the xfs tracepoints to print more information about the inodes - the mode & flags.

-Eric


diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
index e8ce644..c314b87 100644
--- a/fs/xfs/linux-2.6/xfs_trace.h
+++ b/fs/xfs/linux-2.6/xfs_trace.h
@@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,
        TP_STRUCT__entry(
                __field(dev_t, dev)
                __field(xfs_ino_t, ino)
+               __field(__u16, mode)
+               __field(unsigned long, flags)
        ),
        TP_fast_assign(
                __entry->dev = VFS_I(ip)->i_sb->s_dev;
                __entry->ino = ip->i_ino;
+               __entry->mode = VFS_I(ip)->i_mode;
+               __entry->flags = ip->i_flags;
        ),
-       TP_printk("dev %d:%d ino 0x%llx",
+       TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",
                  MAJOR(__entry->dev), MINOR(__entry->dev),
-                 __entry->ino)
+                 __entry->ino, __entry->mode, __entry->flags)
 )

 #define DEFINE_INODE_EVENT(name) \






--
符永涛
<Prev in Thread] Current Thread [Next in Thread>