xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: Eric Sandeen <sandeen@xxxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>, Brian Foster <bfoster@xxxxxxxxxx>, Dave Chinner <david@xxxxxxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Sun, 21 Apr 2013 07:52:17 +0800
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=ShHiYrf3rymu0fE/ulwL66CENXPuBsETNkAi8WLupA4=; b=YGYljgNbxJy7eww5B+fBMbIOW8TBacgAx11H0BOoAtfN+obPd6E+G71PyysNhSIUCZ cAQYaaCdOe4DZh35bE57wv+UVTkhickDCXiILUkf25UkS2Pe08WHM16hphmXCbcwm89D bAvmeSHbQcBFAE7WVycrWnKPBs7kGALF+7KyBvRERvU9adH53U+yFEaiSA4ZA9574hg4 lgqiT5trVPrMN9LS6cCkkllXYiw+LnDQ1XDEqAcAu8xPuMIavFf5kZdR9DWkcZSJfb3H 7jqZaZ60mU+kmpy4x1+joyr+9JDjayguTkV2c080wQtrkkX3pBrngdxNgmKCzHZ24cLb Fp0w==
In-reply-to: <5172B73C.6000900@xxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <CADFMGuLjsNBeWE8wTDBgophhpixm3p+wY=9QWwk5u483zL0C4g@xxxxxxxxxxxxxx> <CADFMGuKuL8=B_NY=pKq5gj3aOK0kW0xuPWA=rSCDyziUgWGX6w@xxxxxxxxxxxxxx> <51716DCB.4060407@xxxxxxxxxxx> <CADFMGuJH106wg7zVQrt604DxvDWB_bnor==NEGpJ1Xcr9b+C8A@xxxxxxxxxxxxxx> <CADFMGuLcve0a5uiOzZYoVze8tm1UXTPxhEqForMWYsvCyuh0sg@xxxxxxxxxxxxxx> <5171790C.70400@xxxxxxxxxxx> <CADFMGuKfyw-mCsRn1Y5H5ek+z_nRMHDmW4bG-Ez9ANJm7_ec5A@xxxxxxxxxxxxxx> <CADFMGuL4+vSH9ZpWODXWbHVz9ndMcg2aZY9b0ccq74SJp3XzEw@xxxxxxxxxxxxxx> <CADFMGuK7FEbWibRrctK7B=XXAfAKtpjRej3NVB2k7JXhhYFLLg@xxxxxxxxxxxxxx> <CADFMGuJozkBQdp5o_BK7HbrPdv6iKUie=jHyz5LrtBBvHY1b4w@xxxxxxxxxxxxxx> <CADFMGuL05J+b=bv5jAneLT451eQFNNz2RNHQHccBOjqWsE68Kw@xxxxxxxxxxxxxx> <51720E49.9020001@xxxxxxxxxxx> <CADFMGu+9SGHJEPtUzuR3eObNwfs6VE0TmJNhJh_vxQ3+KBwocA@xxxxxxxxxxxxxx> <51721798.1060501@xxxxxxxxxxx> <CADFMGuJAXcA0CTYEoaRNvsTv7DhHeOBgk+j_HiMceT1neoEzRQ@xxxxxxxxxxxxxx> <CADFMGuLBat2fWbtpRd=Sb0bQpZiTOdF2p8NA97rak8A=+GhN1A@xxxxxxxxxxxxxx> <5172B73C.6000900@xxxxxxxxxxx>
Dear Brian and Eric and xfs experts,
Thank you very much for helping to address the issue. With your help I am now able to isolate this problem, I point the glusterfs volume indices directory out of xfs filesystem(to a ext4 path) and the shutdown not happens again. Since this directory is not where glusterfs store data it is just index directory containing some kind of flag files it's ok to set it to somewhere else. I'll run more test with this configuration.

Seems glusterfs aggressively call link/remove on the directory and files under it leads to racing. When I move this directory to a ext4 path, after rebalance the directory status is as following:

/data/testbug/.glusterfs:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 indices

/data/testbug/.glusterfs/indices:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drw------- 2 root root 4.0K Apr 20 23:44 xattrop

/data/testbug/.glusterfs/indices/xattrop:
total 8.0K
drw-------  2 root root 4.0K Apr 20 23:44 .
drwxr-xr-x  3 root root 4.0K Apr 20 23:05 ..
---------- 21 root root    0 Apr 20 23:05 2ea59fab-da86-4ccd-a6d3-20ca80d30e8c
---------- 21 root root    0 Apr 20 23:05 33e96373-7fe1-4c09-969e-f01c45ac445e
---------- 21 root root    0 Apr 20 23:05 35341482-c561-4fdd-b505-61c4e189f63c
---------- 21 root root    0 Apr 20 23:05 390e8676-b18a-4769-9e26-ebc47385d022
---------- 21 root root    0 Apr 20 23:05 4f153df5-0101-4375-bb3d-1c94a2ca5c69
---------- 21 root root    0 Apr 20 23:05 58867dfa-d4b1-47fb-9a54-f3cda0411297
---------- 21 root root    0 Apr 20 23:05 7608e535-6de4-40ba-bb63-aba986d77c6a
---------- 21 root root    0 Apr 20 23:05 7f71f1a0-4463-4fdd-b3b8-0ca73b282520
---------- 21 root root    0 Apr 20 23:05 82cfa2b8-0604-4f7c-bf18-5db47a4cc727
---------- 21 root root    0 Apr 20 23:05 8ea1dc4b-801a-49fb-bff2-7579826f5942
---------- 21 root root    0 Apr 20 23:05 9cc746a7-5b47-4b6a-990f-99f378b787bf
---------- 21 root root    0 Apr 20 23:05 aa1cbfb7-5661-4faf-a3e5-7607a2a2b884
---------- 21 root root    0 Apr 20 23:05 b23c5c1d-1076-43eb-a527-01970d5565ab
---------- 21 root root    0 Apr 20 23:05 b336fb93-82a5-45e8-bacc-66a270a12f3f
---------- 21 root root    0 Apr 20 23:05 bc0ada39-a479-4f29-949c-b8b110291699
---------- 21 root root    0 Apr 20 23:05 c7bd2930-3605-47bc-afc7-67254c8309b0
---------- 21 root root    0 Apr 20 23:05 ce517ec5-079a-41d6-b3f6-6b06cc88ace5
---------- 21 root root    0 Apr 20 23:05 d6d949e2-df27-4428-b4bf-5e589c224fec
---------- 21 root root    0 Apr 20 23:05 fc96f94f-4da0-4003-beca-91738e902c03
---------- 21 root root    0 Apr 20 23:05 fe7a2f27-a98a-4958-9024-85bf8671e612
---------- 21 root root    0 Apr 20 23:05 xattrop-bcb327ae-e265-4cd3-b6d4-9babff815ee7




2013/4/20 Eric Sandeen <sandeen@xxxxxxxxxxx>
On 4/20/13 3:10 AM, 符永涛 wrote:
> Dear Eric,
> I have applied your latest patch and collected the following log:
>

If you like, I think you could drop the below patch again; it didn't yield anything interesting and just makes for bigger trace logs.

Every mode was 0100000 (S_ISREG / regular file) and flags were always 0.

>         You could also add this patch to the xfs tracepoints to print more information about the inodes - the mode & flags.
>
>         -Eric
>
>
>         diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
>         index e8ce644..c314b87 100644
>         --- a/fs/xfs/linux-2.6/xfs_trace.h
>         +++ b/fs/xfs/linux-2.6/xfs_trace.h
>         @@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,
>                 TP_STRUCT__entry(
>                         __field(dev_t, dev)
>                         __field(xfs_ino_t, ino)
>         +               __field(__u16, mode)
>         +               __field(unsigned long, flags)
>                 ),
>                 TP_fast_assign(
>                         __entry->dev = VFS_I(ip)->i_sb->s_dev;
>                         __entry->ino = ip->i_ino;
>         +               __entry->mode = VFS_I(ip)->i_mode;
>         +               __entry->flags = ip->i_flags;
>                 ),
>         -       TP_printk("dev %d:%d ino 0x%llx",
>         +       TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",
>                           MAJOR(__entry->dev), MINOR(__entry->dev),
>         -                 __entry->ino)
>         +                 __entry->ino, __entry->mode, __entry->flags)
>          )
>
>          #define DEFINE_INODE_EVENT(name) \
>
>
>
>
>
>
>     --
>     符永涛
>
>
>
>
> --
> 符永涛




--
符永涛
<Prev in Thread] Current Thread [Next in Thread>