Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

[符永涛 <yongtaofu@xxxxxxxxx>]

Dear Brian and Eric and xfs experts,

Thank you very much for helping to address the issue. With your help I am now able to isolate this problem, I point the glusterfs volume indices directory out of xfs filesystem(to a ext4 path) and the shutdown not happens again. Since this directory is not where glusterfs store data it is just index directory containing some kind of flag files it's ok to set it to somewhere else. I'll run more test with this configuration.

Seems glusterfs aggressively call link/remove on the directory and files under it leads to racing. When I move this directory to a ext4 path, after rebalance the directory status is as following:

/data/testbug/.glusterfs:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 indices

/data/testbug/.glusterfs/indices:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drw------- 2 root root 4.0K Apr 20 23:44 xattrop

/data/testbug/.glusterfs/indices/xattrop:
total 8.0K
drw-------  2 root root 4.0K Apr 20 23:44 .
drwxr-xr-x  3 root root 4.0K Apr 20 23:05 ..
---------- 21 root root    0 Apr 20 23:05 2ea59fab-da86-4ccd-a6d3-20ca80d30e8c
---------- 21 root root    0 Apr 20 23:05 33e96373-7fe1-4c09-969e-f01c45ac445e
---------- 21 root root    0 Apr 20 23:05 35341482-c561-4fdd-b505-61c4e189f63c
---------- 21 root root    0 Apr 20 23:05 390e8676-b18a-4769-9e26-ebc47385d022
---------- 21 root root    0 Apr 20 23:05 4f153df5-0101-4375-bb3d-1c94a2ca5c69
---------- 21 root root    0 Apr 20 23:05 58867dfa-d4b1-47fb-9a54-f3cda0411297
---------- 21 root root    0 Apr 20 23:05 7608e535-6de4-40ba-bb63-aba986d77c6a
---------- 21 root root    0 Apr 20 23:05 7f71f1a0-4463-4fdd-b3b8-0ca73b282520
---------- 21 root root    0 Apr 20 23:05 82cfa2b8-0604-4f7c-bf18-5db47a4cc727
---------- 21 root root    0 Apr 20 23:05 8ea1dc4b-801a-49fb-bff2-7579826f5942
---------- 21 root root    0 Apr 20 23:05 9cc746a7-5b47-4b6a-990f-99f378b787bf
---------- 21 root root    0 Apr 20 23:05 aa1cbfb7-5661-4faf-a3e5-7607a2a2b884
---------- 21 root root    0 Apr 20 23:05 b23c5c1d-1076-43eb-a527-01970d5565ab
---------- 21 root root    0 Apr 20 23:05 b336fb93-82a5-45e8-bacc-66a270a12f3f
---------- 21 root root    0 Apr 20 23:05 bc0ada39-a479-4f29-949c-b8b110291699
---------- 21 root root    0 Apr 20 23:05 c7bd2930-3605-47bc-afc7-67254c8309b0
---------- 21 root root    0 Apr 20 23:05 ce517ec5-079a-41d6-b3f6-6b06cc88ace5
---------- 21 root root    0 Apr 20 23:05 d6d949e2-df27-4428-b4bf-5e589c224fec
---------- 21 root root    0 Apr 20 23:05 fc96f94f-4da0-4003-beca-91738e902c03
---------- 21 root root    0 Apr 20 23:05 fe7a2f27-a98a-4958-9024-85bf8671e612
---------- 21 root root    0 Apr 20 23:05 xattrop-bcb327ae-e265-4cd3-b6d4-9babff815ee7

2013/4/20 Eric Sandeen <sandeen@xxxxxxxxxxx>

On 4/20/13 3:10 AM, 符永涛 wrote:
> Dear Eric,
> I have applied your latest patch and collected the following log:
>

If you like, I think you could drop the below patch again; it didn't yield anything interesting and just makes for bigger trace logs.

Every mode was 0100000 (S_ISREG / regular file) and flags were always 0.

>         You could also add this patch to the xfs tracepoints to print more information about the inodes - the mode & flags.
>
>         -Eric
>
>
>         diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
>         index e8ce644..c314b87 100644
>         --- a/fs/xfs/linux-2.6/xfs_trace.h
>         +++ b/fs/xfs/linux-2.6/xfs_trace.h
>         @@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,
>                 TP_STRUCT__entry(
>                         __field(dev_t, dev)
>                         __field(xfs_ino_t, ino)
>         +               __field(__u16, mode)
>         +               __field(unsigned long, flags)
>                 ),
>                 TP_fast_assign(
>                         __entry->dev = VFS_I(ip)->i_sb->s_dev;
>                         __entry->ino = ip->i_ino;
>         +               __entry->mode = VFS_I(ip)->i_mode;
>         +               __entry->flags = ip->i_flags;
>                 ),
>         -       TP_printk("dev %d:%d ino 0x%llx",
>         +       TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",
>                           MAJOR(__entry->dev), MINOR(__entry->dev),
>         -                 __entry->ino)
>         +                 __entry->ino, __entry->mode, __entry->flags)
>          )
>
>          #define DEFINE_INODE_EVENT(name) \
>
>
>
>
>
>
>     --
>     符永涛
>
>
>
>
> --
> 符永涛

--
符永涛