xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

[符永涛]

Dear Brian and Eric and xfs experts,
Thank you very much for helping to address the issue. With your help I am
now able to isolate this problem, I point the glusterfs volume indices
directory out of xfs filesystem(to a ext4 path) and the shutdown not
happens again. Since this directory is not where glusterfs store data it is
just index directory containing some kind of flag files it's ok to set it
to somewhere else. I'll run more test with this configuration.

Seems glusterfs aggressively call link/remove on the directory and files
under it leads to racing. When I move this directory to a ext4 path, after
rebalance the directory status is as following:

/data/testbug/.glusterfs:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 indices

/data/testbug/.glusterfs/indices:
total 12K
drwxr-xr-x 3 root root 4.0K Apr 20 23:05 .
drwxr-xr-x 3 root root 4.0K Apr 20 22:00 ..
drw------- 2 root root 4.0K Apr 20 23:44 xattrop

/data/testbug/.glusterfs/indices/xattrop:
total 8.0K
drw-------  2 root root 4.0K Apr 20 23:44 .
drwxr-xr-x  3 root root 4.0K Apr 20 23:05 ..
---------- 21 root root    0 Apr 20 23:05
2ea59fab-da86-4ccd-a6d3-20ca80d30e8c
---------- 21 root root    0 Apr 20 23:05
33e96373-7fe1-4c09-969e-f01c45ac445e
---------- 21 root root    0 Apr 20 23:05
35341482-c561-4fdd-b505-61c4e189f63c
---------- 21 root root    0 Apr 20 23:05
390e8676-b18a-4769-9e26-ebc47385d022
---------- 21 root root    0 Apr 20 23:05
4f153df5-0101-4375-bb3d-1c94a2ca5c69
---------- 21 root root    0 Apr 20 23:05
58867dfa-d4b1-47fb-9a54-f3cda0411297
---------- 21 root root    0 Apr 20 23:05
7608e535-6de4-40ba-bb63-aba986d77c6a
---------- 21 root root    0 Apr 20 23:05
7f71f1a0-4463-4fdd-b3b8-0ca73b282520
---------- 21 root root    0 Apr 20 23:05
82cfa2b8-0604-4f7c-bf18-5db47a4cc727
---------- 21 root root    0 Apr 20 23:05
8ea1dc4b-801a-49fb-bff2-7579826f5942
---------- 21 root root    0 Apr 20 23:05
9cc746a7-5b47-4b6a-990f-99f378b787bf
---------- 21 root root    0 Apr 20 23:05
aa1cbfb7-5661-4faf-a3e5-7607a2a2b884
---------- 21 root root    0 Apr 20 23:05
b23c5c1d-1076-43eb-a527-01970d5565ab
---------- 21 root root    0 Apr 20 23:05
b336fb93-82a5-45e8-bacc-66a270a12f3f
---------- 21 root root    0 Apr 20 23:05
bc0ada39-a479-4f29-949c-b8b110291699
---------- 21 root root    0 Apr 20 23:05
c7bd2930-3605-47bc-afc7-67254c8309b0
---------- 21 root root    0 Apr 20 23:05
ce517ec5-079a-41d6-b3f6-6b06cc88ace5
---------- 21 root root    0 Apr 20 23:05
d6d949e2-df27-4428-b4bf-5e589c224fec
---------- 21 root root    0 Apr 20 23:05
fc96f94f-4da0-4003-beca-91738e902c03
---------- 21 root root    0 Apr 20 23:05
fe7a2f27-a98a-4958-9024-85bf8671e612
---------- 21 root root    0 Apr 20 23:05
xattrop-bcb327ae-e265-4cd3-b6d4-9babff815ee7




2013/4/20 Eric Sandeen <sandeen at sandeen.net>

> On 4/20/13 3:10 AM, 符永涛 wrote:
> > Dear Eric,
> > I have applied your latest patch and collected the following log:
> >
>
> If you like, I think you could drop the below patch again; it didn't yield
> anything interesting and just makes for bigger trace logs.
>
> Every mode was 0100000 (S_ISREG / regular file) and flags were always 0.
>
> >         You could also add this patch to the xfs tracepoints to print
> more information about the inodes - the mode & flags.
> >
> >         -Eric
> >
> >
> >         diff --git a/fs/xfs/linux-2.6/xfs_trace.h
> b/fs/xfs/linux-2.6/xfs_trace.h
> >         index e8ce644..c314b87 100644
> >         --- a/fs/xfs/linux-2.6/xfs_trace.h
> >         +++ b/fs/xfs/linux-2.6/xfs_trace.h
> >         @@ -544,14 +544,18 @@ DECLARE_EVENT_CLASS(xfs_inode_class,
> >                 TP_STRUCT__entry(
> >                         __field(dev_t, dev)
> >                         __field(xfs_ino_t, ino)
> >         +               __field(__u16, mode)
> >         +               __field(unsigned long, flags)
> >                 ),
> >                 TP_fast_assign(
> >                         __entry->dev = VFS_I(ip)->i_sb->s_dev;
> >                         __entry->ino = ip->i_ino;
> >         +               __entry->mode = VFS_I(ip)->i_mode;
> >         +               __entry->flags = ip->i_flags;
> >                 ),
> >         -       TP_printk("dev %d:%d ino 0x%llx",
> >         +       TP_printk("dev %d:%d ino 0x%llx mode 0%o, flags 0x%lx",
> >                           MAJOR(__entry->dev), MINOR(__entry->dev),
> >         -                 __entry->ino)
> >         +                 __entry->ino, __entry->mode, __entry->flags)
> >          )
> >
> >          #define DEFINE_INODE_EVENT(name) \
> >
> >
> >
> >
> >
> >
> >     --
> >     符永涛
> >
> >
> >
> >
> > --
> > 符永涛
>
>


-- 
符永涛
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20130421/04f0fa89/attachment.html>