xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: 符永涛 <yongtaofu@xxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: Eric Sandeen <sandeen@xxxxxxxxxxx>
Date: Thu, 18 Apr 2013 11:35:49 -0700
Cc: Brian Foster <bfoster@xxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CADFMGuJ-An9MMmYtOKEjt5JdHmvu-cc0G+y361e_fioYf4j7HQ@xxxxxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <20130416162417.GC13938@destitution> <516D87E3.2010605@xxxxxxxxxx> <CADFMGuLgU2++ym4U9axhevV_L2t-2CMVaDoT5cyQcew2YZMsBA@xxxxxxxxxxxxxx> <CADFMGuK3cPSmqZoMdFSSqOnUz3GLv+c01xXHDhzv9_dfUNUPPA@xxxxxxxxxxxxxx> <CADFMGu+XD=-NJm3CAsnKpiMWykPj67t2LHgtxkPpyTpVmG3LDA@xxxxxxxxxxxxxx> <CADFMGu+z32O5mVBfO6gb8F03y9mrwB5=nf+7LMbky4ttk2mbFw@xxxxxxxxxxxxxx> <BA50C18A-BE51-4658-B05E-3EDB0F11BE36@xxxxxxxxxxx> <CADFMGuKfXwY=Z9dg6+wyX7mKzppu1hQhsqxcr9Fd=DjP4hii9w@xxxxxxxxxxxxxx> <CADFMGuLDwde8te0Cs4-T9fdVmq2v22fmSTgkYbULHS3voHzw6A@xxxxxxxxxxxxxx> <CADFMGuKmmJCz+FxsFL9KegqzNhhHsG2ZbVjrwCDyNCh1R_Zgpw@xxxxxxxxxxxxxx> <516FDBDE.1070406@xxxxxxxxxx> <CADFMGuJ-An9MMmYtOKEjt5JdHmvu-cc0G+y361e_fioYf4j7HQ@xxxxxxxxxxxxxx>
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5
On 4/18/13 8:23 AM, 符永涛 wrote:
> Hi Brian and Eric,
> The shutdown is not easy to produce but finally right now 2 of our servers in 
> our test cluster xfs was shutdown.
> 
> the trace output as following
> https://docs.google.com/file/d/0B7n2C4T5tfNCLXRYUWJ0b19JcWc/edit?usp=sharing
> 

here's something interesting, for 2 inodes we have double/racing calls to 
xfs_iunlink:

=== 0x5cc0b ===
           <...>-8336  [004]  6931.372924: xfs_iunlink: dev 8:16 ino 0x5cc0b
           <...>-8336  [004]  6931.372965: xfs_iunlink_remove: dev 8:16 ino 
0x5cc0b
           <...>-27541 [001] 35061.349747: xfs_iunlink: dev 8:16 ino 0x5cc0b
           <...>-3356  [001] 36449.762504: xfs_iunlink_remove: dev 8:16 ino 
0x5cc0b
           <...>-3300  [003] 41013.398566: xfs_iunlink: dev 8:16 ino 0x5cc0b
           <...>-26115 [012] 41013.399884: xfs_iunlink: dev 8:16 ino 0x5cc0b
           <...>-26115 [012] 41013.399935: xfs_iunlink_remove: dev 8:16 ino 
0x5cc0b
           <...>-28961 [000] 68977.951208: xfs_iunlink: dev 8:16 ino 0x5cc0b
           <...>-3364  [021] 81616.210533: xfs_iunlink_remove: dev 8:16 ino 
0x5cc0b

=== 0x7ef8c ===
           <...>-13169 [001] 118751.536025: xfs_iunlink: dev 8:16 ino 0x7ef8c
           <...>-13169 [001] 118751.536049: xfs_iunlink_remove: dev 8:16 ino 
0x7ef8c
           <...>-3594  [015] 119027.006161: xfs_iunlink: dev 8:16 ino 0x7ef8c
           <...>-3594  [015] 119027.006186: xfs_iunlink_remove: dev 8:16 ino 
0x7ef8c
           <...>-3591  [001] 121423.286004: xfs_iunlink: dev 8:16 ino 0x7ef8c
           <...>-4141  [019] 121423.288518: xfs_iunlink: dev 8:16 ino 0x7ef8c
           <...>-4141  [019] 121423.288541: xfs_iunlink_remove: dev 8:16 ino 
0x7ef8c

2 threads on 2 different CPUs adding the same inode to the unlinked list in a 
race;
this will corrupt the list and lead to the failure to find the other inode we're
looking for.  So, progress!  We'll take a look at the iunlink paths.

-Eric



<Prev in Thread] Current Thread [Next in Thread>