xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: 符永涛 <yongtaofu@xxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: Eric Sandeen <sandeen@xxxxxxxxxxx>
Date: Thu, 18 Apr 2013 10:03:22 -0700
Cc: Brian Foster <bfoster@xxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CADFMGuJ-An9MMmYtOKEjt5JdHmvu-cc0G+y361e_fioYf4j7HQ@xxxxxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <20130416162417.GC13938@destitution> <516D87E3.2010605@xxxxxxxxxx> <CADFMGuLgU2++ym4U9axhevV_L2t-2CMVaDoT5cyQcew2YZMsBA@xxxxxxxxxxxxxx> <CADFMGuK3cPSmqZoMdFSSqOnUz3GLv+c01xXHDhzv9_dfUNUPPA@xxxxxxxxxxxxxx> <CADFMGu+XD=-NJm3CAsnKpiMWykPj67t2LHgtxkPpyTpVmG3LDA@xxxxxxxxxxxxxx> <CADFMGu+z32O5mVBfO6gb8F03y9mrwB5=nf+7LMbky4ttk2mbFw@xxxxxxxxxxxxxx> <BA50C18A-BE51-4658-B05E-3EDB0F11BE36@xxxxxxxxxxx> <CADFMGuKfXwY=Z9dg6+wyX7mKzppu1hQhsqxcr9Fd=DjP4hii9w@xxxxxxxxxxxxxx> <CADFMGuLDwde8te0Cs4-T9fdVmq2v22fmSTgkYbULHS3voHzw6A@xxxxxxxxxxxxxx> <CADFMGuKmmJCz+FxsFL9KegqzNhhHsG2ZbVjrwCDyNCh1R_Zgpw@xxxxxxxxxxxxxx> <516FDBDE.1070406@xxxxxxxxxx> <CADFMGuJ-An9MMmYtOKEjt5JdHmvu-cc0G+y361e_fioYf4j7HQ@xxxxxxxxxxxxxx>
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5
On 4/18/13 8:23 AM, 符永涛 wrote:
> Hi Brian and Eric,
> The shutdown is not easy to produce but finally right now 2 of our servers in 
> our test cluster xfs was shutdown.
> 
> the trace output as following
> https://docs.google.com/file/d/0B7n2C4T5tfNCLXRYUWJ0b19JcWc/edit?usp=sharing
> 
> Sorry but the systemtap is interrupt and I didn't noticed that so I didn't 
> get systemtap logs.
> 
> /var/log/message is same as before
> Apr 18 22:43:14 10 kernel: XFS (sdb):         : xfs_inotobp() returned error 
> 22.
> Apr 18 22:43:14 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 
> 22
> Apr 18 22:43:14 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from 
> line 1184 of file fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02d44aa
> Apr 18 22:43:14 10 kernel: XFS (sdb): I/O Error Detected. Shutting down 
> filesystem
> Apr 18 22:43:14 10 kernel: XFS (sdb): Please umount the filesystem and 
> rectify the problem(s)
> Apr 18 22:43:20 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 
> The metadump file is large I'll share it to you soon.
> 

Thanks, we'll take a look.  Just to double check, in the kernel that ran the 
tracepoints, did you use brian's 2nd version of the patch?  I want to make sure 
the tracepoints were at the top of the function.

Since you're patching xfs anyway, can you add something like this for next time:

diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 796edce..cad0e8e 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -1777,8 +1777,9 @@ xfs_iunlink_remove(
                                            &last_ibp, &last_offset, 0);
                        if (error) {
                                xfs_warn(mp,
-                                       "%s: xfs_inotobp() returned error %d.",
-                                       __func__, error);
+                                       "%s: xfs_inotobp() returned error %d "
+                                       "for inode 0x%llx ag %d agino %x\n",
+                                       __func__, error, ip->i_ino, agno, 
agino);
                                return error;
                        }
                        next_agino = be32_to_cpu(last_dip->di_next_unlinked);

so that when we encounter the error we're sure to have the problematic inode 
number.

Thanks,
-Eric

<Prev in Thread] Current Thread [Next in Thread>