xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: Eric Sandeen <sandeen@xxxxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Fri, 19 Apr 2013 23:18:52 +0800
Cc: Brian Foster <bfoster@xxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=iVG8ymN49ukcPyXOID/LGvgXj/+iLzYjsDB5IXrieIg=; b=dHzger0RAnjy2h/lPRVfv8bNVGWpv2m6+vEAnUjWgm1fEst7G0JDCfTiOTMgCQ1N9/ Wu6VFo8di9VsDODBGNq1tF557y/dE/qtRBpfWl+vQ6CFd6pJmOpHkd1V5QQ2+3Ozpdco /VyQ1PN+9JYEruHG1x6bVxld+CHK0ecOBS4RVw4zybnNpW/ykOlS+/FW8vVmOp6KAiT9 e1UVyTB0DmcLM6nhF5qOBYOnDH46BA1RDwbFJpiienazLUXiBMGrdWicYOEAqrJwnTo2 kS9V4CuWnF8eWrtyxpl+gWgUqB6N+vmnNs5KnLS3tdPrYTowKSky1rb20mjhDFIdEaTp j/fA==
In-reply-to: <CADFMGuLjsNBeWE8wTDBgophhpixm3p+wY=9QWwk5u483zL0C4g@xxxxxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <20130416162417.GC13938@destitution> <516D87E3.2010605@xxxxxxxxxx> <CADFMGuLgU2++ym4U9axhevV_L2t-2CMVaDoT5cyQcew2YZMsBA@xxxxxxxxxxxxxx> <CADFMGuK3cPSmqZoMdFSSqOnUz3GLv+c01xXHDhzv9_dfUNUPPA@xxxxxxxxxxxxxx> <CADFMGu+XD=-NJm3CAsnKpiMWykPj67t2LHgtxkPpyTpVmG3LDA@xxxxxxxxxxxxxx> <CADFMGu+z32O5mVBfO6gb8F03y9mrwB5=nf+7LMbky4ttk2mbFw@xxxxxxxxxxxxxx> <BA50C18A-BE51-4658-B05E-3EDB0F11BE36@xxxxxxxxxxx> <CADFMGuKfXwY=Z9dg6+wyX7mKzppu1hQhsqxcr9Fd=DjP4hii9w@xxxxxxxxxxxxxx> <CADFMGuLDwde8te0Cs4-T9fdVmq2v22fmSTgkYbULHS3voHzw6A@xxxxxxxxxxxxxx> <CADFMGuKmmJCz+FxsFL9KegqzNhhHsG2ZbVjrwCDyNCh1R_Zgpw@xxxxxxxxxxxxxx> <516FDBDE.1070406@xxxxxxxxxx> <CADFMGuJ-An9MMmYtOKEjt5JdHmvu-cc0G+y361e_fioYf4j7HQ@xxxxxxxxxxxxxx> <51705EC4.4000306@xxxxxxxxxx> <CADFMGu+hPV9RanG7298TAYY4p9gMiBOk0+mq5gf5rhQUWXf4TQ@xxxxxxxxxxxxxx> <CADFMGuJYDp-YrPDqsz2KKx6_2RCkP37ZNGPLzdTVOpEgKDMsjA@xxxxxxxxxxxxxx> <51715BD4.8080501@xxxxxxxxxxx> <CADFMGuLjsNBeWE8wTDBgophhpixm3p+wY=9QWwk5u483zL0C4g@xxxxxxxxxxxxxx>
Dear Eric,
If it's racing issue where the lock is introduced? I want to study the code from you. Thank you.


2013/4/19 符永涛 <yongtaofu@xxxxxxxxx>
Sure the serious thing here is that it corrupt the unlinked list. The inode 0x1bd33 which trigger xfs shutdown is not  0x6b133.


2013/4/19 Eric Sandeen <sandeen@xxxxxxxxxxx>
On 4/19/13 4:41 AM, 符永涛 wrote:
> Dear Brian and Eric,
>
> kernel kernel-2.6.32-279.19.1.el6.x86_64.rpm <http://mirror.linux.duke.edu/pub/centos/6.3/updates/x86_64/Packages/kernel-2.6.32-279.19.1.el6.x86_64.rpm> still have this problem
> I build the kernel from this srpm
> https://oss.oracle.com/ol6/SRPMS-updates/kernel-2.6.32-279.19.1.el6.src.rpm
>
> today the shutdown happens again during test.
> Seelogs bellow:
>
> /var/log/message
> Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22.
> Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22
> Apr 19 16:40:05 10 kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02d4bda
> Apr 19 16:40:05 10 kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem
> Apr 19 16:40:05 10 kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s)
> Apr 19 16:40:07 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Apr 19 16:40:37 10 kernel: XFS (sdb): xfs_log_force: error 5 returned.
>
> systemtap script output:
> --- xfs_imap -- module("xfs").function("xfs_imap@fs/xfs/xfs_ialloc.c:1257").return -- return=0x16
> vars: mp=0xffff88101801e800 tp=0xffff880ff143ac70 ino=0xffffffff imap=0xffff88100e93bc08 flags=0x0 agbno=? agino=? agno=? blks_per_cluster=? chunk_agbno=? cluster_agbno=? error=? offset=? offset_agbno=? __func__=[...]
> mp: m_agno_log = 0x5, m_agino_log = 0x20
> mp->m_sb: sb_agcount = 0x1c, sb_agblocks = 0xffffff0, sb_inopblog = 0x4, sb_agblklog = 0x1c, sb_dblocks = 0x1b4900000
> imap: im_blkno = 0x0, im_len = 0xe778, im_boffset = 0xd997
> kernel backtrace:
> Returning from:  0xffffffffa02b4260 : xfs_imap+0x0/0x280 [xfs]
> Returning to  :  0xffffffffa02b9d59 : xfs_inotobp+0x49/0xc0 [xfs]
>  0xffffffffa02b9ec1 : xfs_iunlink_remove+0xf1/0x360 [xfs]
>  0xffffffff814ede89
>  0x0 (inexact)
> user backtrace:
>  0x3ec260e5ad [/lib64/libpthread-2.12.so <http://libpthread-2.12.so>+0xe5ad/0x219000]
>
> --- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1681").return -- return=0x16
> vars: tp=0xffff880ff143ac70 ip=0xffff8811ed111000 next_ino=? mp=? agi=? dip=? agibp=? ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff881000000001 bucket_index=? offset=? last_offset=0xffffffffffff8811 error=? __func__=[...]
> ip: i_ino = 0x1bd33, i_flags = 0x0
> ip->i_d: di_nlink = 0x0, di_gen = 0x53068791
>
> debugfs events trace:
> https://docs.google.com/file/d/0B7n2C4T5tfNCREZtdC1yamc0RnM/edit?usp=sharing

Same issue, one file was unlinked twice in a race:

=== ino 0x6b133 ===
           <...>-4477  [003]  2721.176790: xfs_iunlink: dev 8:16 ino 0x6b133
           <...>-4477  [003]  2721.176839: xfs_iunlink_remove: dev 8:16 ino 0x6b133
           <...>-4477  [009]  3320.127227: xfs_iunlink: dev 8:16 ino 0x6b133
           <...>-4477  [001]  3320.141126: xfs_iunlink_remove: dev 8:16 ino 0x6b133
           <...>-4477  [003]  7973.136368: xfs_iunlink: dev 8:16 ino 0x6b133
           <...>-4479  [018]  7973.158457: xfs_iunlink: dev 8:16 ino 0x6b133
           <...>-4479  [018]  7973.158497: xfs_iunlink_remove: dev 8:16 ino 0x6b133

-Eric




--
符永涛



--
符永涛
<Prev in Thread] Current Thread [Next in Thread>