xfs
[Top] [All Lists]

Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging

To: Brian Foster <bfoster@xxxxxxxxxx>
Subject: Re: xfs_iunlink_remove: xfs_inotobp() returned error 22 -- debugging
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Wed, 17 Apr 2013 09:35:55 +0800
Cc: Dave Chinner <david@xxxxxxxxxxxxx>, Eric Sandeen <sandeen@xxxxxxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=34KVwZHmm2Wm8CQCJwuicEnRbRUZ+A+5R2pFXrt2Pms=; b=ir6oO3NK/DUzyiJlVmpPtfP3JNwZY97aUy5FCQimyZflQMiPH5tF0cy8AAl25sx216 qWVphdkfNPQjNYlJR8NH3hJrKiMoTe24hE6Srw7l0ElLX+6ZF/bFBze9j80RUsg7FyV/ cEQxGO7AZcbgTY8+ZMrAI+GiaaHLnynwGM/mnhBfk9IAawQmMBN04ZoV1dy4ONcoc8mL dcKRATJm11Uaz9u0mx38QYTz44UBgTv+dfb1aUEtyq7G9dBVRJDhWmZu03+xleSB6794 qcofjzEEj2XSKQ6mt6NXKjxY/+9mGBMU65LKazobcx2KZ0YVQpI9uySspLWAiQJTr4QZ xulQ==
In-reply-to: <CADFMGuLgU2++ym4U9axhevV_L2t-2CMVaDoT5cyQcew2YZMsBA@xxxxxxxxxxxxxx>
References: <516C89DF.4070904@xxxxxxxxxx> <20130416162417.GC13938@destitution> <516D87E3.2010605@xxxxxxxxxx> <CADFMGuLgU2++ym4U9axhevV_L2t-2CMVaDoT5cyQcew2YZMsBA@xxxxxxxxxxxxxx>
Hi Brain,
I want to ask a question, according to the shutdown trace. The ino in  xfs_iunlink_remove is 0x113, why xfs_imap got ino=0xffffffff ? 

--- xfs_imap -- module("xfs").function("xfs_imap@fs/xfs/xfs_ialloc.c:1257").return -- return=0x16
vars: mp=0xffff882017a50800 tp=0xffff881c81797c70 ino=0xffffffff

--- xfs_iunlink_remove -- module("xfs").function("xfs_iunlink_remove@fs/xfs/xfs_inode.c:1680").return -- return=0x16
vars: tp=0xffff881c81797c70 ip=0xffff881003c13c00 next_ino=? mp=? agi=? dip=? agibp=0xffff880109b47e20 ibp=? agno=? agino=? next_agino=? last_ibp=? last_dip=0xffff882000000000 bucket_index=? offset=? last_offset=0xffffffffffff8810 error=? __func__=[...]
ip: i_ino = 0x113, i_flags = 0x0

Thank you.



2013/4/17 符永涛 <yongtaofu@xxxxxxxxx>
Hi Brain,
Thank you for your update, and I have applied your last kernel patch. However it is not easy to reproduce especially in out test environment. Till now is not happens again. I'll update the kernel patch now. BTW is there any findings in the logs of previous thread?
http://oss.sgi.com/archives/xfs/2013-04/msg00327.html
I guess it tend to happen during glusterfs rebalance because glusterfs moves a lot of file from one server to another and then unlink it.

Thank you.


2013/4/17 Brian Foster <bfoster@xxxxxxxxxx>
On 04/16/2013 12:24 PM, Dave Chinner wrote:
> On Mon, Apr 15, 2013 at 07:14:39PM -0400, Brian Foster wrote:
>> Hi,
>>
>> Thanks for the data in the previous thread:
>>
>> http://oss.sgi.com/archives/xfs/2013-04/msg00327.html
>>
...
>>
>>      echo 1 > /sys/kernel/debug/tracing/events/xfs/xfs_iunlink/enable
>>      echo 1 > /sys/kernel/debug/tracing/events/xfs/xfs_iunlink_remove/enable
>>      ... reproduce ...
>>      cat /sys/kernel/debug/tracing/trace > trace.output
>
> It's better to use trace-cmd for this. it will result in less
> dropped events. i.e.:
>
>       $ trace-cmd record -e xfs_iunlink\*
>       ... reproduce ...
>       ^C
>       $ trace-cmd report > trace.output
>
>> --- a/fs/xfs/linux-2.6/xfs_trace.h
>> +++ b/fs/xfs/linux-2.6/xfs_trace.h
>> @@ -581,6 +581,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);
...
>
> I would suggest that the the tracing shoul dbe at entry of the
> function, otherwise we won't get a tracepoint for the operation that
> triggers the shutdown. (That's the reason most tracepoints in XFS
> are at function entry...)
>

Good points, thanks Dave. A v2 that pulls up the tracepoints towards
function entry is appended.

Brian

>From 280943e78ebe0b97a774cba51e7815c42f044b55 Mon Sep 17 00:00:00 2001
From: Brian Foster <bfoster@xxxxxxxxxx>
Date: Mon, 15 Apr 2013 18:16:24 -0400
Subject: [PATCH v2] xfs: add tracepoints for xfs_iunlink and
xfs_iunlink_remove

---
 fs/xfs/linux-2.6/xfs_trace.h |    2 ++
 fs/xfs/xfs_inode.c           |    4 ++++
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/fs/xfs/linux-2.6/xfs_trace.h b/fs/xfs/linux-2.6/xfs_trace.h
index adc6ec4..338a0f9 100644
--- a/fs/xfs/linux-2.6/xfs_trace.h
+++ b/fs/xfs/linux-2.6/xfs_trace.h
@@ -583,6 +583,8 @@ DEFINE_INODE_EVENT(xfs_file_fsync);
 DEFINE_INODE_EVENT(xfs_destroy_inode);
 DEFINE_INODE_EVENT(xfs_dirty_inode);
 DEFINE_INODE_EVENT(xfs_clear_inode);
+DEFINE_INODE_EVENT(xfs_iunlink);
+DEFINE_INODE_EVENT(xfs_iunlink_remove);

 DEFINE_INODE_EVENT(xfs_dquot_dqalloc);
 DEFINE_INODE_EVENT(xfs_dquot_dqdetach);
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 19900f0..d705c77 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -1615,6 +1615,8 @@ xfs_iunlink(

        mp = tp->t_mountp;

+       trace_xfs_iunlink(ip);
+
        /*
         * Get the agi buffer first.  It ensures lock ordering
         * on the list.
@@ -1694,6 +1696,8 @@ xfs_iunlink_remove(
        mp = tp->t_mountp;
        agno = XFS_INO_TO_AGNO(mp, ip->i_ino);

+       trace_xfs_iunlink_remove(ip);
+
        /*
         * Get the agi buffer first.  It ensures lock ordering
         * on the list.
--
1.7.7.6




--
符永涛



--
符永涛
<Prev in Thread] Current Thread [Next in Thread>