xfs
[Top] [All Lists]

Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inoto

To: Ben Myers <bpm@xxxxxxx>
Subject: Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Tue, 9 Apr 2013 23:00:40 +0800
Cc: xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=sOuPeKX36X9b3PhW2QnIVNpjDYsMWFGZ78iWZugWS1c=; b=nlv9gibUjRBqrSef8J4kU+ewRXzeNHBUMrMftjS+VgrJATlP8ECWEnWHsEX1LvOeN4 NA7CSzlhLBWJgQ1LarTkcGD8r8KNkYJ76GBk1FSC1z5JctXep9gDyWOqDvzSojYSbzhm ck1DFJm+cGr8g+oeS//G4WYKP2yCOaKj445ex3nOckD4sD5Pyn4YHQM78giemH7KdwuU 3fkS6k7y/wdYQWyqoeF0Ipdprmnd4U1ZirfSjcqjLVYaVT8Co3kbGtvbvHp6AX4LKYJL Oy1/LcEEjw1tZP/cQn+rEJupUL1f601y9+OgNQSmg494IDQV2XN8smNI2RFxU26Q4kdE vl5w==
In-reply-to: <20130409145238.GE22182@xxxxxxx>
References: <CADFMGuJm5bPPwbbUtYwrCVDL23KExJTw_-VRX2UEEdZjo+i5oA@xxxxxxxxxxxxxx> <CADFMGu+=MM2yc=_peboV7JTNJ8F05TJfexmEErzcf0D8mAWFRg@xxxxxxxxxxxxxx> <CADFMGuKqkPbpcU=taqjTR4sA3o=w1LLAnKoEuj=OhJqEbQVijw@xxxxxxxxxxxxxx> <20130409145238.GE22182@xxxxxxx>
No crash just force shutdown, thank you for replying to me.

2013/4/9, Ben Myers <bpm@xxxxxxx>:
> Hey Yongtaofu,
>
> On Tue, Apr 09, 2013 at 09:05:32PM +0800, 符永涛 wrote:
>> Also I want to know why all the server, all crash with the same crash
>> stack?
>> Thank you, really need your help.
>
> What you've posted so far looks like evidence of a forced shutdown and not
> a
> crash.  Is there a crash in addition to this forced shutdown?  If so, can
> you
> post the stack for that too?
>
>>
>> 2013/4/9, 符永涛 <yongtaofu@xxxxxxxxx>:
>> > BTW
>> > xfs_info /dev/sdb
>> > meta-data=/dev/sdb               isize=256    agcount=28,
>> > agsize=268435440
>> > blks
>> >          =                       sectsz=512   attr=2
>> > data     =                       bsize=4096   blocks=7324303360,
>> > imaxpct=5
>> >          =                       sunit=16     swidth=160 blks
>> > naming   =version 2              bsize=4096   ascii-ci=0
>> > log      =internal               bsize=4096   blocks=521728, version=2
>> >          =                       sectsz=512   sunit=16 blks,
>> > lazy-count=1
>> > realtime =none                   extsz=4096   blocks=0, rtextents=0
>> >
>> > 2013/4/9, 符永涛 <yongtaofu@xxxxxxxxx>:
>> >> Dear xfs experts,
>> >> I really need your help sincerely!!! In our production enviroment we
>> >> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file
>> >> system crash on some of the server frequently about every two weeks.
>> >> Can you help to give me a direction about how to debug this issue and
>> >> how to avoid it? Thank you very very much!
>> >>
>> >> uname -a
>> >> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64
>> >> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
>> >>
>> >> Every time the crash log is same, as following
>
> An initial guess is that somehow it is looking up a bad inode number, e.g.
> it
> is beyond the end of the filesystem and xfs_dilocate returns EINVAL.
>
> You could 'xfs_repair -n' to see what it finds (without modifying the
> filesystem) as a first step.
>
>> >> 038 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:
>> >> xfs_inotobp() returned error 22.
>
> Were there any lines of output before this?  In some codebases there are
> prints
> in xfs_inotobp that would help show what happened.
>
>> >> 1039 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree
>> >> returned error 22
>> >> 1040 Apr  9 09:41:36 cqdx kernel: XFS (sdb):
>> >> xfs_do_force_shutdown(0x1) called from line 1184 of file
>> >> fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02ee20a
>> >> 1041 Apr  9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.
>> >> Shutting down filesystem
>> >> 1042 Apr  9 09:41:36 cqdx kernel: XFS (sdb): Please umount the
>> >> filesystem and rectify the problem(s)
>> >> 1043 Apr  9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5
>> >> returned.
>> >> 1044 Apr  9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5
>> >> returned.
>> >> 1045 Apr  9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5
>> >> returned.
>> >> 1046 Apr  9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5
>> >> returned.
>
> The error 5 (EIO) look scary but they are due to the forced shutdown, don't
> worry about them.
>
> Thanks,
>       Ben
>


-- 
符永涛

<Prev in Thread] Current Thread [Next in Thread>