xfs
[Top] [All Lists]

Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inoto

To: Eric Sandeen <sandeen@xxxxxxxxxxx>
Subject: Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22
From: 符永涛 <yongtaofu@xxxxxxxxx>
Date: Fri, 12 Apr 2013 09:38:49 +0800
Cc: Brian Foster <bfoster@xxxxxxxxxx>, Ben Myers <bpm@xxxxxxx>, "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ekyi+ylxC5gcuync5LDhtZj5jGkCppQeyEcRXtKGt84=; b=vmr/mPQxk43CdoFRyyK6y9juAbrqtRJeN+FsIMKb61b5PYm4sUfuMudnrpJGx2ywI9 cKXMigo8c4nl4DQDrO935+6cp/TyE2vae5G1ZSLzfBNMcDuZyEB6xME6pIp043KtRJov LzOk/x5T0SUPJRSlmJ8GLsT1UgNQT3v7n7F0Vn9REVDJHLa2hlQut9DAn+E8zyjnFCL6 L0+Vr4v0dgrdha3D8Z3cNFzMrE80kYXEplhyAbWyGM+g4tG+f5GkKUA6xnvDbdPyXZZL ZqPPz2Gwg3oxuDaaF+1ssjX+HkDxmAS5oA+6Pzi/WyKD1mY+eEa/L497AgImtg1RQBFO 8MmQ==
In-reply-to: <CADFMGu+_ppTcr0iZ6YmCGkUT0MJUcLX4iSOOCdURdHVyxKKd6A@xxxxxxxxxxxxxx>
References: <CADFMGuJm5bPPwbbUtYwrCVDL23KExJTw_-VRX2UEEdZjo+i5oA@xxxxxxxxxxxxxx> <CADFMGu+=MM2yc=_peboV7JTNJ8F05TJfexmEErzcf0D8mAWFRg@xxxxxxxxxxxxxx> <CADFMGuKqkPbpcU=taqjTR4sA3o=w1LLAnKoEuj=OhJqEbQVijw@xxxxxxxxxxxxxx> <20130409145238.GE22182@xxxxxxx> <CADFMGuJaiH0wuxOHrDjDn7qRVH+vQkLOOSPUyqSdXnLcS47t3w@xxxxxxxxxxxxxx> <CADFMGu+TdyjTjMTWMwpdHqmszhpCU162UA4Y-njARwSEjM1xNw@xxxxxxxxxxxxxx> <20130410121025.78a42b22@xxxxxxxxxxxxxxxxxxxx> <CADFMGu+yCg4ux0n6S98bqm_cXc=VCcijVBTqwRxvxmtKt_JO-A@xxxxxxxxxxxxxx> <CADFMGuLxgBFU=FUK94tPsCh+qxRW0rEELxSXYoMQLFJ1u3=q0Q@xxxxxxxxxxxxxx> <516746AC.3090808@xxxxxxxxxx> <51675E67.7030600@xxxxxxxxxxx> <CADFMGu+_ppTcr0iZ6YmCGkUT0MJUcLX4iSOOCdURdHVyxKKd6A@xxxxxxxxxxxxxx>
Next time I'll dump it according to your suggestions, thank you.


2013/4/12 符永涛 <yongtaofu@xxxxxxxxx>
Sorry I didn't dump the meta right after unmounting post-failure, I have dumped meta after a mount/umount, I'll share the meta to you soon.


2013/4/12 Eric Sandeen <sandeen@xxxxxxxxxxx>
On 4/11/13 6:26 PM, Brian Foster wrote:
> On 04/11/2013 03:11 PM, 符永涛 wrote:
>> It happens tonight again on one of our servers, how to debug the root
>> cause? Thank you.
>>
>
> Hi,
>
> I've attached a system tap script (stap -v xfs.stp) that should
> hopefully print out a bit more data should the issue happen again. Do
> you have a small enough number of nodes (or predictable enough pattern)
> that you could run this on the nodes that tend to fail and collect the
> output?
>
> Also, could you collect an xfs_metadump of the filesystem in question
> and make it available for download and analysis somewhere? I believe the
> ideal approach is to mount/umount the filesystem first to replay the log
> before collecting a metadump, but somebody could correct me on that (to
> be safe, you could collect multiple dumps: pre-mount and post-mount).

Dave suggested yesterday that this would be best: metadump right
after unmounting post-failure, then mount/umount & generate another metadump.

-Eric

> Could you also describe your workload a little bit? Thanks.
>
> Brian
>
>> Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_iunlink_remove:
>> xfs_inotobp() returned error 22.
>> Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree returned
>> error 22
>> Apr 12 02:32:10 cqdx kernel: XFS (sdb): xfs_do_force_shutdown(0x1)
>> called from line 1184 of file fs/xfs/xfs_vnodeops.c.  Return address =
>> 0xffffffffa02ee20a
>> Apr 12 02:32:10 cqdx kernel: XFS (sdb): I/O Error Detected. Shutting
>> down filesystem
>> Apr 12 02:32:10 cqdx kernel: XFS (sdb): Please umount the filesystem and
>> rectify the problem(s)
>> Apr 12 02:32:19 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
>> Apr 12 02:32:49 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
>> Apr 12 02:33:19 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
>> Apr 12 02:33:49 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
>>




--
符永涛



--
符永涛
<Prev in Thread] Current Thread [Next in Thread>