xfs
[Top] [All Lists]

Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inoto

To: 符永涛 <yongtaofu@xxxxxxxxx>
Subject: Re: need help how to debug xfs crash issue xfs_iunlink_remove: xfs_inotobp() returned error 22
From: Eric Sandeen <sandeen@xxxxxxxxxxx>
Date: Tue, 09 Apr 2013 10:06:06 -0500
Cc: xfs@xxxxxxxxxxx
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CADFMGuJm5bPPwbbUtYwrCVDL23KExJTw_-VRX2UEEdZjo+i5oA@xxxxxxxxxxxxxx>
References: <CADFMGuJm5bPPwbbUtYwrCVDL23KExJTw_-VRX2UEEdZjo+i5oA@xxxxxxxxxxxxxx>
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130328 Thunderbird/17.0.5
On 4/9/13 7:53 AM, 符永涛 wrote:
> Dear xfs experts,
> I really need your help sincerely!!! In our production enviroment we
> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file
> system crash on some of the server frequently about every two weeks.
> Can you help to give me a direction about how to debug this issue and
> how to avoid it? Thank you very very much!

So this happens reliably, but infrequently? (only every 2 weeks or so?)

Can you provoke it any more often?

> uname -a
> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64
> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

That's a RHEL6 kernel; I'm assuming that this is a RHEL clone w/o RH support?

I agree with Ben that I'd like to see xfs_repair output.

Since the fs has shut down, you should unmount, remount, and unmount
again to replay the dirty log.  Then do xfs_repair -n, and provide the output
if it discovers any errors.

Thanks,
-Eric

> Every time the crash log is same, as following
> 
> 038 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:
> xfs_inotobp() returned error 22.
> 1039 Apr  9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree
> returned error 22
> 1040 Apr  9 09:41:36 cqdx kernel: XFS (sdb):
> xfs_do_force_shutdown(0x1) called from line 1184 of file
> fs/xfs/xfs_vnodeops.c.  Return address = 0xffffffffa02ee20a
> 1041 Apr  9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.
> Shutting down filesystem
> 1042 Apr  9 09:41:36 cqdx kernel: XFS (sdb): Please umount the
> filesystem and rectify the problem(s)
> 1043 Apr  9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1044 Apr  9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1045 Apr  9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 1046 Apr  9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 

<Prev in Thread] Current Thread [Next in Thread>