<div dir="ltr">Today 3 of our servers were impacted by the xfs shutdown. The logs are identical.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/9 ·ûÓÀÌÎ <span dir="ltr"><<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">before xfs force shutdown happens there seems no useful log in /var/log/messages<br><br>Apr 9 10:38:08 cqdx smbd[4597]: Unable to connect to CUPS server localhost:631 - Connection refused<br>
Apr 9 10:38:08 cqdx smbd[3394]: [2013/04/09 10:38:08.944125, 0] printing/print_cups.c:468(cups_async_callback)<br>
Apr 9 10:38:08 cqdx smbd[3394]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL<br>Apr 9 10:51:09 cqdx smbd[5205]: [2013/04/09 10:51:09.723610, 0] printing/print_cups.c:109(cups_connect)<br>Apr 9 10:51:09 cqdx smbd[5205]: Unable to connect to CUPS server localhost:631 - Connection refused<br>
Apr 9 10:51:09 cqdx smbd[3394]: [2013/04/09 10:51:09.724132, 0] printing/print_cups.c:468(cups_async_callback)<br>Apr 9 10:51:09 cqdx smbd[3394]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL<br>Apr 9 11:01:30 cqdx kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22.<br>
Apr 9 11:01:30 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22<br>Apr 9 11:01:30 cqdx kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02ee20a<br>
Apr 9 11:01:30 cqdx kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem<br>Apr 9 11:01:30 cqdx kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s)<br>Apr 9 11:01:51 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
Apr 9 11:02:21 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>Apr 9 11:02:51 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>Apr 9 11:03:21 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>
Apr 9 11:03:51 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned.<br>Apr 9 11:03:57 cqdx init: tty (/dev/tty1) main process (3427) killed by TERM signal<br>Apr 9 11:03:57 cqdx init: tty (/dev/tty2) main process (3429) killed by TERM signal<br>
<br></div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">2013/4/9 Ben Myers <span dir="ltr"><<a href="mailto:bpm@sgi.com" target="_blank">bpm@sgi.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey Yongtaofu,<br>
<div><br>
On Tue, Apr 09, 2013 at 09:05:32PM +0800, ·ûÓÀÌÎ wrote:<br>
> Also I want to know why all the server, all crash with the same crash stack?<br>
> Thank you, really need your help.<br>
<br>
</div>What you've posted so far looks like evidence of a forced shutdown and not a<br>
crash. Is there a crash in addition to this forced shutdown? If so, can you<br>
post the stack for that too?<br>
<div><br>
><br>
> 2013/4/9, ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>>:<br>
> > BTW<br>
> > xfs_info /dev/sdb<br>
> > meta-data=/dev/sdb isize=256 agcount=28, agsize=268435440<br>
> > blks<br>
> > = sectsz=512 attr=2<br>
> > data = bsize=4096 blocks=7324303360, imaxpct=5<br>
> > = sunit=16 swidth=160 blks<br>
> > naming =version 2 bsize=4096 ascii-ci=0<br>
> > log =internal bsize=4096 blocks=521728, version=2<br>
> > = sectsz=512 sunit=16 blks, lazy-count=1<br>
> > realtime =none extsz=4096 blocks=0, rtextents=0<br>
> ><br>
> > 2013/4/9, ·ûÓÀÌÎ <<a href="mailto:yongtaofu@gmail.com" target="_blank">yongtaofu@gmail.com</a>>:<br>
> >> Dear xfs experts,<br>
> >> I really need your help sincerely!!! In our production enviroment we<br>
> >> run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file<br>
> >> system crash on some of the server frequently about every two weeks.<br>
> >> Can you help to give me a direction about how to debug this issue and<br>
> >> how to avoid it? Thank you very very much!<br>
> >><br>
> >> uname -a<br>
> >> Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64<br>
> >> #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux<br>
> >><br>
> >> Every time the crash log is same, as following<br>
<br>
</div>An initial guess is that somehow it is looking up a bad inode number, e.g. it<br>
is beyond the end of the filesystem and xfs_dilocate returns EINVAL.<br>
<br>
You could 'xfs_repair -n' to see what it finds (without modifying the<br>
filesystem) as a first step.<br>
<div><br>
> >> 038 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove:<br>
> >> xfs_inotobp() returned error 22.<br>
<br>
</div>Were there any lines of output before this? In some codebases there are prints<br>
in xfs_inotobp that would help show what happened.<br>
<div><br>
> >> 1039 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree<br>
> >> returned error 22<br>
> >> 1040 Apr 9 09:41:36 cqdx kernel: XFS (sdb):<br>
> >> xfs_do_force_shutdown(0x1) called from line 1184 of file<br>
> >> fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02ee20a<br>
> >> 1041 Apr 9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected.<br>
> >> Shutting down filesystem<br>
> >> 1042 Apr 9 09:41:36 cqdx kernel: XFS (sdb): Please umount the<br>
> >> filesystem and rectify the problem(s)<br>
> >> 1043 Apr 9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> >> returned.<br>
> >> 1044 Apr 9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> >> returned.<br>
> >> 1045 Apr 9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> >> returned.<br>
> >> 1046 Apr 9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5<br>
> >> returned.<br>
<br>
</div>The error 5 (EIO) look scary but they are due to the forced shutdown, don't<br>
worry about them.<br>
<br>
Thanks,<br>
Ben<br>
</blockquote></div><br><br clear="all"><br></div></div><span class="HOEnZb"><font color="#888888">-- <br>·ûÓÀÌÎ
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br>·ûÓÀÌÎ
</div>