On Tue, Nov 22, 2011 at 03:53:07PM -0600, Eric Sandeen wrote:
> > Ending recovery (logdev: internal)
> > ...
>
> All that recovery a result of the icky shutdown procedure I guess....
>
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.214688] XFS: Internal
> > error XFS_WANT_CORRUPTED_GOTO at line 1664 of file fs/xfs/xfs_alloc.c.
> > Caller 0xffffffff811d6b71
>
> And this was the first indication of trouble.
allocation btree corruption - we try to free a block, which had already
been inserted into the by-size allocation btree. This very much looks
like a cache failure to me.
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.302172]
> > [<ffffffff810d2b11>] ? sys_truncate+0x171/0x173
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.307846]
> > [<ffffffff8166c07b>] ? system_call_fastpath+0x16/0x1b
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.314031] XFS (sdg1):
> > xfs_do_force_shutdown(0x8) called from line 3864 of file
> > fs/xfs/xfs_bmap.c. Return address = 0xffffffff811e2046
>
> by here it had shut down, and you were just riding along when
> it went kablooey. Any non-xfs error just before this point?
And this was the caller of xfs_free_extent, now shuting the fs down
because of the above error.
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.340451] XFS (sdg1):
> > Corruption of in-memory data detected. Shutting down filesystem
> > Nov 17 16:01:01 cephstore6358 kernel: [ 214.348518] XFS (sdg1):
> > Please umount the filesystem and rectify the problem(s)
> > Nov 17 16:01:01 cephstore6358 kernel: [ 227.789285] XFS (sdg1):
> > xfs_log_force: error 5 returned.
> > Nov 17 16:01:01 cephstore6358 kernel: [ 229.820255] XFS (sdg1):
> > xfs_log_force: error 5 returned.
>
> To be honest I'm not sure offhand if this error 5 (EIO) is a
> result of the shutdown, or the cause of it.
It is. One the filesystem has been shut down xfs_log_force will
always return EIO. The printk for is rather useless, though.
|