help xfs filesystem crash

[Eric Sandeen]

On 1/5/13 7:47 AM, 符永涛 wrote:
> Dear xfs experts,
> 
> We're running glusterfs over top of xfs and recently we have
> encountered xfs filesystem crash two times on two different servers.
> Can you kindly help to give me some insight of how to debug this kind
> of failure?

Please include more information about your system;
at a minimum, what kernel are you running?  See also:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

for a list of all relevant information to gather when reporting a problem.

If this is a distribution kernel, in particular if you are talking about
Red Hat Storage on Red Hat Enterprise Linux, you should probably
contact your Red Hat support people first.

> Bellow is xfs filesystem crash messages:
> server1:
> Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_iunlink_remove:
> xfs_inotobp() returned error 22.

that is EINVAL, returned from xfs_inotbp, which tries to map an inode
number to the buffer containing the on-disk version of the inode.

It would take some debugging (and at least some more information
about your system) to know where the EINVAL came from.

> Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_inactive: xfs_ifree
> returned error 22
> Dec 28 16:55:01 localhost kernel: XFS (dm-0):

So freeing the inode encountered a runtime error (EINVAL) and then:

> xfs_do_force_shutdown(0x1) called from line 1184 of file
> fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa029f20a
> Dec 28 16:55:01 localhost kernel: XFS (dm-0): I/O Error Detected.
> Shutting down filesystem

the filesystem shut down as a result.

-Eric

> Dec 28 16:55:01 localhost kernel: XFS (dm-0): Please umount the
> filesystem and rectify the problem(s)
> Dec 28 16:55:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:55:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:56:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:56:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:57:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:57:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:58:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:58:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:59:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 16:59:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:00:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:00:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:01:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:01:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:02:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:02:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:03:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:03:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:04:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:04:20 localhost sshd[20609]: Accepted publickey for
> fuyongtao from 10.1.12.99 port 56514 ssh2
> Dec 28 17:04:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:05:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:05:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:06:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:06:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:07:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:07:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:08:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:08:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
> Dec 28 17:08:55 localhost kernel: XFS (dm-0):
> xfs_do_force_shutdown(0x1) called from line 1056 of file
> fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa02a6213
> 
> 
> server2:
> Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_iunlink_remove:
> xfs_inotobp() returned error 22.
> Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_inactive: xfs_ifree
> returned error 22
> Dec 27 00:30:02 localhost kernel: XFS (sdb):
> xfs_do_force_shutdown(0x1) called from line 1184 of file
> fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa022420a
> Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_imap_to_bp:
> xfs_trans_read_buf() returned error 5.
> Dec 27 00:30:02 localhost kernel: XFS (sdb): I/O Error Detected.
> Shutting down filesystem
> Dec 27 00:30:02 localhost kernel: XFS (sdb): Please umount the
> filesystem and rectify the problem(s)
> Dec 27 00:30:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:30:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:31:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:31:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:32:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:32:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:33:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:33:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:34:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:34:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:35:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:35:32 localhost smbd[25481]: [2012/12/27 00:35:32.424643, 0]
> printing/print_cups.c:109(cups_connect)
> Dec 27 00:35:32 localhost smbd[25481]: Unable to connect to CUPS
> server localhost:631 - Connection refused
> Dec 27 00:35:32 localhost smbd[2061]: [2012/12/27 00:35:32.424906, 0]
> printing/print_cups.c:468(cups_async_callback)
> Dec 27 00:35:32 localhost smbd[2061]: failed to retrieve printer list:
> NT_STATUS_UNSUCCESSFUL
> Dec 27 00:35:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:36:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> Dec 27 00:36:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
> 
>