help xfs filesystem crash

[符永涛]

Dear xfs experts,

We're running glusterfs over top of xfs and recently we have
encountered xfs filesystem crash two times on two different servers.
Can you kindly help to give me some insight of how to debug this kind
of failure?
Bellow is xfs filesystem crash messages:
server1:
Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_iunlink_remove:
xfs_inotobp() returned error 22.
Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_inactive: xfs_ifree
returned error 22
Dec 28 16:55:01 localhost kernel: XFS (dm-0):
xfs_do_force_shutdown(0x1) called from line 1184 of file
fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa029f20a
Dec 28 16:55:01 localhost kernel: XFS (dm-0): I/O Error Detected.
Shutting down filesystem
Dec 28 16:55:01 localhost kernel: XFS (dm-0): Please umount the
filesystem and rectify the problem(s)
Dec 28 16:55:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:55:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:56:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:56:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:57:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:57:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:58:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:58:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:59:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 16:59:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:00:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:00:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:01:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:01:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:02:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:02:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:03:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:03:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:04:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:04:20 localhost sshd[20609]: Accepted publickey for
fuyongtao from 10.1.12.99 port 56514 ssh2
Dec 28 17:04:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:05:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:05:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:06:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:06:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:07:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:07:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:08:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:08:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned.
Dec 28 17:08:55 localhost kernel: XFS (dm-0):
xfs_do_force_shutdown(0x1) called from line 1056 of file
fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa02a6213


server2:
Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_iunlink_remove:
xfs_inotobp() returned error 22.
Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_inactive: xfs_ifree
returned error 22
Dec 27 00:30:02 localhost kernel: XFS (sdb):
xfs_do_force_shutdown(0x1) called from line 1184 of file
fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa022420a
Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_imap_to_bp:
xfs_trans_read_buf() returned error 5.
Dec 27 00:30:02 localhost kernel: XFS (sdb): I/O Error Detected.
Shutting down filesystem
Dec 27 00:30:02 localhost kernel: XFS (sdb): Please umount the
filesystem and rectify the problem(s)
Dec 27 00:30:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:30:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:31:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:31:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:32:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:32:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:33:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:33:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:34:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:34:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:35:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:35:32 localhost smbd[25481]: [2012/12/27 00:35:32.424643, 0]
printing/print_cups.c:109(cups_connect)
Dec 27 00:35:32 localhost smbd[25481]: Unable to connect to CUPS
server localhost:631 - Connection refused
Dec 27 00:35:32 localhost smbd[2061]: [2012/12/27 00:35:32.424906, 0]
printing/print_cups.c:468(cups_async_callback)
Dec 27 00:35:32 localhost smbd[2061]: failed to retrieve printer list:
NT_STATUS_UNSUCCESSFUL
Dec 27 00:35:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:36:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.
Dec 27 00:36:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned.


-- 
符永涛