xfs
[Top] [All Lists]

XFS issue xfs goes offline with various messages drive not recoverable w

To: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Subject: XFS issue xfs goes offline with various messages drive not recoverable without reboot
From: Simon Dray <sdray@xxxxxxxxxx>
Date: Thu, 25 Sep 2014 07:30:23 +0000
Accept-language: en-GB, en-US
Delivered-to: xfs@xxxxxxxxxxx
Thread-index: Ac/YkpDG9jVR1llKTc+P4l4O2U49Aw==
Thread-topic: XFS issue xfs goes offline with various messages drive not recoverable without reboot

Dear Sirs

 

I wonder if you can help with an issue we see re-occuring on a regular basis with one of our HP systems which uses a HP 420 Raid controller

 

Action taken

 

We first saw the following:
[root@ content]# ls
ls: cannot open directory .: Input/output error


[root@ /]# ls -ltr
ls: cannot access content
total 358
d?????????? ? ? ? ? ? content
drwxr-xr-x. 2 root root 4096 Jun 28 2011 srv
drwxr-xr-x. 2 root root 4096 Jun 28 2011 media
drwxr-xr-x. 2 root root 4096 Feb 22 2012 cgroup
drwx------. 2 root root 16384 Jul 21 2012 lost+found
drwxr-xr-x. 2 root root 4096 Jul 21 2012 selinux

We try to run:
[root@ /]# xfs_check /dev/md0
xfs_check: /dev/md0 contains a mounted and writable filesystem
fatal error -- couldn't initialize XFS library
We also tried to umount the /dev/md0 before runniing xfs_check but no luck. We received the error: device is in use

 

We use xfs for one of our large raid file systems and we are seeing the xfs filesystem go offline with the following messages in dmesg

 

messages-20140921:Sep 18 23:01: kernel: XFS (md0): Device md0: metadata write error block 0x5e28623d8
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): I/O error occurred: meta-data dev md0 block 0x445cccc40 ("xlog_iodone") error 5 buf count 32768
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa2c428dc
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): Log I/O Error Detected. Shutting down filesystem
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): Please umount the filesystem and rectify the problem(s)
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_iunlink_remove: xfs_itobp() returned error 5.
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): I/O error occurred: meta-data dev md0 block 0x445cccc80 ("xlog_iodone") error 5 buf count 32768
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa2c428dc

 

XFS (md0): xfs_log_force: error 5 returned.
XFS (md0): xfs_log_force: error 5 returned.
XFS (md0): xfs_log_force: error 5 returned.

 

In all occurrences the only way to recover from this is to reboot the system and allow xfs_repair to run during boot this clears the issue until next time

 

We have checked the RAID health and nothing seems to be amiss, if you could help with this it would be much appreciated

 

 

Best regards Simon

 

 

Simon Dray s

 

p: +44.1223 716.400

p: +44.1223 716.476

e: sdray@xxxxxxxxxx

 

1st Floor, 335 Cambridge Science Park, Milton Road, Cambridge, Cambridgeshire, CB4 0WN, United Kingdom.

 

Understanding is a three-edged sword

 

<Prev in Thread] Current Thread [Next in Thread>