xfs
[Top] [All Lists]

xfs umount with i/o error hang/memory corruption

To: xfs@xxxxxxxxxxx
Subject: xfs umount with i/o error hang/memory corruption
From: Bob Mastors <bob.mastors@xxxxxxxxxxxxx>
Date: Fri, 4 Apr 2014 12:15:23 -0600
Delivered-to: xfs@xxxxxxxxxxx
Greetings,

I am new to xfs and am running into a problem
and would appreciate any guidance on how to proceed.

After an i/o error from the block device that xfs is using,
an umount results in a message like:
[ Â370.636473] XFS (sdx): Log I/O Error Detected. ÂShutting down filesystem
[ Â370.644073] XFS (h      ïïïh"h     ÂïïïH#h     Âïïïbsg): Please umount the filesystem and rectify the problem(s)
Note the garbage on the previous line which suggests memory corruption.
About half the time I get the garbled log message. About half the time umount hangs.

And then I get this kind of error and the system is unresponsive:
Message from syslogd@debian at Apr Â4 09:27:40 ...
Âkernel:[ Â680.080022] BUG: soft lockup - CPU#2 stuck for 22s! [umount:2849]

The problem appears to be similar to this issue:

I can reproduce the problem easily using open-iscsi to create
the block device with an iscsi initiator.
I use lio to create an iscsi target.

The problem is triggered by doing an iscsi logout which causes
the block device to return i/o errors to xfs.
Steps to reproduce the problem are below.

Using VirtualBox, I can reproduce it with two processors but not one.
I first saw this on a 3.8 kernel and most recently reproduced it with 3.14+.

Thanks,
Bob

virtual box setup
-----------------
1 GB ram
8 GB disk
2 processors
Operating System: Linux
Version: Debian (64 bit)
Bridged networking
iso: debian-7.4.0-amd64-netinst.iso
Software selection
  select SSH server
  select Standard system utilities
  deselect everything else
 ÂÂ
dhcp ip address: 172.17.5.129

upgrade linux to 3.14.0+
------------------------
ssh 172.17.5.129
su
cd /mnt/ksrc/linux
make modules_install
make install
reboot

install tools
-------------
ssh 172.17.5.129
su
apt-get install xfsprogs
apt-get install open-iscsi
apt-get install --no-install-recommends targetcli python-urwid
reboot

reproduce bug
-------------
ssh 172.17.5.129
su
targetcli
  cd backstores
  fileio/ create name=file_backend file_or_dev=my_disk size=100M
  /iscsi create
  portals/ create 172.17.5.129
  cd <
  luns/ create /backstores/fileio/file_backend
  cd <
  set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
  exit

iscsiadm -m discovery -t sendtargets -p 172.17.5.129
iscsiadm -m node --login --target iqn.from_previous_iscsiadm_cmd_output
fdisk -l
mkfs -t xfs -f /dev/sdXX
mkdir /mnt/vola
mount -t xfs /dev/sdXX /mnt/vola
iscsiadm -m node -U all
umount /mnt/vola
<Prev in Thread] Current Thread [Next in Thread>