xfs umount with i/o error hang/memory corruption

[Bob Mastors <bob.mastors@xxxxxxxxxxxxx>]

Greetings,

I am new to xfs and am running into a problem

and would appreciate any guidance on how to proceed.

After an i/o error from the block device that xfs is using,

an umount results in a message like:

[ Â370.636473] XFS (sdx): Log I/O Error Detected. ÂShutting down filesystem

[ Â370.644073] XFS (h      ïïïh"h     ÂïïïH#h     Âïïïbsg): Please umount the filesystem and rectify the problem(s)

Note the garbage on the previous line which suggests memory corruption.

About half the time I get the garbled log message. About half the time umount hangs.

And then I get this kind of error and the system is unresponsive:

Message from syslogd@debian at Apr Â4 09:27:40 ...

Âkernel:[ Â680.080022] BUG: soft lockup - CPU#2 stuck for 22s! [umount:2849]

The problem appears to be similar to this issue:

http://www.spinics.net/lists/linux-xfs/msg00061.html

I can reproduce the problem easily using open-iscsi to create

the block device with an iscsi initiator.

I use lio to create an iscsi target.

The problem is triggered by doing an iscsi logout which causes

the block device to return i/o errors to xfs.

Steps to reproduce the problem are below.

Using VirtualBox, I can reproduce it with two processors but not one.

I first saw this on a 3.8 kernel and most recently reproduced it with 3.14+.

Thanks,

Bob

virtual box setup

-----------------

1 GB ram

8 GB disk

2 processors

Operating System: Linux

Version: Debian (64 bit)

Bridged networking

iso: debian-7.4.0-amd64-netinst.iso

Software selection

  select SSH server

  select Standard system utilities

  deselect everything else

 ÂÂ

dhcp ip address: 172.17.5.129

upgrade linux to 3.14.0+

------------------------

ssh 172.17.5.129

su

cd /mnt/ksrc/linux

make modules_install

make install

reboot

install tools

-------------

ssh 172.17.5.129

su

apt-get install xfsprogs

apt-get install open-iscsi

apt-get install --no-install-recommends targetcli python-urwid

reboot

reproduce bug

-------------

ssh 172.17.5.129

su

targetcli

  cd backstores

  fileio/ create name=file_backend file_or_dev=my_disk size=100M

  /iscsi create

  portals/ create 172.17.5.129

  cd <

  luns/ create /backstores/fileio/file_backend

  cd <

  set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

  exit

iscsiadm -m discovery -t sendtargets -p 172.17.5.129

iscsiadm -m node --login --target iqn.from_previous_iscsiadm_cmd_output

fdisk -l

mkfs -t xfs -f /dev/sdXX

mkdir /mnt/vola

mount -t xfs /dev/sdXX /mnt/vola

iscsiadm -m node -U all

umount /mnt/vola