[XFS] Any process to a particular XFS device hung in D state forever.

[Hugo Kuo]

Hi XFS team,

We encountered a problem frequently in past three weeks. Our daemons store
data to XFS partition associate with xattr.

Disk seems not responding since all processes to this disk in D state and
can't be killed at all.

   - It happens on several disks. I feel it's randomly.
   - Reboot seems solve the problem temporarily.
   - All disks are multipath devices.


I suspected that's an issue from disk corrupted at beginning. But smartctl
doesn't show any clue about disk bad. And reboot makes the problem gone
away.


   - Any process to this disk is blocked. Even a simple $ls . Kernel log
   <https://gist.github.com/HugoKuo/f87748786b26ea04fd9e1d86d9538293>
   - I tested the disk by read bytes on block via $dd . It works fine
   without any error in dmesg.
   - The `xfs_repair -n` output of a problematic mount point [xfs_repair -n]
   <https://gist.github.com/HugoKuo/76f65bdc0b860ca6ed5e786f8c43da0e> . It
   is still processing.
   - Kernel : Linux node9 2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10
   18:01:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
   - OS : CentOS release 6.5 (Final)
   - XFS : xfsprogs.x86_64         3.1.1-14.el6


There's an interesting behaviour of $ls command.

* This is completed in 1sec. Very quick and give me the result in the
test.d864 file $ls /srv/node/d864/tmp > test.d864
* This is hanging $ls /srv/node/d864/tmp

[image: Inline image 1]

I suspect there's something wrong with imap. Is there a known bug ?

Thanks // Hugo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20160419/af8939b3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dd.png
Type: image/png
Size: 307262 bytes
Desc: not available
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20160419/af8939b3/attachment-0001.png>