Re: [XFS] Any process to a particular XFS device hung in D state forever.

[Hugo Kuo <hugo@xxxxxxxxxxxxxx>]

Hi Brain,Â

Here's the a gist include sysrq-trigger and strace of one of the hanging $ls result. This is from another problematic disk (d817) on the same server.

https://gist.github.com/HugoKuo/8eb8208bbb7a7f562a6c9a3eafa8f37f

It looks like the hanging $ls is stuck on getting extend attribute of a file on this disk. The full output can be found in the link above.Â

	lstat("/srv/node/d864/tmp/tmpIRYFaW", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
	capget(0x20080522, 0, NULL) = -1 EFAULT (Bad address)
	getxattr("/srv/node/d864/tmp/tmpIRYFaW", "security.capability"

As for the xfs_repair output in linkÂhttps://gist.github.com/HugoKuo/76f65bdc0b860ca6ed5e786f8c43da0eÂ. Your question is if the node been force rebooted. The answer is NO. Â I didn't reboot this server yet. I force unmounted it via $umount -l <dev>Â. Then run the xfs_repair.Â

$ls /srv/node/d864/tmp > test.d864

$ls /srv/node/d864/tmp

Here's the contents of test.d864Âhttps://gist.github.com/HugoKuo/25f93cd6daf5b0666a2ab85defd63a56

Thanks // Hugo

On Tue, Apr 19, 2016 at 7:30 PM, Brian Foster <bfoster@xxxxxxxxxx> wrote:

On Tue, Apr 19, 2016 at 05:56:19PM +0800, Hugo Kuo wrote:
> Hi XFS team,
>
> We encountered a problem frequently in past three weeks. Our daemons store
> data to XFS partition associate with xattr.
>
> Disk seems not responding since all processes to this disk in D state and
> can't be killed at all.
>
>Â Â - It happens on several disks. I feel it's randomly.
>Â Â - Reboot seems solve the problem temporarily.
>Â Â - All disks are multipath devices.
>
>
> I suspected that's an issue from disk corrupted at beginning. But smartctl
> doesn't show any clue about disk bad. And reboot makes the problem gone
> away.
>
>
>Â Â - Any process to this disk is blocked. Even a simple $ls . Kernel log
>Â Â <https://gist.github.com/HugoKuo/f87748786b26ea04fd9e1d86d9538293>

Looks like it's waiting on an AGF buffer. The buffer could be held by
something else, but we don't have enough information from that one
trace. Could you get all of the blocked tasks when in this state (e.g.,
"echo w > /proc/sysrq-trigger")?

Â

>Â Â - I tested the disk by read bytes on block via $dd . It works fine
>Â Â without any error in dmesg.
>Â Â - The `xfs_repair -n` output of a problematic mount point [xfs_repair -n]
>Â Â <https://gist.github.com/HugoKuo/76f65bdc0b860ca6ed5e786f8c43da0e> . It
>Â Â is still processing.

I presume this was run after a forced reboot..? If so, was the
filesystem remounted first to replay the log (xfs_repair -n doesn't
detect/warn about a dirty log, iirc). If the log was dirty, then repair
is a bit less interesting simply because some corruption is to be
expected in that scenario.

>Â Â - Kernel : Linux node9 2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10
>Â Â 18:01:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>Â Â - OS : CentOS release 6.5 (Final)
>Â Â - XFS : xfsprogs.x86_64Â Â Â Â Â3.1.1-14.el6
>
>
> There's an interesting behaviour of $ls command.
>
> * This is completed in 1sec. Very quick and give me the result in the
> test.d864 file $ls /srv/node/d864/tmp > test.d864
> * This is hanging $ls /srv/node/d864/tmp
>

I'm not following you here. Are you missing an attachment (test.d864)?

Brian

> [image: Inline image 1]
>
> I suspect there's something wrong with imap. Is there a known bug ?
>
> Thanks // Hugo



> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs