xfs
[Top] [All Lists]

Re: xfs/md filesystem hang on drive pull under IO with 2.6.35.13

To: Benedict Singer <bzsing@xxxxxxxxx>
Subject: Re: xfs/md filesystem hang on drive pull under IO with 2.6.35.13
From: NeilBrown <neilb@xxxxxxx>
Date: Wed, 18 Jul 2012 07:47:05 +1000
Cc: xfs@xxxxxxxxxxx, linux-raid@xxxxxxxxxxxxxxx
In-reply-to: <5005D661.8030200@xxxxxxxxx>
References: <5005D661.8030200@xxxxxxxxx>
On Tue, 17 Jul 2012 14:17:21 -0700 Benedict Singer <bzsing@xxxxxxxxx> wrote:

> Hi XFS and MD experts,
> 
> I'm experiencing a problem with a setup running XFS on top of an MD 
> raid. The test I'm running is physically pulling a drive while the 
> system is running, to simulate hardware failure. When the system is 
> idle, this works fine; the md subsystem detects the missing drive and 
> degrades the arrays, and everything keeps running fine. If I pull a 
> drive while heavy IO activity (mostly if not completely reading) is 
> happening on the XFS filesystem, then very often the filesystem seems to 
> "hang" - both processes that were accessing the filesystem at the time 
> hang, as well as any new ones like 'ls'. Luckily, the kernel noticed the 
> hung processes and spit a bunch of useful information to syslog. The 
> relevant snippets are reproduced below, including the messages 
> indicating that the drive was pulled, along with other XFS/MD raid 
> information from the system. The filesystem in question is on 
> /dev/md126, mounted on /var/showstore, and the disk pulled was /dev/sda. 
> Note that at the time this information was collected, the disk had been 
> re-inserted and a rebuild was proceeding.
> 
> Is this a bug (seems like a race condition leading to a deadlock?), 
> and/or is there something to change in the way the system is set up to 
> mitigate or workaround this?

Looks a lot like a bug in md.  It is waiting for the metadata to be marked
'dirty', and is waiting forever. 

2.6.35 is a very old kernel.  It is entirely possible that the bug has been
fixed.
If you can reproduce with something newer I'd certainly be interested.

NeilBrown

Attachment: signature.asc
Description: PGP signature

<Prev in Thread] Current Thread [Next in Thread>