xfs
[Top] [All Lists]

RE: Hung in D state during fclose

To: Dave Chinner <david@xxxxxxxxxxxxx>
Subject: RE: Hung in D state during fclose
From: "Cheung, Norman" <Norman.Cheung@xxxxxxxxxxxxxx>
Date: Thu, 14 Feb 2013 04:53:20 +0000
Accept-language: en-US
Cc: "'linux-xfs@xxxxxxxxxxx'" <linux-xfs@xxxxxxxxxxx>
Delivered-to: linux-xfs@xxxxxxxxxxx
In-reply-to: <20130213051552.GF26694@dastard>
References: <loom.20130212T071115-446@xxxxxxxxxxxxxx> <20130212065545.GC10731@dastard> <3542214BE3A3EF419F236DFE0F878BC90512DC@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <20130212102014.GA26694@dastard> <3542214BE3A3EF419F236DFE0F878BC90517D2@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <20130212202246.GB26694@dastard> <3542214BE3A3EF419F236DFE0F878BC905190D@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <3542214BE3A3EF419F236DFE0F878BC9051A09@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> <20130213051552.GF26694@dastard>
Thread-index: AQHOCOkEoXqR9NnGO06CfqnK8WA+IJh1yiOAgAABbsCAADe0AIAAZz1AgABBHACAAAacIIAAN6NAgABWswCAAYhVYA==
Thread-topic: Hung in D state during fclose

-----Original Message-----
From: xfs-bounces@xxxxxxxxxxx [mailto:xfs-bounces@xxxxxxxxxxx] On Behalf Of 
Dave Chinner
Sent: Tuesday, February 12, 2013 9:16 PM
To: Cheung, Norman
Cc: 'linux-xfs@xxxxxxxxxxx'
Subject: Re: Hung in D state during fclose

On Wed, Feb 13, 2013 at 12:12:47AM +0000, Cheung, Norman wrote:
> Dave,
> 
> One other point I have forgotten to mention is that the parent thread 
> will wait for 5 minutes and then lower the thread priority (from -2 
> back to 20) and set a global variable to signal the threads to exit.  
> The blocked thread responded well and exit from D state and fclose 
> completed with no error.

So it's not hung - it's just very slow?

[NLC] It will pause forever.  I tried replaced the timeout with a sleep loop, 
and it will pause forever.    That is how I got the stack trace.

You have 256GB of memory. It's entirely possible that you've dirtied a large 
amount of memory and everything is simply stuck waiting for writeback to occur. 
Perhaps you should have a look at the utilisation of your disks when this still 
occurs. 'iostat -x -d -m 5' will give you some insight into utilsation when a 
hang occurs...

[NLC] I  have set the dirty_background_bytes to 40M and 
dirty_writeback_centisecs to 400; and watched the Meminfo.  I don't get a  lot 
of dirtied memory accumulation -- 7 to 10 M average.  The disk usage from the 
sar log was steady at 150M/sec for each of the disk; but a few seconds after 
the fclose all disk activities stopped. Also the CPU % was quiet as well.     

> This cause me to wonder if it  is possible that  some XFS threads and 
> my application thread might be in a deadlock.

Deadlocks are permanent, so what you are seeing is not a deadlock....

[NLC] Would it be possible that there is priority inversion between my  disk 
writing threads and the XFS threads & flush threads?  My application thread 
runs at -2 priority.

Many thanks,
Norman


Cheers,

Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs


<Prev in Thread] Current Thread [Next in Thread>