-----Original Message-----
From: xfs-bounces@xxxxxxxxxxx [mailto:xfs-bounces@xxxxxxxxxxx] On Behalf Of
Dave Chinner
Sent: Tuesday, February 12, 2013 9:16 PM
To: Cheung, Norman
Cc: 'linux-xfs@xxxxxxxxxxx'
Subject: Re: Hung in D state during fclose
On Wed, Feb 13, 2013 at 12:12:47AM +0000, Cheung, Norman wrote:
> Dave,
>
> One other point I have forgotten to mention is that the parent thread
> will wait for 5 minutes and then lower the thread priority (from -2
> back to 20) and set a global variable to signal the threads to exit.
> The blocked thread responded well and exit from D state and fclose
> completed with no error.
So it's not hung - it's just very slow?
[NLC] It will pause forever. I tried replaced the timeout with a sleep loop,
and it will pause forever. That is how I got the stack trace.
You have 256GB of memory. It's entirely possible that you've dirtied a large
amount of memory and everything is simply stuck waiting for writeback to occur.
Perhaps you should have a look at the utilisation of your disks when this still
occurs. 'iostat -x -d -m 5' will give you some insight into utilsation when a
hang occurs...
[NLC] I have set the dirty_background_bytes to 40M and
dirty_writeback_centisecs to 400; and watched the Meminfo. I don't get a lot
of dirtied memory accumulation -- 7 to 10 M average. The disk usage from the
sar log was steady at 150M/sec for each of the disk; but a few seconds after
the fclose all disk activities stopped. Also the CPU % was quiet as well.
> This cause me to wonder if it is possible that some XFS threads and
> my application thread might be in a deadlock.
Deadlocks are permanent, so what you are seeing is not a deadlock....
[NLC] Would it be possible that there is priority inversion between my disk
writing threads and the XFS threads & flush threads? My application thread
runs at -2 priority.
Many thanks,
Norman
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
|