Thank you for your help. Two questions:
1) Can you define "much larger number"? I know you recently increased
this number from 10 to 1000, so should I increase it to 10,000? 100,000?
2) Is this a fix or a work-around? If this is a work-around, is there a
fix in the works? Can you explain the issue a bit, or if it's been
covered, can you point me to the explanation? I'd just like to
understand what's going on.
Thanks,
Kris Kersey
David Chinner wrote:
> On Thu, Mar 06, 2008 at 04:31:27PM -0500, Kris Kersey wrote:
>> Hello,
>>
>> I'm working on a NAS product and we're currently having lock-ups that
>> seem to be hanging in XFS code. We're running a NAS that has 1024 NFSD
>> threads accessing three RAID mounts. All three mounts are running XFS
>> file systems. Lately we've had random lockups on these boxes and I am
>> now running a kernel with KDB built-in.
>>
>> The lock-up takes the form of all NFSD threads in D state with one out
>> of three pdflush threads in D state. The assumption can be made that
>> all NFSD threads are waiting on the one pdflush thread to complete. So
>> two times now when an NAS has gotten in this state I have accessed KDB
>> and ran a stack trace on the pdflush thread. Both times the thread was
>> stuck on xlog_grant_log_space+0xdb.
>
> Try bumping XFS_TRANS_PUSH_AIL_RESTARTS to a much larger number and
> seeing if the problem goes away....
>
> Alternatively, that restart hack is backed by a "watchdog" timeout
> in 2.6.25-rc1, so if that is the cause of the problem perhaps the
> latest -rcX kernel will prevent the hang?
>
> BTW, you can get all the traces of D state threads through the sysrq
> interface, so you don't need to drop into kdb to get this.....
>
> Cheers,
>
> Dave.
|