[Top] [All Lists]

Re: Bug#557262: 2.6.31+ XFS - All I/O locks up to D-state after

To: Faidon Liambotis <paravoid@xxxxxxxxxx>
Subject: Re: Bug#557262: 2.6.31+ XFS - All I/O locks up to D-state after 24-48 hours (sysrq-t+w available) - root cause found = asterisk
From: Roger Heflin <rogerheflin@xxxxxxxxx>
Date: Sat, 21 Nov 2009 08:29:18 -0600
Cc: Justin Piszcz <jpiszcz@xxxxxxxxxxxxxxx>, 557262@xxxxxxxxxxxxxxx, Dave Chinner <david@xxxxxxxxxxxxx>, submit@xxxxxxxxxxxxxxx, linux-kernel@xxxxxxxxxxxxxxx, xfs@xxxxxxxxxxx, linux-raid@xxxxxxxxxxxxxxx, asterisk-users@xxxxxxxxxxxxxxxx, Alan Piszcz <ap@xxxxxxxxxxxxx>
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=cCMtOhkcKVU78q6Cc+vCX4bnp47p90MpL4DyhWgcZaM=; b=NWeF/DLKM/jra+UHchoJxdw4keg8ay54Vjtz/OqAsLUvDCqf1PgpjqXkjeKFaEGJEL SyXA10rLD4ofiW/VNnPFDVC/FDa6O6uWLJytko4WLkdo5nS5cFevO6HyxWEz+j5HmU3y ArMVj4n77JmIHMswou5FHGEua4Bx+RoSlUF7Y=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; b=qU8KfQC++ChZAdyc+fnGIHpJ7dsz2pLJwYkRbyewAk9NsSIBfyAuy2PpTlbl4uLPe6 tqiEjgf7xJFTj7HiG6AZUQz4IHt2920c+oVHRyS6sy6fcq1bFJvZPbUKFsl7KN2uvfTt He53zQXjs2Hy1bGmkiA56LxaTezeO2WAzOwgs=
In-reply-to: <4B0729D8.3000105@xxxxxxxxxx>
References: <alpine.DEB.2.00.0910171825270.16781@xxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0910181607040.27363@xxxxxxxxxxxxxxxx> <20091019030456.GS9464@xxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0910190431180.23395@xxxxxxxxxxxxxxxx> <20091020003358.GW9464@xxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0910200431290.21878@xxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0910210618210.10288@xxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0911201530500.10757@xxxxxxxxxxxxxxxx> <4B0729D8.3000105@xxxxxxxxxx>
User-agent: Thunderbird (X11/20090320)
Faidon Liambotis wrote:
Justin Piszcz wrote:
 > Found root cause-- root cause is asterisk PBX software.  I use an
When someone called me, they accidentally dropped the connection, I called
them back in a short period.  It is during this time (and the last time)
this happened that the box froze under multiple(!) kernels, always when
someone was calling.
I don't know what asterisk is doing but top did run before the crash
and asterisk was using 100% CPU and as I noted before all other processes
were in D-state.

When this bug occurs, it freezes I/O to all devices and the only way to
is to reboot the system.
That's obviously *not* the root cause.

It's not normal for an application that isn't even privileged to hang
all I/O and, subsequently everything on a system.

This is almost probably a kernel issue and asterisk just does something
that triggers this bug.


I had an application in 2.6.5 (SLES9)...that would hang XFS.

The underlying application was multi-threaded and both threads were doing full disks syncs every so often, and sometimes when doing the full disk sync the XFS subsystem would deadlock, it appeared to me tha one sync had a lock and was waiting for another, and the other process had the second lock and was waiting for the first... We were able to disable the full disk sync from the application and the deadlock went away. All non-xfs filesytems still worked and could still be accessed. I did report the bug with some traces but I don't believe anyone ever determined where the underlying issues was.

<Prev in Thread] Current Thread [Next in Thread>