XFS hangs with XFS: possible memory allocation deadlock in kmem_alloc

[Dave Chinner]

On Sat, Mar 07, 2015 at 09:07:21AM -0500, Brian Foster wrote:
> On Sat, Mar 07, 2015 at 08:51:50AM +0100, Michael Meier wrote:
> > We've recently upgraded the OS on one of our servers, and since then
> > have been experiencing frequent stalls of the XFS filesystem on it.
> > Other filesystems on the machine seem to still respond fine while XFS
> > hangs. The stalls sometimes last for around 30 minutes, during which all
> > attempts to access that filesystem hang completely - after that, the
> > filesystem suddenly responds instantly again, as if there had never been
> > any problem. The dmesg is full of these messages while it stalls:
> >  XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)
> > These also occour from time to time without the filesystem stalling (or
> > at least it's not noticeable) - the messages appear about once in two
> > hours, the stalls about once a day.
> > 
> > Google did point me to some reports of these messages occouring at the
> > end of 2013, but the kernels in question should all have had the fixes
> > proposed back then - although one message back then suggested there were
> > more places where this problem could occour that were not fixed yet.
> > 
> > Kernels used were:
> > - Ubuntu 3.13.0-44  - shows stalls, according to
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1382333 has the fix
> > - Ubuntu 3.16.0-31  - shows stalls
> > - Ubuntu 3.2.0-various - no stalls in more than 1 year
> > We can actually still boot the machine with the 3.2.0 kernel, and it
> > will run absolutely fine, but as that kernel will not be supported
> > forever, I do not consider that a permanent solution.
> > 
> > The machine should not be low on memory, the disk array far from its
> > limits, and the I/O-load is mostly reads with very little writes, as
> > this is a public FTP server.
> > 
> > I have tried to collect some information, available at
> > https://grid.rrze.uni-erlangen.de/~unrz191/syslog-with-xfs-hangs.log
> > 
> 
> Thanks for the data. Some notes from the backtraces in the first
> instance:
> 
> - xfsaild is down in xlog_cil_force_lsn()->flush_work(). So it's trying
>   to push the log, but the workqueue worker is already running.
> - The workqueue worker is here:
> 
> 	[298163.482697] Workqueue: xfs-cil/dm-0 xlog_cil_push_work [xfs]
> 
> ... and it appears to be blocked on the ctx lock. This means either a
> transaction is completing or somebody else is pushing the cil.
> - Writeback and one or two other transactions are backed up waiting on
>   the ctx lock.
> - rsync is running a transaction completion (e.g., holding ctx lock) and
>   blocked on memory allocation:

Yup, that's prety much it. I suspect that we can do better here; I
think we might be ale to hoist the item formatting and memory
allocation outside the ctx lock - I'll need to do a little more than
have a quick browse of the code to determine if it's safe as we are
replacing log vectors in the when we are doing the allocation.

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com