On Sat, Mar 07, 2015 at 08:51:50AM +0100, Michael Meier wrote:
> We've recently upgraded the OS on one of our servers, and since then
> have been experiencing frequent stalls of the XFS filesystem on it.
> Other filesystems on the machine seem to still respond fine while XFS
> hangs. The stalls sometimes last for around 30 minutes, during which all
> attempts to access that filesystem hang completely - after that, the
> filesystem suddenly responds instantly again, as if there had never been
> any problem. The dmesg is full of these messages while it stalls:
> XFS: possible memory allocation deadlock in kmem_alloc (mode:0x8250)
> These also occour from time to time without the filesystem stalling (or
> at least it's not noticeable) - the messages appear about once in two
> hours, the stalls about once a day.
> Google did point me to some reports of these messages occouring at the
> end of 2013, but the kernels in question should all have had the fixes
> proposed back then - although one message back then suggested there were
> more places where this problem could occour that were not fixed yet.
> Kernels used were:
> - Ubuntu 3.13.0-44 - shows stalls, according to
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1382333 has the fix
> - Ubuntu 3.16.0-31 - shows stalls
> - Ubuntu 3.2.0-various - no stalls in more than 1 year
> We can actually still boot the machine with the 3.2.0 kernel, and it
> will run absolutely fine, but as that kernel will not be supported
> forever, I do not consider that a permanent solution.
> The machine should not be low on memory, the disk array far from its
> limits, and the I/O-load is mostly reads with very little writes, as
> this is a public FTP server.
> I have tried to collect some information, available at
Thanks for the data. Some notes from the backtraces in the first
- xfsaild is down in xlog_cil_force_lsn()->flush_work(). So it's trying
to push the log, but the workqueue worker is already running.
- The workqueue worker is here:
[298163.482697] Workqueue: xfs-cil/dm-0 xlog_cil_push_work [xfs]
... and it appears to be blocked on the ctx lock. This means either a
transaction is completing or somebody else is pushing the cil.
- Writeback and one or two other transactions are backed up waiting on
the ctx lock.
- rsync is running a transaction completion (e.g., holding ctx lock) and
blocked on memory allocation:
rsync D ffff88103f893440 0 44446 43197 0x00000000
ffff8809e4f7b9f0 0000000000000086 ffff880801a15bb0 ffff8809e4f7bfd8
0000000000013440 0000000000013440 ffff8810146428c0 ffff881013dd8000
ffff8809e4f7ba20 00000001046f17ea ffff881013dd8000 000000000000d158
[<ffffffff8107a420>] ? ftrace_raw_event_tick_stop+0xc0/0xc0
[<ffffffff810b4d10>] ? prepare_to_wait_event+0x100/0x100
[<ffffffffc01deaac>] kmem_alloc+0x6c/0xf0 [xfs]
[<ffffffffc022399f>] xfs_log_commit_cil+0x34f/0x470 [xfs]
[<ffffffffc01de37c>] xfs_trans_commit+0x11c/0x230 [xfs]
[<ffffffffc0212c81>] xfs_rename+0x601/0x670 [xfs]
[<ffffffffc01d41c2>] xfs_vn_rename+0x82/0x90 [xfs]
[<ffffffff811d6451>] ? __sb_end_write+0x31/0x60
[<ffffffff811f208f>] ? mnt_drop_write+0x1f/0x30
[<ffffffff811f34b4>] ? mntput+0x24/0x40
[<ffffffff811ea6ac>] ? dput+0x4c/0x180
[<ffffffff811f34b4>] ? mntput+0x24/0x40
[<ffffffff811dd39e>] ? path_put+0x1e/0x30
... so that appears to hold everything else up.
This looks potentially related to the ongoing transaction context memory
allocation discussion, as this code implements a tight retry loop with
time-based task waits and "no fail" allocations. This is also the source
of the "possible memory allocation deadlock" warning.
Dave might be able to comment a bit further on that. I'm not totally
clear on the mm interaction here and if/what a workaround might be. It
might be a good idea to grab the meminfo data when the stall is actually
Considering this is a large memory box (64g), I wonder if some vm tuning
might help mitigate this behavior..? For example, increase
/proc/sys/vm/min_free_kbytes in hopes of allowing more memory for these
allocations when under pressure, or tune down the
dirty_ratio/dirty_background_ratio thresholds to more aggressively get
data onto disk..?
> Michael Meier, Zentrale Systeme
> Friedrich-Alexander-Universitaet Erlangen-Nuernberg
> Regionales Rechenzentrum Erlangen
> Martensstrasse 1, 91058 Erlangen, Germany
> Tel.: +49 9131 85-28973, Fax: +49 9131 302941
> xfs mailing list