On Wed, May 30, 2012 at 09:44:45PM +0300, Raz wrote:
> Hello
> We using 2.6.32 gentoo 64bit. and we're getting task_hung timeout stack.
>
> Our server uses direct IO. It reads files contents to buffers in
> memory and sends them by TCP. In addition, data is received
> by TCP and stored in files on disk.
> Most of the IO is reading data and sending it by TCP sockets.
>
> There are 4 threads reading data from disk into memory buffers. One
> thread per partition.
> There are about 20 threads reading data from the network and saving it
> to disk.
>
> In addition, there is an operation that is done on every file once it is
> downloaded. This operation maps data from file to memory. It is done
> in Java. I assume it is mmap. The mapping is very short.
>
> The bellow is the stack. Is this xfs bug ? root file system is xfs as
> well the data partition.
> Was a fix made in this area ? when was it ?
> thank you
> raz
>
>
INFO: task java:10449 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
java D 0007ffffffffe708 0 10449 10408 0x00000000
ffff88042acd1c28 0000000000000086 0000000000cd1b88 0000000000000000
0000000000000000 ffff88042acd1d7c 000000002c95c410 0000000000000000
0000000000015840 000000000000f9c8 ffff88042c95c410 ffff88042e4d96b0
Call Trace:
[<ffffffffa060bd4a>] ? kmem_zone_alloc+0xaa/0x110 [xfs]
[<ffffffff815f33f5>] __down_write_nested+0xa5/0x100
[<ffffffff815f346e>] __down_write+0x1e/0x40
[<ffffffff815f24ac>] down_write+0x1c/0x40
[<ffffffffa05e7e2c>] xfs_ilock+0x9c/0xb0 [xfs]
[<ffffffffa06093d6>] xfs_free_eofblocks+0x256/0x290 [xfs]
[<ffffffffa0609f1d>] xfs_release+0x14d/0x210 [xfs]
[<ffffffffa0612303>] xfs_file_release+0x23/0x40 [xfs]
[<ffffffff8114f1f9>] __fput+0xe9/0x210
[<ffffffff8114f34b>] fput+0x2b/0x50
[<ffffffff81124349>] remove_vma+0x49/0xb0
[<ffffffff81125a7a>] do_munmap+0x36a/0x3d0
[<ffffffff815f346e>] ? __down_write+0x1e/0x40
[<ffffffff81125b3c>] sys_munmap+0x5c/0xa0
[<ffffffff81013302>] system_call_fastpath+0x16/0x1b
Holding the mmap_sem, blocked on the iolock in exclusive mode waiting for IO to
complete.
java D ffff8803bc495b48 0 11768 10408 0x00000000
ffff8803bc495a58 0000000000000086 ffff8803bc4959a8 ffffffff815f38bc
ffff8803bc4959e8 000000005c14da46 ffff8803bc4959d8 ffffffff811300a2
0000000000015840 000000000000f9c8 ffff8803bc468000 ffff88042e4c2d60
Call Trace:
[<ffffffff815f38bc>] ? _spin_lock+0x1c/0x40
[<ffffffff811300a2>] ? swap_info_get+0x82/0x120
[<ffffffff811488c1>] ? mem_cgroup_commit_charge_swapin+0x21/0x40
[<ffffffff815f353d>] __down_read+0xad/0xfa
[<ffffffff815f24ec>] down_read+0x1c/0x40
[<ffffffff815f6e59>] do_page_fault+0x379/0x3a0
[<ffffffff815f4145>] page_fault+0x25/0x30
[<ffffffff810fe39c>] ? file_read_actor+0x6c/0x180
[<ffffffff810fe437>] ? file_read_actor+0x107/0x180
[<ffffffff81100d02>] generic_file_aio_read+0x492/0x6b0
[<ffffffffa06178f8>] xfs_read+0x138/0x2c0 [xfs]
[<ffffffffa06124ce>] xfs_file_aio_read+0x6e/0x90 [xfs]
[<ffffffff8114d341>] do_sync_read+0x101/0x160
[<ffffffff8108c4a0>] ? autoremove_wake_function+0x0/0x60
[<ffffffff81276b94>] ? security_file_permission+0x24/0x40
[<ffffffff8114dc64>] vfs_read+0xe4/0x1c0
[<ffffffff8114de5f>] sys_read+0x5f/0xc0
[<ffffffff81013302>] system_call_fastpath+0x16/0x1b
Holding the iolock in shared mode, taken a page fault during the
read() call and blocked on the mmap_sem.
IOWs, you're doing read() IO into a mmap()d buffer, and there's a
concurrent munmap() of another region of the same file that is open
under a different file descriptor. ABBA deadlock, and it's been
there for about 10 years. The problem is the munmap() call calling
fput() with the mmap_sem() held.
Here's the latest discussion thread about solving it:
https://lkml.org/lkml/2012/4/19/635
Right now your only option for avoiding the deadlock is "don't do
that". Soon it might be "upgrade to 3.x", but don't hold your
breath...
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|