xfs
[Top] [All Lists]

XFS hung on 2.6.33.3 kernel

To: xfs@xxxxxxxxxxx
Subject: XFS hung on 2.6.33.3 kernel
From: Ilia Mirkin <imirkin@xxxxxxxxxxxx>
Date: Sat, 17 Jul 2010 00:01:11 -0400
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=bj5SFyROR0Q9wJY4wsdxVOfCNBvnXSku2ieADjaL2Qo=; b=K/UiYxuUxOc3CUOnjGNs/A99nFv6J6moJ2X5J5FfYU6LAEIua3OjC5K8w7MguVGp9+ 9Mql5MeNVCGFwqI1+E4kWEPWkqYG44kyJYzA+ne+yDF5U7p58KP94XQhy+bOL5Sy+aBw 1TjT7iGUfDpdJOqdLaCmKsvzlNzey/qSam0xA=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=NWlFDNX+EgflOiHwQb5wCszIb8p0PeQYpd47VBdtjzDxNb3Ji0KuRvxji/G71BJGLY n01n4hTCBJ7czLMVjVIWFXA9fja2jXlwG3uedkkW2/K8TmYc6RAtjxQ/lDQ6y/MPk1Zm ekk+MHgen2jml7Yhytwcw+ZpgPG6CeuAmtiro=
Sender: ibmirkin@xxxxxxxxx
Hi XFS developers,

I ended up getting this on a 2.6.33.3 kernel. I realize this is not
the latest, but I've only gotten this once, so it will not be possible
to easily determine whether this is gone in the latest release or not.

The details of the situation is that there is an XFS filesystem on a
hardware RAID10 array (directly on the device, no partition table),
and it exclusively houses a mysql (5.0) database. The database gets
backed up nightly, which is basically a glorified tar'ing of most of
the partition. The non-mysql process you see there are related to a
mysql backup program (which coincidentally triggers WARN's in xfs
code, as per http://marc.info/?l=linux-xfs&m=127632333825584&w=2;
however it sounded like those WARN's were not warning about an
in-kernel problems; curiously, the last WARN's in the kernel log
appear to have been from the prior day's backup run, not the one that
triggered the hang). There's also a sync that I ran as a last-ditch
effort at saving the system. On reboot, everything came back up OK.

[

For reference, the warns look like this:

[5444728.784257] ------------[ cut here ]------------
[5444728.784269] WARNING: at fs/xfs/linux-2.6/xfs_lrw.c:651
xfs_write+0x3dc/0x784()
[5444728.784272] Hardware name: PowerEdge R710
[5444728.784275] Modules linked in: nfsd cifs iTCO_wdt iTCO_vendor_support
[5444728.784286] Pid: 7765, comm: mysqld Tainted: G        W
2.6.33-gentoo-r2 #1
[5444728.784287] Call Trace:
[5444728.784291]  [<ffffffff8120da46>] ? xfs_write+0x3dc/0x784
[5444728.784295]  [<ffffffff81038733>] warn_slowpath_common+0x77/0xa4
[5444728.784297]  [<ffffffff8103876f>] warn_slowpath_null+0xf/0x11
[5444728.784299]  [<ffffffff8120da46>] xfs_write+0x3dc/0x784
[5444728.784303]  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
[5444728.784306]  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
[5444728.784310]  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
[5444728.784312]  [<ffffffff810d8624>] vfs_write+0xab/0x105
[5444728.784314]  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
[5444728.784319]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
[5444728.784321] ---[ end trace 8b0c2a6e5e867a01 ]---

]

While the system was stuck, vmstat did not report any block-in/out on
the device. I have rrd's of those if necessary. Let me know if I might
be able to provide any other useful info, or if you think that this
sort of thing should be fixed in the latest linux kernel release. I
saw that there have been a lot of XFS updates since 2.6.33, but none
stood out as addressing this type of situation.

Here is the output of "echo w > /proc/sysrq-trigger":

 SysRq : Show Blocked State
   task                        PC stack   pid father
 xfsdatad/7    D ffff88123885d3e0  6792   772      2 0x00000000
  ffff88123887fd90 0000000000000046 0000000000000000 ffff88123887fd5c
  0000000000000000 000000000000de78 ffff88123887ffd8 0000000000013c00
  ffff88123887ffd8 0000000000013c00 ffff88123885d3e0 ffff881239d23d00
 Call Trace:
  [<ffffffff81045f9c>] ? __mod_timer+0x13d/0x14f
  [<ffffffff814eb81c>] schedule_timeout+0x98/0xbf
  [<ffffffff81045b27>] ? process_timeout+0x0/0xb
  [<ffffffff814eb85c>] schedule_timeout_uninterruptible+0x19/0x1b
  [<ffffffff81206512>] xfs_end_io+0x7d/0x8c
  [<ffffffff8104de65>] worker_thread+0x18d/0x222
  [<ffffffff81206495>] ? xfs_end_io+0x0/0x8c
  [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
  [<ffffffff8104dcd8>] ? worker_thread+0x0/0x222
  [<ffffffff81051336>] kthread+0x7d/0x85
  [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
  [<ffffffff810512b9>] ? kthread+0x0/0x85
  [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
 mysqld        D ffff880938821380  4696  7750      1 0x00000000
  ffff880650e01cb8 0000000000000082 00715f4f00000001 0000000000000000
  ffff88090a0bd3e0 000000000000de78 ffff880650e01fd8 0000000000013c00
  ffff880650e01fd8 0000000000013c00 ffff88090a0bd3e0 ffff88090a0bf260
 Call Trace:
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4760  7751      1 0x00000000
  ffff880650e6bcb8 0000000000000082 007cd91100000001 ffff88090a0bcc78
  ffff88095fc13c00 000000000000de78 ffff880650e6bfd8 0000000000013c00
  ffff880650e6bfd8 0000000000013c00 ffff88090a0bf260 ffff88090a0b8000
 Call Trace:
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4768  7752      1 0x00000000
  ffff880650e6dcb8 0000000000000082 0085722500000001 ffff88090a0b8038
  ffff880028273c00 000000000000de78 ffff880650e6dfd8 0000000000013c00
  ffff880650e6dfd8 0000000000013c00 ffff88090a0bcc40 ffff88090a0b96e0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4768  7755      1 0x00000000
  ffff88029a60fcb8 0000000000000082 0000000000000000 ffff88090a0b9718
  ffff88095fc13c00 000000000000de78 ffff88029a60ffd8 0000000000013c00
  ffff88029a60ffd8 0000000000013c00 ffff88090a0b8000 ffff881239d23d00
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4696  7756      1 0x00000000
  ffff88029a633cb8 0000000000000082 0091c4f90a0b9718 ffff88090a0bf298
  ffff880028273c00 000000000000de78 ffff88029a633fd8 0000000000013c00
  ffff88029a633fd8 0000000000013c00 ffff88090a0be320 ffff8812384c8000
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4696  7757      1 0x00000000
  ffff8801b4fdfcb8 0000000000000082 0000000000000000 ffff88090a0beaf8
  ffff88095fc33c00 000000000000de78 ffff8801b4fdffd8 0000000000013c00
  ffff8801b4fdffd8 0000000000013c00 ffff88090a0b9e80 ffff880939c9adc0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7758      1 0x00000000
  ffff8801b4f1bc08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff8801b4f1bb88 000000000000de78 ffff8801b4f1bfd8 0000000000013c00
  ffff8801b4f1bfd8 0000000000013c00 ffff88090a0bb560 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4648  7759      1 0x00000000
  ffff88029a627c08 0000000000000082 000000009a627b88 ffffffff8102cbe0
  ffff88029a627b88 000000000000de78 ffff88029a627fd8 0000000000013c00
  ffff88029a627fd8 0000000000013c00 ffff88090a0ba620 ffff8812116787a0
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810ff862>] ? __fsnotify_parent+0xc7/0xd3
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7760      1 0x00000000
  ffff88029a611c08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff88029a611b88 000000000000de78 ffff88029a611fd8 0000000000013c00
  ffff88029a611fd8 0000000000013c00 ffff88090a0badc0 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff810033ce>] ? apic_timer_interrupt+0xe/0x20
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7761      1 0x00000000
  ffff88029a60bc08 0000000000000082 0000000000000000 ffff88002826de78
  0000000249c7788e 000000000000de78 ffff88029a60bfd8 0000000000013c00
  ffff88029a60bfd8 0000000000013c00 ffff88090a0bdb80 ffff881239d23d00
 Call Trace:
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff81101c00>] ? inotify_dentry_parent_queue_event+0x80/0x8c
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7762      1 0x00000000
  ffff8806357d5c08 0000000000000082 0000000000000000 ffffffff8102cbe0
  ffff8806357d5b88 000000000000de78 ffff8806357d5fd8 0000000000013c00
  ffff8806357d5fd8 0000000000013c00 ffff88090a0bc4a0 ffff880939ce8000
 Call Trace:
  [<ffffffff8102cbe0>] ? check_preempt_wakeup+0xde/0x15d
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff81101c00>] ? inotify_dentry_parent_queue_event+0x80/0x8c
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D ffff880938821380  4648  7763      1 0x00000000
  ffff8801b4febbb8 0000000000000082 ffff8801b4febb58 ffff880937ea2df8
  000000000000de78 000000000000de78 ffff8801b4febfd8 0000000000013c00
  ffff8801b4febfd8 0000000000013c00 ffff880937ea6ac0 ffff8812116787a0
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ec6fe>] rwsem_down_write_failed+0x20/0x2a
  [<ffffffff812504b3>] call_rwsem_down_write_failed+0x13/0x20
  [<ffffffff814ec005>] ? down_write+0x2d/0x31
  [<ffffffff811ebd14>] xfs_ilock+0x23/0x94
  [<ffffffff8120d92f>] xfs_write+0x2c5/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4648  7764      1 0x00000000
  ffff880287b4dc08 0000000000000082 0000000000000000 ffff880287b4dec8
  ffff880287b4db78 000000000000de78 ffff880287b4dfd8 0000000000013c00
  ffff880287b4dfd8 0000000000013c00 ffff880937ea4c40 ffff880939ce8000
 Call Trace:
  [<ffffffff814ec6ac>] ? rwsem_down_failed_common+0x96/0xc8
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120d922>] xfs_write+0x2b8/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff814ecb8e>] ? common_interrupt+0xe/0x13
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000002  4648  7765      1 0x00000000
  ffff880287b4ba78 0000000000000082 0000000000000000 0000000000000001
  0000000000000096 000000000000de78 ffff880287b4bfd8 0000000000013c00
  ffff880287b4bfd8 0000000000013c00 ffff880937ea2dc0 ffff88123885d3e0
 Call Trace:
  [<ffffffff8104e062>] ? insert_work+0x92/0x9e
  [<ffffffff8105945a>] ? ktime_get_ts+0xb1/0xbe
  [<ffffffff814eb4b9>] io_schedule+0x7b/0xc1
  [<ffffffff8109fc02>] sync_page+0x41/0x45
  [<ffffffff814eb8d9>] __wait_on_bit_lock+0x45/0x8c
  [<ffffffff8109fbc1>] ? sync_page+0x0/0x45
  [<ffffffff8109fb84>] __lock_page+0x63/0x6a
  [<ffffffff810517e3>] ? wake_bit_function+0x0/0x2a
  [<ffffffff810a82de>] lock_page+0x24/0x28
  [<ffffffff810a8364>] invalidate_inode_pages2_range+0x82/0x2af
  [<ffffffff8120646a>] ? xfs_vm_direct_IO+0x90/0xbb
  [<ffffffff812066a8>] ? xfs_get_blocks_direct+0x0/0x17
  [<ffffffff812062c8>] ? xfs_end_io_direct+0x0/0x4e
  [<ffffffff810a085d>] generic_file_direct_write+0xf5/0x14b
  [<ffffffff8120db02>] xfs_write+0x498/0x784
  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
  [<ffffffff810d8624>] vfs_write+0xab/0x105
  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4240  7787      1 0x00000000
  ffff8801b4f67cb8 0000000000000082 0000000000000000 0000000000000296
  ffff8801b4f67c58 000000000000de78 ffff8801b4f67fd8 0000000000013c00
  ffff8801b4f67fd8 0000000000013c00 ffff8809388d6320 ffff880939ce8000
 Call Trace:
  [<ffffffff8105ee38>] ? wake_futex+0x3b/0x57
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff811f6f46>] ? xfs_log_reserve+0x101/0x10a
  [<ffffffff81204b5c>] ? xfs_fsync+0x135/0x16a
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3776 22038      1 0x00000000
  ffff880da3973cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880da3973c28 000000000000de78 ffff880da3973fd8 0000000000013c00
  ffff880da3973fd8 0000000000013c00 ffff88097f63c4a0 ffff881239d20f40
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3776 29433      1 0x00000000
  ffff881236b49cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff881236b49c28 000000000000de78 ffff881236b49fd8 0000000000013c00
  ffff881236b49fd8 0000000000013c00 ffff880dc6e244a0 ffff881239d20f40
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  4544  5673      1 0x00000000
  ffff880fa9189cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880d751f0d00 000000000000de78 ffff880fa9189fd8 0000000000013c00
  ffff880fa9189fd8 0000000000013c00 ffff880bf12c2620 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff8102832b>] ? pick_next_task+0x22/0x3e
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  3632  6093      1 0x00000000
  ffff880e39e61cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880e39e61c28 000000000000de78 ffff880e39e61fd8 0000000000013c00
  ffff880e39e61fd8 0000000000013c00 ffff8811f78d1e80 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6840      1 0x00000000
  ffff880e3d89fcb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880e3d89fc28 000000000000de78 ffff880e3d89ffd8 0000000000013c00
  ffff880e3d89ffd8 0000000000013c00 ffff8809aba6f260 ffff880939c9c4a0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6841      1 0x00000000
  ffff8803f0d73cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff8803f0d73c28 000000000000de78 ffff8803f0d73fd8 0000000000013c00
  ffff8803f0d73fd8 0000000000013c00 ffff8807c98444a0 ffff880939c9f260
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5328  6883      1 0x00000000
  ffff880a3994fcb8 0000000000000082 0000000000000000 ffffffff813fc68a
  0000000000000005 000000000000de78 ffff880a3994ffd8 0000000000013c00
  ffff880a3994ffd8 0000000000013c00 ffff8811567cdb80 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff810ed26a>] ? mntput_no_expire+0x24/0xe0
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5312  6887      1 0x00000000
  ffff880fb56c9cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff881039c8a080 000000000000de78 ffff880fb56c9fd8 0000000000013c00
  ffff880fb56c9fd8 0000000000013c00 ffff880e3659cc40 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6888      1 0x00000000
  ffff88030d555cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff880671892d80 000000000000de78 ffff88030d555fd8 0000000000013c00
  ffff88030d555fd8 0000000000013c00 ffff88071109d3e0 ffff880939c9adc0
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5336  6890      1 0x00000000
  ffff880690c15cb8 0000000000000082 0000000000000000 ffff8812116787d8
  ffff88095fc93c00 000000000000de78 ffff880690c15fd8 0000000000013c00
  ffff880690c15fd8 0000000000013c00 ffff8808b9585b80 ffffffff81a0e020
 Call Trace:
  [<ffffffff81027beb>] ? activate_task+0x28/0x31
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 mysqld        D 0000000000000000  5312  6899      1 0x00000000
  ffff88090a091cb8 0000000000000082 0000000000000000 ffffffff813fc68a
  ffff88090a091c28 000000000000de78 ffff88090a091fd8 0000000000013c00
  ffff88090a091fd8 0000000000013c00 ffff88085196cc40 ffffffff81a0e020
 Call Trace:
  [<ffffffff813fc68a>] ? release_sock+0xb2/0xba
  [<ffffffff814ebc5a>] __mutex_lock_common+0x139/0x1a2
  [<ffffffff814ebcd7>] __mutex_lock_slowpath+0x14/0x16
  [<ffffffff814ebda5>] mutex_lock+0x1e/0x35
  [<ffffffff8120e1bf>] xfs_read+0x100/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff810d7dc4>] ? do_sync_read+0xc0/0x106
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d88c9>] sys_pread64+0x5c/0x7d
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 tar4ibd       D 0000000000000000  4552  6721   6720 0x00000004
  ffff88016a7c3740 0000000000000086 0000000000000000 ffff88095fc13c00
  0000000000000001 000000000000de78 ffff88016a7c3fd8 0000000000013c00
  ffff88016a7c3fd8 0000000000013c00 ffff8806b39344a0 ffff881239d253e0
 Call Trace:
  [<ffffffff81034837>] ? wake_up_process+0x10/0x12
  [<ffffffff8124c328>] ? __rwsem_do_wake+0x80/0x152
  [<ffffffff814ec6ac>] rwsem_down_failed_common+0x96/0xc8
  [<ffffffff81244595>] ? cfq_close_cooperator+0xd9/0x184
  [<ffffffff814ec72e>] rwsem_down_read_failed+0x26/0x30
  [<ffffffff81250484>] call_rwsem_down_read_failed+0x14/0x30
  [<ffffffff814ec028>] ? down_read+0x1f/0x23
  [<ffffffff811ebd4e>] xfs_ilock+0x5d/0x94
  [<ffffffff811ebdae>] xfs_ilock_map_shared+0x29/0x2f
  [<ffffffff811f231b>] xfs_iomap+0x9f/0x39c
  [<ffffffff81239215>] ? __blk_run_queue+0x4a/0x70
  [<ffffffff81247a91>] ? cpumask_next_and+0x2b/0x3c
  [<ffffffff812065bd>] __xfs_get_blocks+0x71/0x15c
  [<ffffffff812066d1>] xfs_get_blocks+0x12/0x14
  [<ffffffff810fec23>] do_mpage_readpage+0x173/0x467
  [<ffffffff810b2234>] ? __inc_zone_page_state+0x25/0x27
  [<ffffffff8109fecc>] ? add_to_page_cache_locked+0x82/0xc6
  [<ffffffff810ff04a>] mpage_readpages+0xd6/0x11c
  [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
  [<ffffffff812066bf>] ? xfs_get_blocks+0x0/0x14
  [<ffffffff81207724>] xfs_vm_readpages+0x1a/0x1c
  [<ffffffff810a72c4>] __do_page_cache_readahead+0x10c/0x1a2
  [<ffffffff810a7376>] ra_submit+0x1c/0x20
  [<ffffffff810a7606>] ondemand_readahead+0x1b2/0x1c5
  [<ffffffff810a76f3>] page_cache_sync_readahead+0x38/0x3a
  [<ffffffff810a0e0c>] generic_file_aio_read+0x229/0x582
  [<ffffffff8120e262>] xfs_read+0x1a3/0x1f9
  [<ffffffff8120a574>] xfs_file_aio_read+0x58/0x5c
  [<ffffffff810d7dc4>] do_sync_read+0xc0/0x106
  [<ffffffff81032879>] ? finish_task_switch+0x43/0xac
  [<ffffffff810d8813>] vfs_read+0xa8/0x102
  [<ffffffff810d8931>] sys_read+0x47/0x70
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
 sync          D 0000000000000000  4496  8295   8248 0x00000000
  ffff8805b78b1cd8 0000000000000086 0000000000000000 0000000000013c00
  ffff8809388acac0 000000000000de78 ffff8805b78b1fd8 0000000000013c00
  ffff8805b78b1fd8 0000000000013c00 ffff8801002dadc0 ffff881239d23d00
 Call Trace:
  [<ffffffff810519e4>] ? prepare_to_wait+0x70/0x7b
  [<ffffffff81207928>] xfs_ioend_wait+0x83/0x9b
  [<ffffffff810517ab>] ? autoremove_wake_function+0x0/0x38
  [<ffffffff8121071b>] ? xfs_sync_inode_valid+0x68/0x9a
  [<ffffffff81210aaa>] xfs_sync_inode_data+0x98/0xbf
  [<ffffffff81210bf0>] xfs_inode_ag_walk+0x95/0xd4
  [<ffffffff81210a12>] ? xfs_sync_inode_data+0x0/0xbf
  [<ffffffff8109fdac>] ? wait_on_page_bit+0x6e/0x75
  [<ffffffff81210c82>] xfs_inode_ag_iterator+0x53/0x81
  [<ffffffff81210a12>] ? xfs_sync_inode_data+0x0/0xbf
  [<ffffffff81210e49>] xfs_sync_data+0x26/0x4d
  [<ffffffff81210ebf>] xfs_quiesce_data+0x1e/0x57
  [<ffffffff8120e841>] xfs_fs_sync_fs+0x26/0xd7
  [<ffffffff8111a159>] ? vfs_quota_sync+0x196/0x1a7
  [<ffffffff8111ba70>] sync_quota_sb+0x4e/0xfc
  [<ffffffff810f5e64>] __sync_filesystem+0x41/0x72
  [<ffffffff810f5f37>] sync_filesystems+0xa2/0xee
  [<ffffffff810f5fd4>] sys_sync+0x1c/0x2e
  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b


-- 
Ilia Mirkin
imirkin@xxxxxxxxxxxx

<Prev in Thread] Current Thread [Next in Thread>