question about xfs soft lockup stuck
符永涛
yongtaofu at gmail.com
Wed Jun 26 22:20:33 CDT 2013
Dear Brian and xfs experts,
Recently we have encountered an issue related to xfs. The log reports xfs
soft lockup stuck which is similar to this bug:
http://bugs.centos.org/view.php?id=6287
I wan to ask a question. Is this issue already been fixed in kernel 358 or
not(BZ#921961, BZ#896224)?
Thank you very much.
logs:
Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#0 stuck for 67s!
[xfslogd/0:2014]
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel: CPU 0
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel:
Jun 26 17:57:06 cqdx kernel: Pid: 2014, comm: xfslogd/0 Not tainted
2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55c>]
[<ffffffff814ec55c>] _spin_lock+0x1c/0x30
Jun 26 17:57:06 cqdx kernel: RSP: 0000:ffff88101465bd50 EFLAGS: 00000293
Jun 26 17:57:06 cqdx kernel: RAX: 0000000000005468 RBX: ffff88101465bd50
RCX: ffffe8efaf600e88
Jun 26 17:57:06 cqdx kernel: RDX: 0000000000005466 RSI: ffff882011755898
RDI: ffff88201955b100
Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: 0000000000000000
R09: 00000000ffffffff
Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
R12: ffff88101465bd70
Jun 26 17:57:06 cqdx kernel: R13: ffffffff8100b9ce R14: 0000000000000400
R15: 0000000000000094
Jun 26 17:57:06 cqdx kernel: FS: 0000000000000000(0000)
GS:ffff880028200000(0000) knlGS:0000000000000000
Jun 26 17:57:06 cqdx kernel: CS: 0010 DS: 0018 ES: 0018 CR0:
000000008005003b
Jun 26 17:57:06 cqdx kernel: CR2: 00007f9e001dc0a0 CR3: 000000201311e000
CR4: 00000000000406f0
Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Jun 26 17:57:06 cqdx kernel: Process xfslogd/0 (pid: 2014, threadinfo
ffff88101465a000, task ffff881015b75500)
Jun 26 17:57:06 cqdx kernel: Stack:
Jun 26 17:57:06 cqdx kernel: ffff88101465bd80 ffffffffa02a2b3b
0000000000000001 ffff8820137c7300
Jun 26 17:57:06 cqdx kernel: <d> ffff882016daee40 ffff882017e03800
ffff88101465bda0 ffffffffa02a2adc
Jun 26 17:57:06 cqdx kernel: <d> ffff882016d330a8 ffff882016daee40
ffff88101465be10 ffffffffa02a2d51
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
xfs_buf_iodone+0x2b/0x60 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
xfs_buf_do_callbacks+0x3c/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
xfs_buf_iodone_work+0x0/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
xfs_buf_iodone_work+0x26/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
worker_thread+0x170/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
autoremove_wake_function+0x0/0x40
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jun 26 17:57:06 cqdx kernel: Code: 81 2f 00 00 00 01 74 05 e8 72 8c d8 ff
c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8
10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40
00 55
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
xfs_buf_iodone+0x2b/0x60 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
xfs_buf_do_callbacks+0x3c/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
xfs_buf_iodone_work+0x0/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
xfs_buf_iodone_work+0x26/0x50 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
worker_thread+0x170/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
autoremove_wake_function+0x0/0x40
Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#15 stuck for 67s!
[glusterfsd:3245]
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel: CPU 15
Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun 26 17:57:06 cqdx kernel:
Jun 26 17:57:06 cqdx kernel: Pid: 3245, comm: glusterfsd Not tainted
2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55e>]
[<ffffffff814ec55e>] _spin_lock+0x1e/0x30
Jun 26 17:57:06 cqdx kernel: RSP: 0018:ffff880dceb5d528 EFLAGS: 00000297
Jun 26 17:57:06 cqdx kernel: RAX: 00000000000087eb RBX: ffff880dceb5d528
RCX: ffff8820137923e0
Jun 26 17:57:06 cqdx kernel: RDX: 00000000000087ea RSI: 0000000000006e5c
RDI: ffff882016d330a8
Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: ffff880dceb5d674
R09: ffff880dceb5d670
Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
R12: 0000000000000000
Jun 26 17:57:06 cqdx kernel: R13: 0000000000000cb5 R14: ffff8810788166e8
R15: ffff882016ea9400
Jun 26 17:57:06 cqdx kernel: FS: 00007f6cf3fff700(0000)
GS:ffff8810788e0000(0000) knlGS:0000000000000000
Jun 26 17:57:06 cqdx kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jun 26 17:57:06 cqdx kernel: CR2: 0000000000481046 CR3: 000000201311e000
CR4: 00000000000406e0
Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
Jun 26 17:57:06 cqdx kernel: Process glusterfsd (pid: 3245, threadinfo
ffff880dceb5c000, task ffff8810191ecaa0)
Jun 26 17:57:06 cqdx kernel: Stack:
Jun 26 17:57:06 cqdx kernel: ffff880dceb5d5d8 ffffffffa02c27a8
ffff881078816680 ffff880dceb5d670
Jun 26 17:57:06 cqdx kernel: <d> ffff880dceb5d674 ffff880dceb5d668
00006e5c0000000f 0000000000000001
Jun 26 17:57:06 cqdx kernel: <d> 00000000000172a0 ffff882016d33080
ffff880dceb5d588 0000000000000082
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
try_to_wake_up+0x24c/0x3e0
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
xlog_cil_push+0x261/0x3d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
xlog_cil_force_lsn+0x115/0x120 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
xfs_bmap_add_extent+0xad/0x3c0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2308>] ?
_xfs_log_force+0x68/0x240 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2718>] ? xfs_log_force+0x18/0x40
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d99d0>] ?
xfs_buf_cond_lock+0x90/0xc0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dadb3>] ?
_xfs_buf_find+0xe3/0x280 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dafbb>] ? xfs_buf_get+0x6b/0x1a0
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d16d8>] ?
xfs_trans_get_buf+0xe8/0x180 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a4a67>] ?
xfs_da_do_buf+0x4c7/0x770 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a5fda>] ?
xfs_da_get_buf+0x2a/0x30 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
xfs_dir2_data_init+0x4d/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
xfs_dir2_data_init+0x4d/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a8c23>] ?
xfs_dir2_sf_to_block+0xe3/0x5e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b5ce6>] ? xfs_iunlock+0x86/0x100
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118ecaa>] ? dput+0x9a/0x150
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b0c18>] ?
xfs_dir2_sf_addname+0x458/0x5b0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d6687>] ?
kmem_zone_alloc+0x77/0xf0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a81bc>] ?
xfs_dir_createname+0x14c/0x1b0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d3390>] ? xfs_link+0x1f0/0x300
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff81195560>] ?
mntput_no_expire+0x30/0x110
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118e7a7>] ? __d_lookup+0xa7/0x150
Jun 26 17:57:06 cqdx kernel: [<ffffffff8107721a>] ? capable+0x2a/0x60
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02e1120>] ? xfs_vn_link+0x40/0x70
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8118379c>] ? vfs_link+0x11c/0x190
Jun 26 17:57:06 cqdx kernel: [<ffffffff81182aaa>] ? lookup_hash+0x3a/0x50
Jun 26 17:57:06 cqdx kernel: [<ffffffff81186b71>] ? sys_linkat+0x141/0x160
Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b83b>] ? vfs_stat+0x1b/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b864>] ? sys_newstat+0x24/0x50
Jun 26 17:57:06 cqdx kernel: [<ffffffff81186bae>] ? sys_link+0x1e/0x20
Jun 26 17:57:06 cqdx kernel: [<ffffffff8100b072>] ?
system_call_fastpath+0x16/0x1b
Jun 26 17:57:06 cqdx kernel: Code: 00 00 00 01 74 05 e8 72 8c d8 ff c9 c3
55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39
c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55
48 89
Jun 26 17:57:06 cqdx kernel: Call Trace:
Jun 26 17:57:06 cqdx kernel: [<ffffffff8106210b>] ?
enqueue_task_fair+0xfb/0x100
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
try_to_wake_up+0x24c/0x3e0
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
[xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
xlog_cil_push+0x261/0x3d0 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
xlog_cil_force_lsn+0x115/0x120 [xfs]
Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
xfs_bmap_add_extent+0xad/0x3c0 [xfs]
--
符永涛
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.sgi.com/pipermail/xfs/attachments/20130627/730e5452/attachment-0001.html>
More information about the xfs
mailing list