xfs
[Top] [All Lists]

Re: question about xfs soft lockup stuck

To: çææ <yongtaofu@xxxxxxxxx>
Subject: Re: question about xfs soft lockup stuck
From: Brian Foster <bfoster@xxxxxxxxxx>
Date: Thu, 27 Jun 2013 10:02:55 -0400
Cc: "xfs@xxxxxxxxxxx" <xfs@xxxxxxxxxxx>
Delivered-to: xfs@xxxxxxxxxxx
In-reply-to: <CADFMGu+eJ+skfYHZY8FVV8y2fqjEuRR4LAGPE2BMgYCoMu73wg@xxxxxxxxxxxxxx>
References: <CADFMGu+eJ+skfYHZY8FVV8y2fqjEuRR4LAGPE2BMgYCoMu73wg@xxxxxxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130514 Thunderbird/17.0.6
On 06/26/2013 11:20 PM, çææ wrote:
> Dear Brian and xfs experts,
> 
> Recently we have encountered an issue related to xfs. The log reports xfs
> soft lockup stuck which is similar to this bug:
> http://bugs.centos.org/view.php?id=6287
> 
> I wan to ask a question. Is this issue already been fixed in kernel 358 or
> not(BZ#921961, BZ#896224)?
> 

Both of those bz's have metadata that shows the fixed-in release. The
latter is a backport that notes the fix being pulled back into
kernel-2.6.32-358.4.1.el6.

Brian

> 
> Thank you very much.
> 
> logs:
> Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#0 stuck for 67s!
> [xfslogd/0:2014]
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel: CPU 0
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel:
> Jun 26 17:57:06 cqdx kernel: Pid: 2014, comm: xfslogd/0 Not tainted
> 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
> Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55c>]
> [<ffffffff814ec55c>] _spin_lock+0x1c/0x30
> Jun 26 17:57:06 cqdx kernel: RSP: 0000:ffff88101465bd50  EFLAGS: 00000293
> Jun 26 17:57:06 cqdx kernel: RAX: 0000000000005468 RBX: ffff88101465bd50
> RCX: ffffe8efaf600e88
> Jun 26 17:57:06 cqdx kernel: RDX: 0000000000005466 RSI: ffff882011755898
> RDI: ffff88201955b100
> Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: 0000000000000000
> R09: 00000000ffffffff
> Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
> R12: ffff88101465bd70
> Jun 26 17:57:06 cqdx kernel: R13: ffffffff8100b9ce R14: 0000000000000400
> R15: 0000000000000094
> Jun 26 17:57:06 cqdx kernel: FS:  0000000000000000(0000)
> GS:ffff880028200000(0000) knlGS:0000000000000000
> Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0018 ES: 0018 CR0:
> 000000008005003b
> Jun 26 17:57:06 cqdx kernel: CR2: 00007f9e001dc0a0 CR3: 000000201311e000
> CR4: 00000000000406f0
> Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Jun 26 17:57:06 cqdx kernel: Process xfslogd/0 (pid: 2014, threadinfo
> ffff88101465a000, task ffff881015b75500)
> Jun 26 17:57:06 cqdx kernel: Stack:
> Jun 26 17:57:06 cqdx kernel: ffff88101465bd80 ffffffffa02a2b3b
> 0000000000000001 ffff8820137c7300
> Jun 26 17:57:06 cqdx kernel: <d> ffff882016daee40 ffff882017e03800
> ffff88101465bda0 ffffffffa02a2adc
> Jun 26 17:57:06 cqdx kernel: <d> ffff882016d330a8 ffff882016daee40
> ffff88101465be10 ffffffffa02a2d51
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
> xfs_buf_iodone+0x2b/0x60 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
> xfs_buf_do_callbacks+0x3c/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
> xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
> xfs_buf_iodone_work+0x0/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
> xfs_buf_iodone_work+0x26/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
> worker_thread+0x170/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
> autoremove_wake_function+0x0/0x40
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> Jun 26 17:57:06 cqdx kernel: Code: 81 2f 00 00 00 01 74 05 e8 72 8c d8 ff
> c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8
> 10 39 c2 74 0e <f3> 90 0f b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40
> 00 55
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2b3b>] ?
> xfs_buf_iodone+0x2b/0x60 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2adc>] ?
> xfs_buf_do_callbacks+0x3c/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a2d51>] ?
> xfs_buf_iodone_callbacks+0x51/0x270 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db190>] ?
> xfs_buf_iodone_work+0x0/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02db1b6>] ?
> xfs_buf_iodone_work+0x26/0x50 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108b120>] ?
> worker_thread+0x170/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090990>] ?
> autoremove_wake_function+0x0/0x40
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8108afb0>] ? worker_thread+0x0/0x2a0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090626>] ? kthread+0x96/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81090590>] ? kthread+0x0/0xa0
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
> Jun 26 17:57:06 cqdx kernel: BUG: soft lockup - CPU#15 stuck for 67s!
> [glusterfsd:3245]
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel: CPU 15
> Jun 26 17:57:06 cqdx kernel: Modules linked in: fuse ipv6 xfs exportfs
> vhost_net macvtap macvlan tun kvm_intel kvm power_meter dcdbas microcode
> sb_edac edac_core iTCO_wdt iTCO_vendor_support shpchp igb sg ixgbe dca mdio
> ext4 mbcache jbd2 sd_mod crc_t10dif wmi megaraid_sas dm_mirror
> dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
> Jun 26 17:57:06 cqdx kernel:
> Jun 26 17:57:06 cqdx kernel: Pid: 3245, comm: glusterfsd Not tainted
> 2.6.32-279.19.1.01.qiyi.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0T0WRN
> Jun 26 17:57:06 cqdx kernel: RIP: 0010:[<ffffffff814ec55e>]
> [<ffffffff814ec55e>] _spin_lock+0x1e/0x30
> Jun 26 17:57:06 cqdx kernel: RSP: 0018:ffff880dceb5d528  EFLAGS: 00000297
> Jun 26 17:57:06 cqdx kernel: RAX: 00000000000087eb RBX: ffff880dceb5d528
> RCX: ffff8820137923e0
> Jun 26 17:57:06 cqdx kernel: RDX: 00000000000087ea RSI: 0000000000006e5c
> RDI: ffff882016d330a8
> Jun 26 17:57:06 cqdx kernel: RBP: ffffffff8100bb8e R08: ffff880dceb5d674
> R09: ffff880dceb5d670
> Jun 26 17:57:06 cqdx kernel: R10: 0000000000000000 R11: 0000000000000000
> R12: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: R13: 0000000000000cb5 R14: ffff8810788166e8
> R15: ffff882016ea9400
> Jun 26 17:57:06 cqdx kernel: FS:  00007f6cf3fff700(0000)
> GS:ffff8810788e0000(0000) knlGS:0000000000000000
> Jun 26 17:57:06 cqdx kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> Jun 26 17:57:06 cqdx kernel: CR2: 0000000000481046 CR3: 000000201311e000
> CR4: 00000000000406e0
> Jun 26 17:57:06 cqdx kernel: DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> Jun 26 17:57:06 cqdx kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
> DR7: 0000000000000400
> Jun 26 17:57:06 cqdx kernel: Process glusterfsd (pid: 3245, threadinfo
> ffff880dceb5c000, task ffff8810191ecaa0)
> Jun 26 17:57:06 cqdx kernel: Stack:
> Jun 26 17:57:06 cqdx kernel: ffff880dceb5d5d8 ffffffffa02c27a8
> ffff881078816680 ffff880dceb5d670
> Jun 26 17:57:06 cqdx kernel: <d> ffff880dceb5d674 ffff880dceb5d668
> 00006e5c0000000f 0000000000000001
> Jun 26 17:57:06 cqdx kernel: <d> 00000000000172a0 ffff882016d33080
> ffff880dceb5d588 0000000000000082
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
> xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
> try_to_wake_up+0x24c/0x3e0
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
> xlog_cil_push+0x261/0x3d0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
> xlog_cil_force_lsn+0x115/0x120 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
> xfs_bmap_add_extent+0xad/0x3c0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2308>] ?
> _xfs_log_force+0x68/0x240 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2718>] ? xfs_log_force+0x18/0x40
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d99d0>] ?
> xfs_buf_cond_lock+0x90/0xc0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dadb3>] ?
> _xfs_buf_find+0xe3/0x280 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02dafbb>] ? xfs_buf_get+0x6b/0x1a0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d16d8>] ?
> xfs_trans_get_buf+0xe8/0x180 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a4a67>] ?
> xfs_da_do_buf+0x4c7/0x770 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a5fda>] ?
> xfs_da_get_buf+0x2a/0x30 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
> xfs_dir2_data_init+0x4d/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02aa49d>] ?
> xfs_dir2_data_init+0x4d/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a8c23>] ?
> xfs_dir2_sf_to_block+0xe3/0x5e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b5ce6>] ? xfs_iunlock+0x86/0x100
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118ecaa>] ? dput+0x9a/0x150
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02b0c18>] ?
> xfs_dir2_sf_addname+0x458/0x5b0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d6687>] ?
> kmem_zone_alloc+0x77/0xf0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02a81bc>] ?
> xfs_dir_createname+0x14c/0x1b0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02d3390>] ? xfs_link+0x1f0/0x300
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81195560>] ?
> mntput_no_expire+0x30/0x110
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118e7a7>] ? __d_lookup+0xa7/0x150
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8107721a>] ? capable+0x2a/0x60
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02e1120>] ? xfs_vn_link+0x40/0x70
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8118379c>] ? vfs_link+0x11c/0x190
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81182aaa>] ? lookup_hash+0x3a/0x50
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81186b71>] ? sys_linkat+0x141/0x160
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b83b>] ? vfs_stat+0x1b/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8117b864>] ? sys_newstat+0x24/0x50
> Jun 26 17:57:06 cqdx kernel: [<ffffffff81186bae>] ? sys_link+0x1e/0x20
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8100b072>] ?
> system_call_fastpath+0x16/0x1b
> Jun 26 17:57:06 cqdx kernel: Code: 00 00 00 01 74 05 e8 72 8c d8 ff c9 c3
> 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39
> c2 74 0e f3 90 <0f> b7 17 eb f5 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55
> 48 89
> Jun 26 17:57:06 cqdx kernel: Call Trace:
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8106210b>] ?
> enqueue_task_fair+0xfb/0x100
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c27a8>] ?
> xlog_state_get_iclog_space+0x68/0x2e0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8105f8ac>] ?
> try_to_wake_up+0x24c/0x3e0
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c2bf1>] ? xlog_write+0x181/0x6d0
> [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffff8126f9f5>] ? rwsem_wake+0x75/0x170
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c3c41>] ?
> xlog_cil_push+0x261/0x3d0 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa02c43a5>] ?
> xlog_cil_force_lsn+0x115/0x120 [xfs]
> Jun 26 17:57:06 cqdx kernel: [<ffffffffa029a30d>] ?
> xfs_bmap_add_extent+0xad/0x3c0 [xfs]
> 
> 

<Prev in Thread] Current Thread [Next in Thread>