BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_ail_min_lsn+0x24/0x60 [xfs] [] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [] xlog_state_release_iclog+0xb3/0xd0 [xfs] [] _xfs_log_force+0x126/0x2a0 [xfs] [] ? down_trylock+0x36/0x50 [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] xfs_buf_item_trylock+0x58/0xc0 [xfs] [] xfsaild_push+0x24a/0x680 [xfs] [] xfsaild+0x62/0xc0 [xfs] [] ? xfsaild_push+0x680/0x680 [xfs] [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_ail_min_lsn+0x24/0x60 [xfs] [] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [] xlog_state_release_iclog+0xb3/0xd0 [xfs] [] _xfs_log_force+0x126/0x2a0 [xfs] [] ? down_trylock+0x36/0x50 [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] xfs_buf_item_trylock+0x58/0xc0 [xfs] [] xfsaild_push+0x24a/0x680 [xfs] [] xfsaild+0x62/0xc0 [xfs] [] ? xfsaild_push+0x680/0x680 [xfs] [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_buf_iodone+0x2b/0x60 [xfs] [] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [] xfs_buf_iodone_work+0x23/0x50 [xfs] [] process_one_work+0x127/0x470 [] worker_thread+0x164/0x370 [] ? manage_workers.isra.31+0x130/0x130 [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_buf_iodone+0x2b/0x60 [xfs] [] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [] xfs_buf_iodone_work+0x23/0x50 [xfs] [] process_one_work+0x127/0x470 [] worker_thread+0x164/0x370 [] ? manage_workers.isra.31+0x130/0x130 [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#7 stuck for 23s! [swift-object-se:4340] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 7 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4340, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803362a1498 EFLAGS: 00000297 RAX: 0000000000007c1c RBX: 0000000700000006 RCX: ffff8804186a7ce8 RDX: 0000000000007c1d RSI: ffff8804112c9700 RDI: ffff88082420fea8 RBP: ffff8803362a1498 R08: ffff8803362a15d0 R09: ffff8803362a15dc R10: 000000000000121c R11: 0000000000000006 R12: ffffffffa01a6587 R13: ffff8803362a1428 R14: ffff8803362a1544 R15: 0000000000000000 FS: 00007f3db1ffb700(0000) GS:ffff88083fc60000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000002955f6000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4340, threadinfo ffff8803362a0000, task ffff8804112c9700) Stack: ffff8803362a14a8 ffffffff816606be ffff8803362a1548 ffffffffa01cc548 ffff8803362a15dc ffff8803362a15d0 ffff8803362a15c0 000013248103ec59 00000000000144c0 ffff8804112c9700 0000000000000286 ffff880556253d78 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? up+0x32/0x50 [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? xfs_iext_insert+0x7e/0x100 [xfs] [] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [] _xfs_log_force+0x68/0x2a0 [xfs] [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] ? _xfs_buf_find+0x4b/0x240 [xfs] [] _xfs_buf_find+0xbc/0x240 [xfs] [] xfs_buf_get+0x34/0x1c0 [xfs] [] xfs_trans_get_buf+0x127/0x1a0 [xfs] [] xfs_da_do_buf+0x3d9/0x630 [xfs] [] xfs_da_get_buf+0x21/0x30 [xfs] [] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] ? kmem_zone_zalloc+0x39/0x50 [xfs] [] xfs_attr_set_int+0x406/0x4e0 [xfs] [] xfs_attr_set+0x7f/0x90 [xfs] [] xfs_xattr_set+0x47/0x60 [xfs] [] generic_setxattr+0x6b/0x90 [] __vfs_setxattr_noperm+0x7b/0x1c0 [] ? evm_inode_setxattr+0xe/0x10 [] vfs_setxattr+0xbc/0xc0 [] setxattr+0xc6/0x120 [] ? fsnotify+0x1c2/0x2a0 [] ? do_futex+0x7c/0x1b0 [] sys_fsetxattr+0xbb/0xe0 [] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? up+0x32/0x50 [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? xfs_iext_insert+0x7e/0x100 [xfs] [] ? xfs_bmap_add_extent_hole_real+0x112/0xad0 [xfs] [] _xfs_log_force+0x68/0x2a0 [xfs] [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] ? _xfs_buf_find+0x4b/0x240 [xfs] [] _xfs_buf_find+0xbc/0x240 [xfs] [] xfs_buf_get+0x34/0x1c0 [xfs] [] xfs_trans_get_buf+0x127/0x1a0 [xfs] [] xfs_da_do_buf+0x3d9/0x630 [xfs] [] xfs_da_get_buf+0x21/0x30 [xfs] [] xfs_attr_leaf_create.isra.2+0x47/0x100 [xfs] [] xfs_attr_shortform_to_leaf+0x104/0x240 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] ? kmem_zone_zalloc+0x39/0x50 [xfs] [] xfs_attr_set_int+0x406/0x4e0 [xfs] [] xfs_attr_set+0x7f/0x90 [xfs] [] xfs_xattr_set+0x47/0x60 [xfs] [] generic_setxattr+0x6b/0x90 [] __vfs_setxattr_noperm+0x7b/0x1c0 [] ? evm_inode_setxattr+0xe/0x10 [] vfs_setxattr+0xbc/0xc0 [] setxattr+0xc6/0x120 [] ? fsnotify+0x1c2/0x2a0 [] ? do_futex+0x7c/0x1b0 [] sys_fsetxattr+0xbb/0xe0 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#9 stuck for 23s! [swift-object-se:4362] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 9 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4362, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8803df863b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff880418632708 RDX: 0000000000007c1f RSI: ffff880110e99700 RDI: ffff88082420fea8 RBP: ffff8803df863b68 R08: ffff8803df863ca0 R09: ffff8803df863cac R10: 00000000000021c4 R11: 000000000000000a R12: ffff8803df863b50 R13: ffff88082420fe00 R14: ffff880808307a40 R15: ffff8803df863ae8 FS: 00007f3d6f7fe700(0000) GS:ffff88083fc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4362, threadinfo ffff8803df862000, task ffff880110e99700) Stack: ffff8803df863b78 ffffffff816606be ffff8803df863c18 ffffffffa01cc548 ffff8803df863cac ffff8803df863ca0 ffff8803df863c90 000022e4244506c0 00000000000144c0 ffff880110e99700 ffff8803df863df5 000000000000000e Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#28 stuck for 22s! [swift-object-se:4354] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 28 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 4354, comm: swift-object-se Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff8802a0d95b68 EFLAGS: 00000293 RAX: 0000000000007c1c RBX: ffffffff811641cf RCX: ffff88080830a960 RDX: 0000000000007c1e RSI: ffff8801c6235c00 RDI: ffff88082420fea8 RBP: ffff8802a0d95b68 R08: ffff8802a0d95ca0 R09: ffff8802a0d95cac R10: 0000000000002134 R11: 000000000000000a R12: ffff8802a0d95b50 R13: ffff88082420fe00 R14: ffff880424d676f8 R15: ffff8802a0d95ae8 FS: 00007f3d8dffb700(0000) GS:ffff88042fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3dbcd1b000 CR3: 00000004140bf000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swift-object-se (pid: 4354, threadinfo ffff8802a0d94000, task ffff8801c6235c00) Stack: ffff8802a0d95b78 ffffffff816606be ffff8802a0d95c18 ffffffffa01cc548 ffff8802a0d95cac ffff8802a0d95ca0 ffff8802a0d95c90 000022546322c000 00000000000144c0 ffff8801c6235c00 ffff8802a0d95df5 000000000000000e Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xlog_state_get_iclog_space+0x58/0x2e0 [xfs] [] ? kmem_cache_alloc+0x11d/0x140 [] xlog_write+0xe9/0x400 [xfs] [] ? kmem_zone_alloc+0x67/0xe0 [xfs] [] xlog_cil_push+0x20c/0x380 [xfs] [] xlog_cil_force_lsn+0x10e/0x120 [xfs] [] ? setxattr+0xd1/0x120 [] _xfs_log_force_lsn+0x57/0x2e0 [xfs] [] ? down_read+0x16/0x2b [] ? xfs_iunlock+0xe6/0x150 [xfs] [] ? xfs_file_fsync+0x113/0x300 [xfs] [] xfs_file_fsync+0x1fc/0x300 [xfs] [] do_fsync+0x56/0x80 [] sys_fsync+0x10/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#30 stuck for 22s! [xfsaild/sdf1:2876] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 30 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 2876, comm: xfsaild/sdf1 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x25/0x30 RSP: 0018:ffff88041116bc20 EFLAGS: 00000297 RAX: 00000000000007e5 RBX: 0000000000000000 RCX: 0000000000000231 RDX: 00000000000007e6 RSI: ffff88082420fea8 RDI: ffff880824b32e78 RBP: ffff88041116bc20 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88041116ba24 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8807eeedad80 R14: 0003000100000000 R15: ffff88041116bb88 FS: 0000000000000000(0000) GS:ffff88042fde0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f3dbcd1b000 CR3: 0000000001c05000 CR4: 00000000000406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xfsaild/sdf1 (pid: 2876, threadinfo ffff88041116a000, task ffff880819d35c00) Stack: ffff88041116bc30 ffffffff816606be ffff88041116bc50 ffffffffa01d13d4 ffff88082420fe00 ffff88082420fe00 ffff88041116bc70 ffffffffa01cc3fe ffff88041116bcc0 ffff88042121c800 ffff88041116bca0 ffffffffa01cc4d3 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_ail_min_lsn+0x24/0x60 [xfs] [] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [] xlog_state_release_iclog+0xb3/0xd0 [xfs] [] _xfs_log_force+0x126/0x2a0 [xfs] [] ? down_trylock+0x36/0x50 [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] xfs_buf_item_trylock+0x58/0xc0 [xfs] [] xfsaild_push+0x24a/0x680 [xfs] [] xfsaild+0x62/0xc0 [xfs] [] ? xfsaild_push+0x680/0x680 [xfs] [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 0f b7 07 <66> 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 e5 89 d1 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_ail_min_lsn+0x24/0x60 [xfs] [] xlog_assign_tail_lsn+0x1e/0x40 [xfs] [] xlog_state_release_iclog+0xb3/0xd0 [xfs] [] _xfs_log_force+0x126/0x2a0 [xfs] [] ? down_trylock+0x36/0x50 [] xfs_log_force+0x18/0x40 [xfs] [] xfs_buf_trylock+0xd0/0xe0 [xfs] [] xfs_buf_item_trylock+0x58/0xc0 [xfs] [] xfsaild_push+0x24a/0x680 [xfs] [] xfsaild+0x62/0xc0 [xfs] [] ? xfsaild_push+0x680/0x680 [xfs] [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:3:27580] Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] CPU 0 Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu xfs vesafb 8021q garp bonding joydev bridge stp shpchp dcdbas sb_edac mac_hid edac_core wmi acpi_power_meter lp mei(C) parport usbhid hid ixgbe mdio igb megaraid_sas dca [last unloaded: ipmi_si] Pid: 27580, comm: kworker/0:3 Tainted: G WC 3.2.0-51-generic #77-Ubuntu Dell Inc. PowerEdge R720xd/0X6H47 RIP: 0010:[] [] __ticket_spin_lock+0x22/0x30 RSP: 0018:ffff8801aeae3d40 EFLAGS: 00000293 RAX: 00000000000007e5 RBX: ffff8801aeae3cd0 RCX: 0000000000000003 RDX: 00000000000007e7 RSI: ffff880823a117d0 RDI: ffff880824b32e78 RBP: ffff8801aeae3d40 R08: ffff8801aeae2000 R09: 0000000000000093 R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002 R13: ffff88042fc137c0 R14: ffff880425a37a00 R15: 000000000000014b FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007effa8b52ef7 CR3: 0000000001c05000 CR4: 00000000000406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/0:3 (pid: 27580, threadinfo ffff8801aeae2000, task ffff8800a5015c00) Stack: ffff8801aeae3d50 ffffffff816606be ffff8801aeae3d80 ffffffffa01cfe1b ffff8806e3d3dc48 ffff8800339f1600 ffff880421b76940 ffff8801bbe53500 ffff8801aeae3da0 ffffffffa01cef1c 0000000000000000 ffff880421b76940 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_buf_iodone+0x2b/0x60 [xfs] [] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [] xfs_buf_iodone_work+0x23/0x50 [xfs] [] process_one_work+0x127/0x470 [] worker_thread+0x164/0x370 [] ? manage_workers.isra.31+0x130/0x130 [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13 Code: 90 90 90 90 90 90 90 90 90 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 13 66 0f 1f 84 00 00 00 00 00 f3 90 <0f> b7 07 66 39 d0 75 f6 5d c3 0f 1f 40 00 8b 17 55 31 c0 48 89 Call Trace: [] _raw_spin_lock+0xe/0x20 [] xfs_buf_iodone+0x2b/0x60 [xfs] [] xfs_buf_do_callbacks+0x3c/0x50 [xfs] [] xfs_buf_iodone_callbacks+0x46/0x220 [xfs] [] xfs_buf_iodone_work+0x23/0x50 [xfs] [] process_one_work+0x127/0x470 [] worker_thread+0x164/0x370 [] ? manage_workers.isra.31+0x130/0x130 [] kthread+0x8c/0xa0 [] kernel_thread_helper+0x4/0x10 [] ? flush_kthread_worker+0xa0/0xa0 [] ? gs_change+0x13/0x13