On Tue, Jan 03, 2006 at 11:20:11AM -0700, Wes L wrote:
> [1.] One line summary of the problem:
> XFS + Serpent Cipher on entire disk results in BUG: Soft Lockup on CPU1
> [2.] Full description of the problem/report:
>
> Bug will repeat every 2-3 seconds (as long as file copy is in progress),
> usually alternating between CPU0 and CPU1
> Attached are my current kernel compile options.
>
> cryptsetup -c serpent -h sha512 create disk1 /dev/hde
> mkfs.xfs /dev/mapper/disk1
> mount /dev/mapper/disk1 /mnt/disk1
> cp (very large directory) to /mnt/disk1
> 10-15 seconds later:
>
> Jan 3 09:36:23 alpha BUG: soft lockup detected on CPU#1!
> Jan 3 09:36:23 alpha
.....
> Jan 3 09:36:23 alpha [<c026adf1>] xfs_cluster_write+0x39/0x45
> Jan 3 09:36:23 alpha [<c026b2da>] xfs_page_state_convert+0x4dd/0x52e
> Jan 3 09:36:23 alpha [<c01366d9>] find_get_pages_tag+0x2a/0x63
> Jan 3 09:36:23 alpha [<c026b86c>] linvfs_writepage+0x91/0xc6
> Jan 3 09:36:23 alpha [<c016d584>] mpage_writepages+0x1a5/0x2fe
> Jan 3 09:36:23 alpha [<c026b7db>] linvfs_writepage+0x0/0xc6
> Jan 3 09:36:23 alpha [<c016be87>] __sync_single_inode+0x5e/0x1ba
> Jan 3 09:36:23 alpha [<c016c11a>] __writeback_single_inode+0x137/0x13f
> Jan 3 09:36:23 alpha [<c0260782>] xfs_trans_first_ail+0xe/0x16
> Jan 3 09:36:23 alpha [<c025463f>] xfs_log_need_covered+0x56/0x86
> Jan 3 09:36:23 alpha [<c03b0970>] dm_table_any_congested+0xd/0x47
> Jan 3 09:36:23 alpha [<c016c2cc>] sync_sb_inodes+0x1aa/0x271
> Jan 3 09:36:23 alpha [<c013bc41>] pdflush+0x0/0x2d
> Jan 3 09:36:23 alpha [<c016c40b>] writeback_inodes+0x78/0xc7
> Jan 3 09:36:23 alpha [<c013b4f7>] wb_kupdate+0x92/0xf7
> Jan 3 09:36:23 alpha [<c013bbaf>] __pdflush+0xe3/0x175
> Jan 3 09:36:23 alpha [<c013bc69>] pdflush+0x28/0x2d
> Jan 3 09:36:23 alpha [<c013b465>] wb_kupdate+0x0/0xf7
> Jan 3 09:36:23 alpha [<c012b257>] kthread+0x75/0x9d
> Jan 3 09:36:23 alpha [<c012b1e2>] kthread+0x0/0x9d
> Jan 3 09:36:23 alpha [<c0100e79>] kernel_thread_helper+0x5/0xb
So pdflush is doing background writeout, and XFS is doing write
clustering. XFS in current kernels do not pay attention to the
writeback control structure and the guidelines for how many pages to
write in a loop here, so can cluster up to 8GiB in a single call to
xfs_cluster_write(). There is no conditional reschedule in this
loop.
Normally this is not a problem as block device congestion will cause
the loop to sleep regularly. However, with the encryption layer,
this is probably not occurring due to encryption slowing the write
speed and we are burning large amounts of uninterrupted CPU time in
this loop because the underlying block device never becomes
congested and so we don't sleep.
[ Note that there are mods pending that make XFS obey the wbc
guidelines when clustering writes. ]
Also, you're not running a CONFIG_PREEMPT kernel, so nothing other
than interrupts will be able to use the CPU while in this loop.
In the mean time, try reducing the queue depth on the underlying
block device (/sys/block/<dev>/queue/nr_requests) to get congestion
to occur sooner. You could also try using a faster cipher.
HTH.
Cheers,
Dave.
--
Dave Chinner
R&D Software Enginner
SGI Australian Software Group
|