hi Andi,
On Thu, Jun 03, 2004 at 03:05:46AM +0200, Andi Kleen wrote:
>
> I found an easily reproducible way to deadlock XFS on 2.6.7-rc2/x86-64.
>
> Create a few GB XFS.
> Fill it with 100MB files so that only a few MBs are left
> Run fsstress -p30 -n50000 -d /xfsmount
> Run a while true ; do cp -a /bin /xfsmount ; done in parallel
> (the cp should run out of disk space all the time)
> (I did this all as root)
Yeah, test 083 in the xfstests is doing a similar kind of thing.
> Deadlocks in less than half a minute. The processes become unkillable
>
> Here's a sysrq-t listing from after the fact.
>
> The deadlock seems to be on some pagebuf page, a lot of processes
> stall forever trying to get the semaphore of a pagebuf.
I hit one deadlock in 2.6 out of space handling today, not sure
its the same as this one - I'll have a fix for that tomorrow.
That one I found is a deadlock on the regular file IO path, the
one here looks more like a metadata deadlock, though its hard to
tell really from these stack traces.
thanks.
> fsstress S 00000100362908f8 0 4388 4297 4389
> (NOTLB)
> 000001003686dea8 0000000000000006 0000007d00508418 00000100362907f0
> 0000000000008a6c 0000010035ee8000 0000010035e52100 ffffffff8011ff12
> 0000010035bd0ee0 00000100362908f8
> Call Trace:<ffffffff8011ff12>{do_page_fault+498}
> <ffffffff801347d2>{sys_wait4+1330}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff801101aa>{system_call+126}
> fsstress R running task 0 4389 4388 4390 (NOTLB)
> fsstress R running task 0 4390 4388 4391 4389 (NOTLB)
> fsstress R running task 0 4391 4388 4392 4390 (NOTLB)
> fsstress R running task 0 4392 4388 4393 4391 (NOTLB)
> fsstress R running task 0 4393 4388 4394 4392 (NOTLB)
> fsstress R running task 0 4394 4388 4395 4393 (NOTLB)
> fsstress R running task 0 4395 4388 4396 4394 (NOTLB)
> fsstress R running task 0 4396 4388 4397 4395 (NOTLB)
> fsstress D 00000100358d1980 0 4397 4388 4398 4396
> (NOTLB)
> 0000010035da6ca8 0000000000000002 0000007a35ccb688 0000010035ee87f0
> 00000000000008a5 ffffffff80488b00 0000010035da6ca8 ffffffff8012f013
> 0000000300000000 00000100358d1980
> Call Trace:<ffffffff8012f013>{__wake_up_common+67}
> <ffffffff803f97ff>{__down+143}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff803fa281>{__down_failed+53}
> <ffffffff8026edb9>{.text.lock.xfs_buf+15}
> <ffffffff8026e63f>{_pagebuf_find+367}
> <ffffffff8026e713>{pagebuf_get+147}
> <ffffffff8026153f>{xfs_trans_read_buf+719}
> <ffffffff80261436>{xfs_trans_read_buf+454}
> <ffffffff802194eb>{xfs_alloc_read_agf+123}
> <ffffffff80231d2d>{xfs_btree_setbuf+45}
> <ffffffff8021b2e3>{xfs_alloc_fix_freelist+323}
> <ffffffff8021a2a2>{xfs_alloc_ag_vextent+258}
> <ffffffff8021b8b3>{xfs_alloc_vextent+627}
> <ffffffff8022c7c3>{xfs_bmapi+5923}
> <ffffffff80254e1d>{xlog_assign_tail_lsn+13}
> <ffffffff80255ca0>{xlog_write+1024}
> <ffffffff80229c77>{xfs_bmap_search_extents+71}
> <ffffffff80252322>{xfs_iomap_write_allocate+546}
> <ffffffff802e4b04>{__elv_add_request+84}
> <ffffffff802519bb>{xfs_iomap+747}
> <ffffffff80131ea0>{autoremove_wake_function+0}
> <ffffffff8026b5b2>{xfs_map_blocks+114}
> <ffffffff8026c53f>{xfs_page_state_convert+591}
> <ffffffff80227953>{xfs_bmap_add_extent+1107}
> <ffffffff8027faa9>{rb_insert_color+217}
> <ffffffff802e844e>{__make_request+1262}
> <ffffffff8026cbb8>{linvfs_writepage+184}
> <ffffffff801838d4>{mpage_writepages+420}
> <ffffffff80229c77>{xfs_bmap_search_extents+71}
> <ffffffff8022b6f0>{xfs_bmapi+1616}
> <ffffffff80229c77>{xfs_bmap_search_extents+71}
> <ffffffff8026cb00>{linvfs_writepage+0}
> <ffffffff80229b20>{xfs_bmap_do_search_extents+544}
> <ffffffff8022b45a>{xfs_bmapi+954}
> <ffffffff802e6c23>{generic_make_request+339}
> <ffffffff8014ca24>{__filemap_fdatawrite+132}
> <ffffffff802527a7>{xfs_iomap_write_delay+903}
> <ffffffff80251962>{xfs_iomap+658}
> <ffffffff8026b6d8>{linvfs_get_block_core+136}
> <ffffffff80168314>{alloc_buffer_head+36}
> <ffffffff80168b56>{create_buffers+102}
> <ffffffff8026b85b>{linvfs_get_block+27}
> <ffffffff8016a16d>{__block_prepare_write+365}
> <ffffffff8026b840>{linvfs_get_block+0}
> <ffffffff8016a40a>{block_prepare_write+26}
> <ffffffff8014d39d>{generic_file_aio_write_nolock+1357}
> <ffffffff8027278a>{xfs_inval_cached_pages+266}
> <ffffffff80272b6f>{xfs_write+943}
> <ffffffff8026ef33>{linvfs_write+99}
> <ffffffff80167425>{do_sync_write+101}
> <ffffffff80268fea>{xfs_access+74} <ffffffff8024c4f0>{xfs_iaccess+224}
> <ffffffff802609fc>{xfs_trans_unlocked_item+44}
> <ffffffff8016f2aa>{cp_new_stat+234}
> <ffffffff8016f5be>{sys_newfstat+46} <ffffffff80167527>{vfs_write+199}
> <ffffffff80167659>{sys_write+73} <ffffffff801101aa>{system_call+126}
>
> fsstress R running task 0 4398 4388 4399 4397 (NOTLB)
> fsstress R running task 0 4399 4388 4400 4398 (NOTLB)
> fsstress R running task 0 4400 4388 4401 4399 (NOTLB)
> fsstress R running task 0 4401 4388 4402 4400 (NOTLB)
> fsstress R running task 0 4402 4388 4403 4401 (NOTLB)
> fsstress R running task 0 4403 4388 4404 4402 (NOTLB)
> fsstress R running task 0 4404 4388 4405 4403 (NOTLB)
> fsstress R running task 0 4405 4388 4406 4404 (NOTLB)
> fsstress R running task 0 4406 4388 4407 4405 (NOTLB)
> fsstress R running task 0 4407 4388 4408 4406 (NOTLB)
> fsstress R running task 0 4408 4388 4409 4407 (NOTLB)
> fsstress R running task 0 4409 4388 4410 4408 (NOTLB)
> fsstress D 0000010001b5e198 0 4410 4388 4411 4409
> (NOTLB)
> 0000010035ddb9f8 0000000000000002 0000000000000c6e 0000010035a8e000
> 000000000000146b ffffffff80488b00 0000000000000000 0000000000000002
> 0000010035ddba48 0000010001b5e198
> Call Trace:<ffffffff803f9e5f>{io_schedule+15}
> <ffffffff8014b83d>{__lock_page+237}
> <ffffffff8014b880>{page_wake_function+0}
> <ffffffff8014b880>{page_wake_function+0}
> <ffffffff8018386e>{mpage_writepages+318}
> <ffffffff802e6d38>{submit_bio+248}
> <ffffffff802e7a14>{generic_unplug_device+20}
> <ffffffff8026cb00>{linvfs_writepage+0}
> <ffffffff8026dfb9>{pagebuf_iorequest+345}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff8024b5c2>{xfs_ilock_nowait+162}
> <ffffffff8026505d>{xfs_inode_flush+317}
> <ffffffff8012ee7c>{recalc_task_prio+444}
> <ffffffff80182c96>{__writeback_single_inode+262}
> <ffffffff801831c7>{sync_sb_inodes+519}
> <ffffffff80183399>{sync_inodes_sb+121}
> <ffffffff801833cb>{sync_inodes+27}
> <ffffffff8016965c>{do_sync+44} <ffffffff8016969e>{sys_sync+14}
> <ffffffff801101aa>{system_call+126}
> fsstress R running task 0 4411 4388 4412 4410 (NOTLB)
> fsstress R running task 0 4412 4388 4413 4411 (NOTLB)
> fsstress R running task 0 4413 4388 4414 4412 (NOTLB)
> fsstress R running task 0 4414 4388 4415 4413 (NOTLB)
> fsstress R running task 0 4415 4388 4416 4414 (NOTLB)
> fsstress R running task 0 4416 4388 4417 4415 (NOTLB)
> fsstress R running task 0 4417 4388 4418 4416 (NOTLB)
> fsstress R running task 0 4418 4388 4417 (NOTLB)
> bash S 00000100366588f8 0 4419 1867 5391
> (NOTLB)
> 00000100353dbea8 0000000000000002 0000007600592760 00000100366587f0
> 00000000000147df 0000010035bef000 00000100363c0728 ffffffff8011ff12
> 0000010034291698 00000100366588f8
> Call Trace:<ffffffff8011ff12>{do_page_fault+498}
> <ffffffff801347d2>{sys_wait4+1330}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff801101aa>{system_call+126}
> cp D 0000010035ccb680 0 5391 4419
> (NOTLB)
> 00000100350e37d8 0000000000000002 00000076350e37c8 0000010036658000
> 00000000000008f0 0000010035ee87f0 00000100358dd480 0000010035bef000
> 000000b77a7b1584 0000010035ccb680
> Call Trace:<ffffffff803f97ff>{__down+143}
> <ffffffff8012fd70>{default_wake_function+0}
> <ffffffff803fa281>{__down_failed+53}
> <ffffffff8026edb9>{.text.lock.xfs_buf+15}
> <ffffffff8026e63f>{_pagebuf_find+367}
> <ffffffff8026e713>{pagebuf_get+147}
> <ffffffff802196c2>{xfs_alloc_log_agf+66}
> <ffffffff8021d5db>{xfs_alloc_insert+2987}
> <ffffffff80261436>{xfs_trans_read_buf+454}
> <ffffffff802194eb>{xfs_alloc_read_agf+123}
> <ffffffff8026105f>{xfs_trans_brelse+127}
> <ffffffff8021b2e3>{xfs_alloc_fix_freelist+323}
> <ffffffff80219d23>{xfs_free_ag_extent+1603}
> <ffffffff8021b5f8>{xfs_free_extent+152}
> <ffffffff802291fd>{xfs_bmap_finish+253}
> <ffffffff8024f3b3>{xfs_itruncate_finish+419}
> <ffffffff80260682>{_xfs_trans_alloc+98}
> <ffffffff802698b7>{xfs_setattr+2231}
> <ffffffff801c5a92>{ext3_get_block_handle+226}
> <ffffffff80271fc7>{linvfs_setattr+343}
> <ffffffff8017e20d>{notify_change+301}
> <ffffffff80166271>{do_truncate+81}
> <ffffffff8024b4d6>{xfs_iunlock+102}
> <ffffffff80271514>{linvfs_permission+20}
> <ffffffff80174274>{may_open+388} <ffffffff8017549f>{open_namei+799}
> <ffffffff801655f7>{filp_open+39} <ffffffff80175715>{getname+149}
> <ffffffff8016566c>{sys_open+76} <ffffffff801101aa>{system_call+126}
>
> sync01 R running task 0 5507 2086 (NOTLB)
> bash R running task 0 5530 1868 (NOTLB)
>
>
>
>
>
--
Nathan
|