I've updated to 2.4.18 w/ a XFS CVS download from 03/29/2002. During SPEC
testing, the VM takes over all CPU load as pagebuf_iostart starts waiting
for memory, and then kmalloc starts waiting for memory. All of this time
spent in shrink_cache causes the SPEC test to time out. Once the test
stops, the box settles down and VM CPU load goes away. All of the
shrink_cache functions are waiting for schedule() to come back, because of
the test for current->need_resched at the top of the shrink_cache loop. For
grins, I commented out that test, and now many nfsd processes are sitting in
_pagebuf_find_lockable_buffer->pagebuf_iostart's call to pagebuf_iowait.
Could the revamped pagebuf locking cause this behaviour?
Erik
Here are some decoded Alt-Sysrq traces during the tests:
Trace 1: both kswapd and nfsd decided to go into shrink_cache code
task: kswapd (pid: 7)
c014b9bc: c014b885 t _text_lock_inode
c014aae7: c014aaac t dispose_list
c014ad39: c014ac7c T prune_icache
c014ad77: c014ad5c T shrink_icache_memory
c012eb09: c012ea9c t shrink_caches
c012eb5c: c012eb20 T try_to_free_pages
c012ebf3: c012ebb0 t kswapd_balance_pgdat
c012ec4e: c012ec3c t kswapd_balance
c012ed5d: c012ecc4 T kswapd
c01055a4: c010557c T kernel_thread
task: nfsd (pid: 619)
c0114c08: c0114b8e t _text_lock_sched
c012e6bd: c012e624 t shrink_cache
c012eaf2: c012ea9c t shrink_caches
c012eb5c: c012eb20 T try_to_free_pages
c012f460: c012f404 t balance_classzone
c012f682: c012f57c T __alloc_pages
c01dd40b: c01dd274 T _pagebuf_lookup_pages
c012f402: c012f3ec T _alloc_pages
c01dd430: c01dd274 T _pagebuf_lookup_pages
c01dd7f3: c01dd790 T pagebuf_get
c01d2184: c01d2148 T xfs_trans_read_buf
c01a9413: c01a8f3c t xfs_da_do_buf
c01a96a9: c01a967c T xfs_da_read_buf
c01ace45: c01ace10 t xfs_dir2_block_lookup_int
c01ace45: c01ace10 t xfs_dir2_block_lookup_int
c019cce3: c019cc04 T xfs_bmap_last_offset
c01acd6f: c01acd54 T xfs_dir2_block_lookup
c01ab500: c01ab43c t xfs_dir2_lookup
c01ab51a: c01ab43c t xfs_dir2_lookup
c028c443: c028c430 T qdisc_restart
c01d34d6: c01d3418 T xfs_dir_lookup_int
c01be2f7: c01be28c T xfs_ilock
c01d7ad7: c01d7a44 t xfs_lookup
c01d686b: c01d683c t xfs_access
c01e44d2: c01e4454 t linvfs_lookup
c0140b31: c0140a98 T lookup_hash
c0140bc7: c0140b70 T lookup_one_len
c016720d: c0166f40 T nfsd_lookup
c016c9c8: c016c8f4 t nfsd3_proc_lookup
c01644f3: c0164420 t nfsd_dispatch
c02c66b5: c02c6428 T svc_process
c01642ef: c01640f8 t nfsd
c01055a4: c010557c T kernel_thread
Trace 2: kmalloc starts to fail as both pagebuf_get and kmalloc go looking
for pages
task: nfsd (pid: 687)
c012e6bd: c012e624 t shrink_cache
c012eaf2: c012ea9c t shrink_caches
c012eb5c: c012eb20 T try_to_free_pages
c012f460: c012f404 t balance_classzone
c012f682: c012f57c T __alloc_pages
c01dd40b: c01dd274 T _pagebuf_lookup_pages
c012f402: c012f3ec T _alloc_pages
c01dd430: c01dd274 T _pagebuf_lookup_pages
c01dd7f3: c01dd790 T pagebuf_get
c01d2184: c01d2148 T xfs_trans_read_buf
c01be9a8: c01be8a8 T xfs_itobp
c01bf9b7: c01bf96c T xfs_iread
c01bda32: c01bd82c T xfs_iget_core
c01bddee: c01bdd64 T xfs_iget
c01d353f: c01d3418 T xfs_dir_lookup_int
c01be2f7: c01be28c T xfs_ilock
c01d7ad7: c01d7a44 t xfs_lookup
c01e44d2: c01e4454 t linvfs_lookup
c0140b31: c0140a98 T lookup_hash
c0140bc7: c0140b70 T lookup_one_len
c016720d: c0166f40 T nfsd_lookup
c016c9c8: c016c8f4 t nfsd3_proc_lookup
c01644f3: c0164420 t nfsd_dispatch
c02c66b5: c02c6428 T svc_process
c01642ef: c01640f8 t nfsd
c01055a4: c010557c T kernel_thread
task: nfsd (pid: 732)
c012e6bd: c012e624 t shrink_cache
c012eaf2: c012ea9c t shrink_caches
c012eb5c: c012eb20 T try_to_free_pages
c012f460: c012f404 t balance_classzone
c0114c0f: c0114b8e t _text_lock_sched
c012f682: c012f57c T __alloc_pages
c012f402: c012f3ec T _alloc_pages
c012f6ea: c012f6e0 T __get_free_pages
c012cdfe: c012cd50 t kmem_cache_grow
c012d328: c012d1f8 T kmalloc
c01e193f: c01e18e8 t linvfs_readdir
c01a9823: c01a9808 t xfs_da_buf_make
c01a94eb: c01a8f3c t xfs_da_do_buf
c01ace45: c01ace10 t xfs_dir2_block_lookup_int
c029331a: c0293000 T ip_rcv
c028638e: c0286224 t net_rx_action
c0144bd4: c0144b40 T vfs_readdir
c0165a40: c0165a40 t filldir_one
c0165b35: c0165a8c t nfsd_get_name
c0165a40: c0165a40 t filldir_one
c0165f20: c0165efc t splice
c01d5bd6: c01d5990 T xfs_getattr
c01e9d8b: c01e9d54 T vn_revalidate
c01e4aea: c01e4ad4 T linvfs_revalidate_core
c01e4513: c01e4454 t linvfs_lookup
c0148bb9: c0148ba0 T dput
c0165ef1: c0165dfc T nfsd_findparent
c01662d0: c0166078 t find_fh_dentry
c01665a8: c01663ac T fh_verify
c0113547: c01132e8 t reschedule_idle
c0166fb2: c0166f40 T nfsd_lookup
c016c9c8: c016c8f4 t nfsd3_proc_lookup
c01644f3: c0164420 t nfsd_dispatch
c02c66b5: c02c6428 T svc_process
c01642ef: c01640f8 t nfsd
c01055a4: c010557c T kernel_thread
Trace 3: now only kmalloc is failing. Lots of nfsd's are stuck in the exact
same place
task: nfsd (pid: 646)
c012e6bd: c012e624 t shrink_cache
c012eaf2: c012ea9c t shrink_caches
c012eb5c: c012eb20 T try_to_free_pages
c012f460: c012f404 t balance_classzone
c012f682: c012f57c T __alloc_pages
c012f402: c012f3ec T _alloc_pages
c012f6ea: c012f6e0 T __get_free_pages
c012cdfe: c012cd50 t kmem_cache_grow
c012d328: c012d1f8 T kmalloc
c01e193f: c01e18e8 t linvfs_readdir
c01a9823: c01a9808 t xfs_da_buf_make
c01e0c7b: c01e0bb0 T _pagebuf_find_lockable_buffer
c0127608: c01275e8 T __find_lock_page
c01dd40b: c01dd274 T _pagebuf_lookup_pages
c01dd522: c01dd274 T _pagebuf_lookup_pages
c01dd7f3: c01dd790 T pagebuf_get
c01d2184: c01d2148 T xfs_trans_read_buf
c0144bd4: c0144b40 T vfs_readdir
c0165a40: c0165a40 t filldir_one
c0165b35: c0165a8c t nfsd_get_name
c0165a40: c0165a40 t filldir_one
c0165f20: c0165efc t splice
c01d5bd6: c01d5990 T xfs_getattr
c01e9d8b: c01e9d54 T vn_revalidate
c01492f3: c01492d8 T d_alloc
c0148bb9: c0148ba0 T dput
c0165ef1: c0165dfc T nfsd_findparent
c01662d0: c0166078 t find_fh_dentry
c01665a8: c01663ac T fh_verify
c0113547: c01132e8 t reschedule_idle
c0166fb2: c0166f40 T nfsd_lookup
c016c9c8: c016c8f4 t nfsd3_proc_lookup
c01644f3: c0164420 t nfsd_dispatch
c02c66b5: c02c6428 T svc_process
c01642ef: c01640f8 t nfsd
c01055a4: c010557c T kernel_thread
Trace 4: test with no need_resched check in shrink_cache
task: nfsd (pid: 566)
c0105ae4: c0105a78 T __down
c0105c80: c0105c78 T __down_failed
c01def4e: c01deea5 t _text_lock_page_buf
c01ddd0f: c01ddc98 T pagebuf_iostart
c01dd7d8: c01dd730 T pagebuf_get
c01d2124: c01d20e8 T xfs_trans_read_buf
c01a93b3: c01a8edc t xfs_da_do_buf
c01e0c1b: c01e0b50 T _pagebuf_find_lockable_buffer
c0127608: c01275e8 T __find_lock_page
c01dd3ab: c01dd214 T _pagebuf_lookup_pages
c01a9649: c01a961c T xfs_da_read_buf
c01acde5: c01acdb0 t xfs_dir2_block_lookup_int
c01acde5: c01acdb0 t xfs_dir2_block_lookup_int
c019cc83: c019cba4 T xfs_bmap_last_offset
c01acd0f: c01accf4 T xfs_dir2_block_lookup
c01ab4a0: c01ab3dc t xfs_dir2_lookup
c01ab4ba: c01ab3dc t xfs_dir2_lookup
c01b3a06: c01b39b4 T xfs_dir2_sf_create
c01d3476: c01d33b8 T xfs_dir_lookup_int
c01be297: c01be22c T xfs_ilock
c01d7a77: c01d79e4 t xfs_lookup
c01e4472: c01e43f4 t linvfs_lookup
c0165dd0: c0165d9c T nfsd_findparent
c0166246: c0166018 t find_fh_dentry
c0166548: c016634c T fh_verify
c011334a: c01132e8 t reschedule_idle
c0166f52: c0166ee0 T nfsd_lookup
c016c968: c016c894 t nfsd3_proc_lookup
c0164493: c01643c0 t nfsd_dispatch
c02c6655: c02c63c8 T svc_process
c016428f: c0164098 t nfsd
c01055a4: c010557c T kernel_thread
|