xfs
[Top] [All Lists]

BUG 813113 - kswapd issues

To: nb@xxxxxxx
Subject: BUG 813113 - kswapd issues
From: pv@xxxxxxxxxxxxxxxxx (ananth@xxxxxxxxxxxx)
Date: Mon, 22 Jan 2001 14:41:29 -0800 (PST)
Cc: linux-xfs@xxxxxxxxxxx
Reply-to: sgi.bugs.xfs@xxxxxxxxxxxxxxxxx
Sender: owner-linux-xfs@xxxxxxxxxxx
Webexec: webpvsubmit,PvProjectIncident
Webpv: sgigate.sgi.com
View Incident: 
http://co-op.engr.sgi.com/BugWorks/code/bwxquery.cgi?search=Search&wlong=1&view_type=Bug&wi=813113

Submitter : ananth                    Submitter Domain : engr               
Assigned Engineer : nb                Assigned Domain : sgi.com             
Assigned Group : xfs-linux            Category : software                   
Customer Reported : T                 Priority : 2                          
Project : xfs-linux                   Status : open                         
Description :
Below is the state of a system that has kswapd (pid = 3)
and a process performing I/O (pid = 1120) that are deadlocked
waiting for each other. I'm sure that the wakeup_kswapd()
changes in 2.4.1preX change this situation somewhat. However,
in the long run, allocating memory under IOLOCK could
still leave avenues for similar problems.

Steve, we'll use this bug to record problems in this area.
This instance is the fdatasync case, while running a 4-threaded
doio on a 64MB machine (takes about an hour to hit it).
If you find other backtraces, lets gather them here.
Please feel free to assign the bug to yourself ;-)
I'll provide an update w/ the 2.4.1pre patch applied.

ananth.

---------------------------------------------------------------
Entering kdb (current=0xc02bc000, pid 0) on processor 0 due to Keyboard Entry
[0]kdb> ps
Task Addr    Pid     Parent  [*] cpu  State    Thread   Command
0xc1160000 00000001 00000000  0  000  stop  0xc1160350 init
0xc1154000 00000002 00000001  0  000  stop  0xc1154350 keventd
0xc1146000 00000003 00000001  0  001  stop  0xc1146350 kswapd
0xc1144000 00000004 00000001  0  001  stop  0xc1144350 kreclaimd
0xc1140000 00000005 00000001  0  001  stop  0xc1140350 bdflush
0xc11fe000 00000006 00000001  0  001  stop  0xc11fe350 kupdate
0xc3e4c000 00000301 00000001  0  001  stop  0xc3e4c350 syslogd
0xc3544000 00000311 00000001  0  001  stop  0xc3544350 klogd
0xc34aa000 00000326 00000001  0  001  stop  0xc34aa350 portmap
0xc3456000 00000351 00000001  0  001  stop  0xc3456350 rpc.statd
0xc3c18000 00000369 00000001  0  000  stop  0xc3c18350 ypbind
0xc34e8000 00000371 00000369  0  001  stop  0xc34e8350 ypbind
0xc3374000 00000372 00000371  0  000  stop  0xc3374350 ypbind
0xc36ce000 00000373 00000371  0  001  stop  0xc36ce350 ypbind
0xc3410000 00000432 00000001  0  001  stop  0xc3410350 identd
0xc3c36000 00000434 00000432  0  000  stop  0xc3c36350 identd
0xc3d56000 00000435 00000434  0  000  stop  0xc3d56350 identd
0xc331e000 00000436 00000434  0  001  stop  0xc331e350 identd
0xc3336000 00000437 00000434  0  000  stop  0xc3336350 identd
0xc330c000 00000451 00000001  0  001  stop  0xc330c350 atd
0xc32ce000 00000482 00000001  0  000  stop  0xc32ce350 inetd
[0]more> 
0xc31ac000 00000518 00000001  0  000  stop  0xc31ac350 lpd
0xc3040000 00000538 00000001  0  001  stop  0xc3040350 amd
0xc3062000 00000541 00000001  0  001  stop  0xc3062350 rpciod
0xc3002000 00000542 00000001  0  000  stop  0xc3002350 lockd
0xc2dbc000 00000570 00000001  0  001  stop  0xc2dbc350 sendmail
0xc2f2a000 00000586 00000001  0  000  stop  0xc2f2a350 gpm
0xc2df6000 00000601 00000001  0  001  stop  0xc2df6350 crond
0xc2ef0000 00000632 00000001  0  001  stop  0xc2ef0350 rhnsd
0xc3eaa000 00000648 00000001  0  001  stop  0xc3eaa350 mingetty
0xc2eda000 00000649 00000001  0  001  stop  0xc2eda350 mingetty
0xc2dda000 00000650 00000001  0  000  stop  0xc2dda350 mingetty
0xc2ecc000 00000651 00000001  0  000  stop  0xc2ecc350 mingetty
0xc2c78000 00000652 00000001  0  000  stop  0xc2c78350 mingetty
0xc2dce000 00000653 00000001  0  001  stop  0xc2dce350 mingetty
0xc2e9a000 00000654 00000001  0  001  stop  0xc2e9a350 login
0xc2ef6000 00000655 00000001  0  001  stop  0xc2ef6350 getty
0xc2c40000 00000658 00000654  0  000  stop  0xc2c40350 bash
0xc0d2e000 00000682 00000001  0  000  stop  0xc0d2e350 pagebuf_daemon
0xc04fc000 00000683 00000001  0  001  stop  0xc04fc350 page_daemon
0xc3ef0000 00001115 00000658  0  000  stop  0xc3ef0350 bash
0xc232a000 00001116 00001115  0  000  stop  0xc232a350 iogen-l
0xc3930000 00001117 00001115  0  001  stop  0xc3930350 doio-l
0xc2898000 00001118 00001117  0  000  stop  0xc2898350 doio-l
[0]more> 
0xc268c000 00001119 00001117  0  001  stop  0xc268c350 doio-l
0xc1552000 00001120 00001117  0  000  stop  0xc1552350 doio-l
0xc2c3c000 00001121 00001117  0  000  stop  0xc2c3c350 doio-l
0xc04f6000 00001145 00000632  0  001  stop  0xc04f6350 rhn_check
[0]kdb> btp 683
    EBP       EIP         Function(args)
0xc04fdf78 0xc011284a schedule+0x41e (0xc04fdf8c)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
0xc04fdfa0 0xc011237a schedule_timeout+0x7a (0xc101f538, 0x0)
                               kernel .text 0xc0100000 0xc0112300 0xc011239c
0xc04fdfc4 0xc0112e4e interruptible_sleep_on_timeout+0x52 (0xf00, 0xc0d47cc0, 
0xc0d47d78)
                               kernel .text 0xc0100000 0xc0112dfc 0xc0112e78
           0xc48650f0 [pagebuf]page_cleaner_daemon+0x280
                               pagebuf .text 0xc485f060 0xc4864e70 0xc4865138
           0xc010750f kernel_thread+0x23
                               kernel .text 0xc0100000 0xc01074ec 0xc010751c
[0]kdb> md pb_delalloc_pages 1
0xc48678d8 00000000 00000200 00000401 00000001  ................
[0]kdb> btp 3 
    EBP       EIP         Function(args)
0xc1147eb0 0xc011284a schedule+0x41e
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc486cb0e [xfs_support]lock_wait+0xa6 (0xc046ec48, 0xc046ec54, 0x1)
                               xfs_support .text 0xc486c060 0xc486ca68 
0xc486cb3c
           0xc486cbed [xfs_support]mrupdatef_Rsmp_7860bc50+0x29 (0xc046ec30, 
0x288)
                               xfs_support .text 0xc486c060 0xc486cbc4 
0xc486cc08
           0xc48ab260 [xfs]xfs_ilock_ra+0x34 (0xc046eb7c, 0x1)
                               xfs .text 0xc4872060 0xc48ab22c 0xc48ab2c0
           0xc48ab2d3 [xfs]xfs_ilock+0x13 (0xc48c9808, 0xc046eb7c, 0x1)
                               xfs .text 0xc4872060 0xc48ab2c0 0xc48ab2d8
           0xc48c9808 [xfs]xfs_rwlock+0x4c (0xc046eb94, 0x2)
                               xfs .text 0xc4872060 0xc48c97bc 0xc48c9814
           0xc48cf1ad [xfs]linvfs_write_full_page+0x85 (0xc1070e80)
                               xfs .text 0xc4872060 0xc48cf128 0xc48cf224
           0xc0124a76 filemap_fdatasync+0x86 (0xc12ddc24, 0x10f00)
                               kernel .text 0xc0100000 0xc01249f0 0xc0124af8
           0xc01492b2 sync_all_inodes+0xfa
                               kernel .text 0xc0100000 0xc01491b8 0xc0149334
0xc1147fa4 0xc0149785 prune_icache+0x31 (0x0)
                               kernel .text 0xc0100000 0xc0149754 0xc0149864
           0xc0149885 shrink_icache_memory+0x21 (0x6, 0x4, 0x6, 0x4)
[0]more> 
                               kernel .text 0xc0100000 0xc0149864 0xc0149894
           0xc012d5b3 do_try_to_free_pages+0x5b (0x4, 0x1, 0xc1161fb4)
                               kernel .text 0xc0100000 0xc012d558 0xc012d5d8
           0xc012d666 kswapd+0x8e
                               kernel .text 0xc0100000 0xc012d5d8 0xc012d710
           0xc010750f kernel_thread+0x23
                               kernel .text 0xc0100000 0xc01074ec 0xc010751c
[0]kdb> btp 4
    EBP       EIP         Function(args)
0xc1145fac 0xc011284a schedule+0x41e (0xc02aa460)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
0xc1145fcc 0xc0112dd9 interruptible_sleep_on+0x4d (0x10f00, 0xc1161fa8)
                               kernel .text 0xc0100000 0xc0112d8c 0xc0112dfc
           0xc012d86f kreclaimd+0x5b
                               kernel .text 0xc0100000 0xc012d814 0xc012d8f0
           0xc010750f kernel_thread+0x23
                               kernel .text 0xc0100000 0xc01074ec 0xc010751c
[0]kdb> btp 5
    EBP       EIP         Function(args)
0xc1141fd8 0xc011284a schedule+0x41e (0x10f00)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc0138551 bdflush+0xd1
                               kernel .text 0xc0100000 0xc0138480 0xc013855c
           0xc010750f kernel_thread+0x23
                               kernel .text 0xc0100000 0xc01074ec 0xc010751c
[0]kdb> btp 6
    EBP       EIP         Function(args)
0xc11ffd74 0xc011284a schedule+0x41e
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc012d7cb wakeup_kswapd+0xbb (0x1)
                               kernel .text 0xc0100000 0xc012d710 0xc012d7e8
           0xc012e4fe __alloc_pages+0x256
                               kernel .text 0xc0100000 0xc012e2a8 0xc012e5b0
           0xc485fdf3 [pagebuf]_pagebuf_lookup_pages+0x327 (0xc21f7260, 0x8000, 
0x0, 0x2000, 0x202201)
                               pagebuf .text 0xc485f060 0xc485facc 0xc4860068
           0xc4860231 [pagebuf]pagebuf_get_Rsmp_de233e60+0x171 (0xc3e68660, 
0x8000, 0x0, 0x2000, 0x2201)
                               pagebuf .text 0xc485f060 0xc48600c0 0xc48602ec
           0xc48c0426 [xfs]xfs_trans_read_buf+0x5e (0xc3500000, 0x0, 
0xc3500164, 0x40, 0x0)
                               xfs .text 0xc4872060 0xc48c03c8 0xc48c0704
           0xc48ab872 [xfs]xfs_itobp+0x11e (0xc3500000, 0x0, 0xc046eb7c, 
0xc11fff78, 0xc11fff7c)
                               xfs .text 0xc4872060 0xc48ab754 0xc48ab92c
           0xc48c30ca [xfs]xfs_syncsub+0x542 (0xc3500000, 0x31, 0x0, 0x0)
                               xfs .text 0xc4872060 0xc48c2b88 0xc48c3734
           0xc48c2b81 [xfs]xfs_sync+0x15 (0xc3500000, 0x31, 0xc48dd5e0)
                               xfs .text 0xc4872060 0xc48c2b6c 0xc48c2b88
           0xc48d3878 [xfs]linvfs_write_super+0x28 (0xc2f20000)
                               xfs .text 0xc4872060 0xc48d3850 0xc48d3884
           0xc0139567 sync_supers+0x6f (0x0)
[0]more> 
                               kernel .text 0xc0100000 0xc01394f8 0xc013959c
           0xc0138356 sync_old_buffers+0x2a (0x10f00)
                               kernel .text 0xc0100000 0xc013832c 0xc01383bc
           0xc0138665 kupdate+0x109
                               kernel .text 0xc0100000 0xc013855c 0xc0138670
           0xc010750f kernel_thread+0x23
                               kernel .text 0xc0100000 0xc01074ec 0xc010751c
[0]kdb> btp 1117
    EBP       EIP         Function(args)
0xc3931f7c 0xc011284a schedule+0x41e (0xc3930000)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc01185b6 sys_wait4+0x3c6 (0xffffffff, 0xbffffc5c, 0x0, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc01181f0 0xc01185ec
           0xc0108ff3 system_call+0x33
                               kernel .text 0xc0100000 0xc0108fc0 0xc0108ff8
[0]kdb> btp 1118
    EBP       EIP         Function(args)
0xc2899e84 0xc011284a schedule+0x41e (0xc3b24e0c)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc0145b06 interruptible_sleep_on_locked+0x6a (0xc3b24e28, 0x0, 
0xc3b247f0, 0xc3b24e0c)
                               kernel .text 0xc0100000 0xc0145a9c 0xc0145b40
           0xc0145b5c locks_block_on+0x1c (0xc3b247f0, 0xc3b24e0c)
                               kernel .text 0xc0100000 0xc0145b40 0xc0145b6c
           0xc014618e posix_lock_file+0xe2 (0xc2f26900, 0xc3b24e0c, 0x1)
                               kernel .text 0xc0100000 0xc01460ac 0xc0146640
           0xc014701d fcntl_setlk+0x12d (0x2, 0x7, 0xbffff5ec)
                               kernel .text 0xc0100000 0xc0146ef0 0xc01470d0
           0xc01430f1 do_fcntl+0x1c9 (0x2, 0x7, 0xbffff5ec, 0xc2f26900)
                               kernel .text 0xc0100000 0xc0142f28 0xc0143214
           0xc0143305 sys_fcntl64+0xb5 (0x2, 0x7, 0xbffff5ec, 0xbffffcdc, 0x2)
                               kernel .text 0xc0100000 0xc0143250 0xc0143358
           0xc0108ff3 system_call+0x33
                               kernel .text 0xc0100000 0xc0108fc0 0xc0108ff8
[0]kdb> btp 1119
    EBP       EIP         Function(args)
0xc268deb0 0xc011284a schedule+0x41e (0x21da)
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
0xc268dec8 0xc0107a54 __down+0x6c
                               kernel .text 0xc0100000 0xc01079e8 0xc0107ab0
           0xc0107bf8 __down_failed+0x8 (0x21da, 0xc268df6c, 0x0, 0x0, 0x0)
                               kernel .text 0xc0100000 0xc0107bf0 0xc0107bfc
           0xc48d4b87 [xfs].text.lock+0x4cd
                               xfs .text.lock 0xc48d46ba 0xc48d46ba 0xc48d4cd2
           0xc48cc809 [xfs]linvfs_write+0x45 (0xc3dc8a20, 0x4c49c00a, 0x21da, 
0xc3dc8a40, 0xc3dc8a20)
                               xfs .text 0xc4872060 0xc48cc7c4 0xc48cc8dc
           0xc0133ea6 do_readv_writev+0x1d6 (0x0, 0xc3dc8a20, 0xbffff450, 0x1)
                               kernel .text 0xc0100000 0xc0133cd0 0xc0133f24
           0xc0133fb9 sys_writev+0x41 (0x3, 0xbffff450, 0x1, 0x1, 0x3)
                               kernel .text 0xc0100000 0xc0133f78 0xc0133fcc
           0xc0108ff3 system_call+0x33
                               kernel .text 0xc0100000 0xc0108fc0 0xc0108ff8
[0]kdb> btp 1120
    EBP       EIP         Function(args)
0xc1553d34 0xc011284a schedule+0x41e
                               kernel .text 0xc0100000 0xc011242c 0xc0112a60
           0xc012d7cb wakeup_kswapd+0xbb (0x1)
                               kernel .text 0xc0100000 0xc012d710 0xc012d7e8
           0xc012e4fe __alloc_pages+0x256
                               kernel .text 0xc0100000 0xc012e2a8 0xc012e5b0
           0xc01278fe grab_cache_page+0x66 (0xc12ddc24, 0x358)
                               kernel .text 0xc0100000 0xc0127898 0xc012793c
           0xc4863e8b [pagebuf]__pagebuf_do_delwri+0x6f (0xc12ddb80, 0x358000, 
0x0, 0x8000, 0x4c4b0993)
                               pagebuf .text 0xc485f060 0xc4863e1c 0xc4864150
           0xc486429a [pagebuf]_pagebuf_file_write+0x14a (0xc1f19ce0, 
0x4c4b0993, 0x788c, 0xc1553ec0, 0x2)
                               pagebuf .text 0xc485f060 0xc4864150 0xc48642d8
           0xc48644db [pagebuf]pagebuf_generic_file_write_Rsmp_fba104b7+0x203 
(0xc1f19ce0, 0x4c4b0993, 0x788c, 0xc1553f88, 0xc046eb7c)
                               pagebuf .text 0xc485f060 0xc48642d8 0xc48647b4
           0xc48d0203 [xfs]xfs_write+0x197 (0xc046eb94, 0xc1553f7c, 0x0, 0x0, 
0x0)
                               xfs .text 0xc4872060 0xc48d006c 0xc48d02c0
           0xc48cc8b1 [xfs]linvfs_write+0xed (0xc1f19ce0, 0x4c49c008, 0x1c217, 
0xc1f19d00)
                               xfs .text 0xc4872060 0xc48cc7c4 0xc48cc8dc
           0xc0133c9a sys_write+0x8e (0x4, 0x4c49c008, 0x1c217, 0xbffff8e8, 0x0)
                               kernel .text 0xc0100000 0xc0133c0c 0xc0133cd0
           0xc0108ff3 system_call+0x33
[0]more> 
                               kernel .text 0xc0100000 0xc0108fc0 0xc0108ff8
------------------------------

<Prev in Thread] Current Thread [Next in Thread>
  • BUG 813113 - kswapd issues, ananth@xxxxxxxxxxxx <=