While I was running regression tests last night on
2.4.3-SGI_XFS_1.0.1smp I hit a deadlock where 4-5 processes were all
hung, but the system was still responsive. I think this may be related
to the problems Christopher McCrory was having. Don't panic, I only
caught this after over seven hours of constant regression testing.
I was able to grab the system state using magic sysrq and Keith sent me
a fresh copy of ksymoops that is supposed to handle it. I have included
the output below, but I don't trust it. Keith, I'm using mangled kernel
versions which I don't think depmod or ksymoops handles correctly. i.e.
I specify depfile, generic_stringfile, path[toplevel], etc in
/etc/modules.conf.
Without further ado, here is the processed "Show State" data. I don't
think it makes much sense, but I'll let the experts judge it. The
processes "rm," "symlink02," "kupdated," "pmd," and "renamer" were all
in the "D" state.
ksymoops 2.4.2 on i686 2.4.3-SGI_XFS_1.0.1smp. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.3-SGI_XFS_1.0.1smp/ (default)
-m /usr/src/linux/System.map (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Error (expand_objects): cannot stat(/lib/qla1280.o) for qla1280
Error (expand_objects): cannot stat(/lib/ncr53c8xx.o) for ncr53c8xx
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (pclose_local): find_objects pclose failed 0x100
Error (regular_file): read_system_map stat /usr/src/linux/System.map failed
init S C476DF0C 0 1 0 19883 (NOTLB)
Using defaults from ksymoops -t elf32-i386 -a i386
Call Trace: [<c0114691>] [<c01145b4>] [<c0146b28>] [<c0147102>] [<c0106e23>]
keventd S 00000000 0 2 1 (L-TLB) 3
Call Trace: [<c0122fc8>] [<c0105643>]
kswapd S C4769FA0 0 3 1 (L-TLB) 4 2
Call Trace: [<c0114691>] [<c01145b4>] [<c0114f82>] [<c029476e>] [<c012fbc9>]
[<c0105643>]
kreclaimd S 00000286 0 4 1 (L-TLB) 5 3
Call Trace: [<c0114f0d>] [<c0294778>] [<c012fcbb>] [<c010563a>] [<c0105643>]
bdflush S C477C000 0 5 1 (L-TLB) 6 4
Call Trace: [<c029550e>] [<c012f813>] [<c013b17e>] [<c0105643>]
kupdated D C477A000 0 6 1 (L-TLB) 7 5
Call Trace: [<c01d877e>] [<c01d885d>] [<c01d3e13>] [<c01ad831>] [<c01ad873>]
[<c01d3e13>] [<c01d3e13>] [<c01d27dd>] [<c016a910>] [<c01d2760>]
[<c01c6037>]
[<c0169674>] [<c01d2760>] [<c01d293c>] [<c01d294f>] [<c01d2760>]
[<c01d293c>]
[<c0127550>] [<c01d293c>] [<c014b937>] [<c013af87>] [<c013b28d>]
[<c0105643>]
pagebuf_daemon S 00000286 0 7 1 (L-TLB) 8 6
Call Trace: [<c0114f0d>] [<c01685cc>] [<c01684d0>] [<c0105643>]
mdrecoveryd S F717C000 0 8 1 (L-TLB) 594 7
Call Trace: [<c02afdf1>] [<c0230682>] [<c0105643>]
syslogd S 7FFFFFFF 0 594 1 (NOTLB) 599 8
Call Trace: [<c0114627>] [<c0232437>] [<c0146b28>] [<c0147102>] [<c0106e23>]
klogd R F5E48000 4160 599 1 (NOTLB) 613 594
Call Trace: [<c0231ffd>] [<c0116ffc>] [<c01575e5>] [<c0136a1a>] [<c0106e23>]
[<c010002b>]
portmap S 7FFFFFFF 0 613 1 (NOTLB) 628 599
Call Trace: [<c024ebff>] [<c0232437>] [<c0114627>] [<c01473d3>] [<c014735f>]
[<c0147663>] [<c0106e23>]
rpc.statd S 7FFFFFFF 0 628 1 (NOTLB) 717 613
Call Trace: [<c0114627>] [<c0232437>] [<c0146b28>] [<c0147102>] [<c0136607>]
[<c0106e23>]
ypbind S 7FFFFFFF 0 717 1 719 (NOTLB) 776 628
Call Trace: [<c024ebff>] [<c0232437>] [<c0114627>] [<c01473d3>] [<c014735f>]
[<c0147663>] [<c0106e23>]
ypbind S F5F8BF10 4768 719 717 721 (NOTLB)
Call Trace: [<c0114691>] [<c01145b4>] [<c014735f>] [<c0147663>] [<c01086c9>]
[<c01086e9>] [<c0106e23>] [<c010002b>]
ypbind S 00000000 4160 720 719 (NOTLB) 721
Call Trace: [<c0105fdb>] [<c0106e23>] [<c010002b>]
ypbind S F66B9F78 1392 721 719 (NOTLB) 720
Call Trace: [<c0130870>] [<c0146916>] [<c0114691>] [<c01145b4>] [<c011f1ab>]
[<c0106e23>] [<c010002b>]
automount S F64C2000 16 776 1 (NOTLB) 786 717
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F5CF2000 4160 786 1 (NOTLB) 796 776
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F5D56000 0 796 1 (NOTLB) 806 786
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F6712000 4488 806 1 (NOTLB) 816 796
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F5E2C000 4160 816 1 (NOTLB) 826 806
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F65B4000 0 826 1 (NOTLB) 836 816
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F66AC000 4 836 1 (NOTLB) 846 826
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F5D42000 4160 846 1 (NOTLB) 856 836
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S C44F2000 0 856 1 (NOTLB) 866 846
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S C44FE000 0 866 1 (NOTLB) 876 856
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S C4504000 0 876 1 (NOTLB) 886 866
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S C4608000 0 886 1 (NOTLB) 893 876
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
automount S F6454000 0 893 1 (NOTLB) 905 886
Call Trace: [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
atd S F6689F78 0 905 1 (NOTLB) 1000 893
Call Trace: [<c0114691>] [<c01145b4>] [<c011f1ab>] [<c0106e23>]
sshd S 7FFFFFFF 0 1000 1 (NOTLB) 1080 905
Call Trace: [<c0114627>] [<c0232437>] [<c0146b28>] [<c0147102>] [<c0106e23>]
xinetd S 7FFFFFFF 0 1080 1 19899 (NOTLB) 1113 1000
Call Trace: [<c0114627>] [<c0232437>] [<c0146b28>] [<c0147102>] [<c0136607>]
[<c0106e23>]
sendmail S F5B1BF0C 2416 1113 1 (NOTLB) 1126 1080
Call Trace: [<c01469ae>] [<c0114691>] [<c01145b4>] [<c0146b28>] [<c0147102>]
[<c0106e23>]
crond S F5AFDF78 0 1126 1 19880 (NOTLB) 1165 1113
Call Trace: [<c0114691>] [<c01145b4>] [<c011f1ab>] [<c0106e23>]
mingetty S 7FFFFFFF 0 1165 1 (NOTLB) 1166 1126
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
mingetty S 7FFFFFFF 0 1166 1 (NOTLB) 1167 1165
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
mingetty S 7FFFFFFF 0 1167 1 (NOTLB) 1168 1166
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
mingetty S 7FFFFFFF 0 1168 1 (NOTLB) 1169 1167
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
mingetty S 7FFFFFFF 0 1169 1 (NOTLB) 5626 1168
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
in.rlogind S 7FFFFFFF 0 1328 1080 1355 (NOTLB) 19750
Call Trace: [<c01ef51c>] [<c0114627>] [<c0146b28>] [<c0147102>] [<c0106e23>]
[<c010002b>]
login S 00000000 2416 1355 1328 1356 (NOTLB)
Call Trace: [<c0114b12>] [<c011a596>] [<c0106e23>]
tcsh S 00000000 0 1356 1355 5567 (NOTLB)
Call Trace: [<c0114b12>] [<c0105fdb>] [<c0106e23>]
su S 00000000 0 5567 1356 5568 (NOTLB)
Call Trace: [<c011a596>] [<c0106e23>]
bash S 00000000 0 5568 5567 19749 (NOTLB)
Call Trace: [<c01ea876>] [<c01eac4d>] [<c011a596>] [<c0106e23>]
pmd S 7FFFFFFF 3608 5626 1 23861 (NOTLB) 5978 1169
Call Trace: [<c0114627>] [<c0232437>] [<c0146b28>] [<c0147102>] [<c0106e23>]
[<c010002b>]
getpgid2_01 S F4F8C000 0 5978 1 (NOTLB) 23496 5626
Call Trace: [<c010ca92>] [<c0106e23>]
getpgid2_01 S F300C000 2416 23496 1 (NOTLB) 19776 5978
Call Trace: [<c010ca92>] [<c0106e23>]
pmd S F43D8000 0 23861 5626 19698 (NOTLB)
Call Trace: [<c01145a7>] [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
renamer01 S 00000000 0 19588 23861 19622 (NOTLB) 19659
Call Trace: [<c0114b12>] [<c0105fdb>] [<c0106e23>]
renamer D F645C0E0 0 19622 19588 (NOTLB)
Call Trace: [<c01d2555>] [<c0141935>] [<c0105b6b>] [<c0105d18>] [<c028799c>]
[<c0106e23>] [<c010002b>]
rwtest S 00000000 0 19659 23861 19699 (NOTLB) 19698 19588
Call Trace:
wait_on_irq, CPU 0:
irq: 1 [ 0 1 0 0 ]
bh: 1 [ 1 0 0 0 ]
Stack dumps:
CPU 1:6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
Call Trace: [<c0166599>] [<c0166599>] [<c0166599>] [<c0166599>] [<c0166806>]
[<c0166806>] [<c01675d7>] [<c019386b>] [<c01145a7>] [<c01b5ff4>]
[<c01b609d>]
[<f880758a>] [<f88077c0>] [<f8807a9a>] [<f8817d86>] [<f880671d>]
[<f8806cc6>]
[<c011eb87>] [<c011eee0>] [<c011ef63>] [<c010c160>] [<c01084d6>]
[<c01086c9>]
[<c01086e9>] [<c0106ee4>] [<c0114b12>] [<c0114691>] [<c01145b4>]
[<c0114f82>]
[<c029476e>] [<c012fbc9>] [<c0105643>]
CPU 2:6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
Call Trace: [<c01001c8>] [<c01effa0>] [<c01f04e2>] [<c0166599>] [<c0166599>]
[<f9c04e6a>] [<c0166599>] [<c0166599>] [<f894b5b0>] [<f8945a36>]
[<c02405e4>]
[<c0238d06>] [<c0234312>] [<c0262e2e>] [<c0263099>] [<c02602cb>]
[<f894b5b0>]
[<f8945a36>] [<c02405e4>] [<c0238d06>] [<c024cca3>] [<c023ef54>]
[<c024b722>]
[<c024cc00>] [<c024b5cc>] [<c024bb0d>] [<c023ef54>] [<c024b5cc>]
[<c025f81d>]
[<c025a59f>] [<c025a662>] [<c025b18b>] [<c0258648>] [<c01fff66>]
[<c0200000>]
[<c011731b>] [<c011737f>] [<c011745b>] [<c0117696>] [<c011761e>]
[<c01070bf>]
[<c028cde0>] [<c0107170>] [<c0115783>] [<c0115852>] [<c0201b6d>]
[<c0201d1c>]
[<c02a3b80>] [<c02a3af6>] [<c0201cb3>] [<c02002fc>] [<c011eb87>]
[<c0201286>]
[<c0201314>] [<c01084d6>] [<c01086c9>] [<c0105280>] [<c0105280>]
[<c0106ee4>]
[<c0105280>] [<c0105280>] [<c0100018>] [<c01052ad>] [<c0105312>]
[<c011737f>]
CPU 3:6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db 6db6db6d db6db6db
Call Trace:
CPU 0:c02e3ecc c028d37d 00000000 00000000 00000000 c010832d c028d392 f7159ec0
00000004 00000090 00000001 c01fb677 c03691c0 00000000 c01fb640 00000286
c011e81c 00000000 00000001 00001020 c03691c0 c0388a80 00000000 c011eda9
Call Trace: [<c028d37d>] [<c010832d>] [<c028d392>] [<c01fb677>] [<c01fb640>]
[<c011e81c>] [<c011eda9>] [<c011b552>] [<c011b435>] [<c011b2da>]
[<c010870a>]
[<c0105280>] [<c0105280>] [<c0106ee4>] [<c0105280>] [<c0105280>]
[<c0100018>]
[<c01052ad>] [<c0105312>] [<c0105000>] [<c01001cf>]
[<c0114b12>] [<c0105fdb>] [<c0106e23>]
symlink02 D E4744000 0 19698 23861 (NOTLB) 19659
Call Trace: [<c01305b2>] [<c01c2792>] [<c01d877e>] [<c01d881c>] [<c01c74f2>]
[<c01ad856>] [<c01ad873>] [<c01c74f2>] [<c01c74f2>] [<c01d2555>]
[<c0141935>]
[<c0142459>] [<c01428d8>] [<c01360ba>] [<c0106e23>]
rm D E42FA000 0 19699 19659 (NOTLB)
Call Trace: [<c01c312c>] [<c014b6da>] [<c014c7bc>] [<c01d7c22>] [<c01ca027>]
[<c01ad99b>] [<c01d1e67>] [<c0143c2f>] [<c0143d32>] [<c0106e23>]
top S F4A23F0C 0 19749 5568 (NOTLB)
Call Trace: [<c01469ae>] [<c0114691>] [<c01145b4>] [<c0146b28>] [<c0147102>]
[<c0106e23>] [<c010002b>]
in.rlogind S 7FFFFFFF 0 19750 1080 19777 (NOTLB) 19899 1328
Call Trace: [<c01ef51c>] [<c0114627>] [<c0146b28>] [<c0147102>] [<c0106e23>]
rpciod S E5922000 0 19775 1 (L-TLB) 19832 19776
Call Trace: [<f89c4912>] [<f89bd2f0>] [<f89bde2b>] [<f89bdc4c>] [<f89c6d84>]
[<f89c6d7c>] [<f89c6d7c>] [<c0105643>] [<f89c6d84>] [<f89c6d84>]
lockd S 7FFFFFFF 0 19776 1 (L-TLB) 19775 23496
Call Trace: [<c0114627>] [<c0136607>] [<f89c0cbe>] [<f898bc44>] [<c0105643>]
login S 00000000 4540 19777 19750 19778 (NOTLB)
Call Trace: [<c0114b12>] [<c011a596>] [<c0106e23>] [<c010002b>]
tcsh S 00000000 0 19778 19777 19802 (NOTLB)
Call Trace: [<c0114b12>] [<c0105fdb>] [<c0106e23>]
su S 00000000 1392 19802 19778 19803 (NOTLB)
Call Trace: [<c011a596>] [<c0106e23>]
bash S 7FFFFFFF 0 19803 19802 19831 (NOTLB)
Call Trace: [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>] [<c0106e23>]
strace T 00000014 4768 19831 19803 19700 (NOTLB)
Call Trace: [<c0106c37>] [<c011a5c1>] [<c0106e74>]
pmd D E2E6C000 0 19700 19831 (NOTLB)
Call Trace: [<c029abdc>] [<c029abdc>] [<c01d877e>] [<c01d881c>] [<c01c74f2>]
[<c01ad856>] [<c01ad873>] [<c01c74f2>] [<c01c74f2>] [<c01d2555>]
[<c0141935>]
[<c0142459>] [<c013fbf1>] [<c013fbff>] [<c01406c6>] [<c01305b2>]
[<c0124cf8>]
[<c0124d0b>] [<c01254db>] [<c0113c9f>] [<c0113b00>] [<c01207c4>]
[<c01417cb>]
[<c0105a43>] [<c0106e23>]
mingetty S 7FFFFFFF 0 19832 1 (NOTLB) 19883 19775
Call Trace: [<c0114627>] [<c01f5dae>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>]
[<c0106e23>]
crond S E3C84000 0 19880 1126 19881 (NOTLB)
Call Trace: [<c012634a>] [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
run-parts S 00000000 0 19881 19880 19884 (NOTLB)
Call Trace: [<c011a596>] [<c0106e23>]
sa1 S 00000000 0 19883 1 19886 (NOTLB) 19832
Call Trace: [<c011a596>] [<c0106e23>]
awk S E4F86000 0 19884 19881 (NOTLB)
Call Trace: [<c014a2f1>] [<c0140add>] [<c0140bb9>] [<c0136a1a>] [<c0106e23>]
sadc S E5C92000 0 19886 19883 (NOTLB)
Call Trace: [<c0137f62>] [<c010ca92>] [<c0106e23>]
in.rlogind S 7FFFFFFF 0 19899 1080 19902 (NOTLB) 19750
Call Trace: [<c01ef51c>] [<c0114627>] [<c0146b28>] [<c0147102>] [<c0106e23>]
login S 00000000 0 19902 19899 19903 (NOTLB)
Call Trace: [<c0114b12>] [<c011a596>] [<c0106e23>]
tcsh S 7FFFFFFF 0 19903 19902 (NOTLB)
Call Trace: [<c01254db>] [<c0114627>] [<c01ed24d>] [<c01e917c>] [<c0136a1a>]
[<c0106e23>]
Warning (Oops_read): Code line not seen, dumping what data is available
Proc; init
>>EIP; c476df0c <___strtok+43aecc0/38440e14> <=====
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; keventd
>>EIP; 00000000 Before first symbol
Trace; c0122fc8 <schedule_task+198/254>
Trace; c0105643 <kernel_thread+23/17c>
Proc; kswapd
>>EIP; c4769fa0 <___strtok+43aad54/38440e14> <=====
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c0114f82 <interruptible_sleep_on_timeout+52/7c>
Trace; c029476e <rwsem_up_write_wake+117b2/2ac44>
Trace; c012fbc9 <free_shortage+3d5/d54>
Trace; c0105643 <kernel_thread+23/17c>
Proc; kreclaimd
>>EIP; 00000286 Before first symbol <=====
Trace; c0114f0d <interruptible_sleep_on+4d/70>
Trace; c0294778 <rwsem_up_write_wake+117bc/2ac44>
Trace; c012fcbb <free_shortage+4c7/d54>
Trace; c010563a <kernel_thread+1a/17c>
Trace; c0105643 <kernel_thread+23/17c>
Proc; bdflush
>>EIP; c477c000 <___strtok+43bcdb4/38440e14> <=====
Trace; c029550e <rwsem_up_write_wake+12552/2ac44>
Trace; c012f813 <free_shortage+1f/d54>
Trace; c013b17e <block_sync_page+1ae/330>
Trace; c0105643 <kernel_thread+23/17c>
Proc; kupdated
>>EIP; c477a000 <___strtok+43badb4/38440e14> <=====
Trace; c01d877e <mrlock_init+fe/12c>
Trace; c01d885d <mrupdatef+29/50>
Trace; c01d3e13 <pagebuf_unlock+68ecf/6cdbc>
Trace; c01ad831 <pagebuf_unlock+428ed/6cdbc>
Trace; c01ad873 <pagebuf_unlock+4292f/6cdbc>
Trace; c01d3e13 <pagebuf_unlock+68ecf/6cdbc>
Trace; c01d3e13 <pagebuf_unlock+68ecf/6cdbc>
Trace; c01d27dd <pagebuf_unlock+67899/6cdbc>
Trace; c016a910 <pagebuf_toss_page+60/4d8>
Trace; c01d2760 <pagebuf_unlock+6781c/6cdbc>
Trace; c01c6037 <pagebuf_unlock+5b0f3/6cdbc>
Trace; c0169674 <pagebuf_write_full_page+64/584>
Trace; c01d2760 <pagebuf_unlock+6781c/6cdbc>
Trace; c01d293c <pagebuf_unlock+679f8/6cdbc>
Trace; c01d294f <pagebuf_unlock+67a0b/6cdbc>
Trace; c01d2760 <pagebuf_unlock+6781c/6cdbc>
Trace; c01d293c <pagebuf_unlock+679f8/6cdbc>
Trace; c0127550 <filemap_fdatasync+c0/120>
Trace; c01d293c <pagebuf_unlock+679f8/6cdbc>
Trace; c014b937 <__mark_inode_dirty+36f/840>
Trace; c013af87 <block_symlink+817/860>
Trace; c013b28d <block_sync_page+2bd/330>
Trace; c0105643 <kernel_thread+23/17c>
Proc; pagebuf_daemon
>>EIP; 00000286 Before first symbol <=====
Trace; c0114f0d <interruptible_sleep_on+4d/70>
Trace; c01685cc <pagebuf_delwri_dequeue+138/2c0>
Trace; c01684d0 <pagebuf_delwri_dequeue+3c/2c0>
Trace; c0105643 <kernel_thread+23/17c>
Proc; mdrecoveryd
>>EIP; f717c000 <___strtok+36dbcdb4/38440e14> <=====
Trace; c02afdf1 <fb_con+21f1/19b20>
Trace; c0230682 <md_update_sb+32a2/3300>
Trace; c0105643 <kernel_thread+23/17c>
Proc; syslogd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; klogd
>>EIP; f5e48000 <___strtok+35a88db4/38440e14> <=====
Trace; c0231ffd <sock_sendmsg+81/a4>
Trace; c0116ffc <panic+1bc/694>
Trace; c01575e5 <remove_proc_entry+125d/2e38>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; portmap
>>EIP; 7fffffff Before first symbol <=====
Trace; c024ebff <tcp_poll+2f/160>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01473d3 <__pollwait+ab3/11c8>
Trace; c014735f <__pollwait+a3f/11c8>
Trace; c0147663 <__pollwait+d43/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; rpc.statd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0136607 <filp_close+9b/a4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; ypbind
>>EIP; 7fffffff Before first symbol <=====
Trace; c024ebff <tcp_poll+2f/160>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01473d3 <__pollwait+ab3/11c8>
Trace; c014735f <__pollwait+a3f/11c8>
Trace; c0147663 <__pollwait+d43/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; ypbind
>>EIP; f5f8bf10 <___strtok+35bcccc4/38440e14> <=====
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c014735f <__pollwait+a3f/11c8>
Trace; c0147663 <__pollwait+d43/11c8>
Trace; c01086c9 <enable_irq+14d/1a0>
Trace; c01086e9 <enable_irq+16d/1a0>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; ypbind
>>EIP; 00000000 Before first symbol
Trace; c0105fdb <__read_lock_failed+27b/2510>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; ypbind
>>EIP; f66b9f78 <___strtok+362fad2c/38440e14> <=====
Trace; c0130870 <free_pages+24/28>
Trace; c0146916 <poll_freewait+46/50>
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c011f1ab <del_timer_sync+bdb/e10>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; automount
>>EIP; f64c2000 <___strtok+36102db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f5cf2000 <___strtok+35932db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f5d56000 <___strtok+35996db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f6712000 <___strtok+36352db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f5e2c000 <___strtok+35a6cdb4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f65b4000 <___strtok+361f4db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f66ac000 <___strtok+362ecdb4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f5d42000 <___strtok+35982db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; c44f2000 <___strtok+4132db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; c44fe000 <___strtok+413edb4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; c4504000 <___strtok+4144db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; c4608000 <___strtok+4248db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; automount
>>EIP; f6454000 <___strtok+36094db4/38440e14> <=====
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; atd
>>EIP; f6689f78 <___strtok+362cad2c/38440e14> <=====
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c011f1ab <del_timer_sync+bdb/e10>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; sshd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; xinetd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0136607 <filp_close+9b/a4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; sendmail
>>EIP; f5b1bf0c <___strtok+3575ccc0/38440e14> <=====
Trace; c01469ae <__pollwait+8e/11c8>
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; crond
>>EIP; f5afdf78 <___strtok+3573ed2c/38440e14> <=====
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c011f1ab <del_timer_sync+bdb/e10>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; in.rlogind
>>EIP; 7fffffff Before first symbol <=====
Trace; c01ef51c <n_tty_ioctl+15fc/1a9c>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; login
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; tcsh
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c0105fdb <__read_lock_failed+27b/2510>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; su
>>EIP; 00000000 Before first symbol
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; bash
>>EIP; 00000000 Before first symbol
Trace; c01ea876 <tty_hung_up_p+19de/200c>
Trace; c01eac4d <tty_hung_up_p+1db5/200c>
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; pmd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0232437 <sock_recvmsg+417/748>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; getpgid2_01
>>EIP; f4f8c000 <___strtok+34bccdb4/38440e14> <=====
Trace; c010ca92 <get_cmos_time+8a2/8b0>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; getpgid2_01
>>EIP; f300c000 <___strtok+32c4cdb4/38440e14> <=====
Trace; c010ca92 <get_cmos_time+8a2/8b0>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; pmd
>>EIP; f43d8000 <___strtok+34018db4/38440e14> <=====
Trace; c01145a7 <iounmap+2f3/35c>
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; renamer01
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c0105fdb <__read_lock_failed+27b/2510>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; renamer
>>EIP; f645c0e0 <___strtok+3609ce94/38440e14> <=====
Trace; c01d2555 <pagebuf_unlock+67611/6cdbc>
Trace; c0141935 <permission+49/84>
Trace; c0105b6b <get_wchan+f7/29c>
Trace; c0105d18 <__down_failed+8/c>
Trace; c028799c <rwsem_up_write_wake+49e0/2ac44>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; rwtest
>>EIP; 00000000 Before first symbol
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166806 <load_nls_default+c0e/1470>
Trace; c0166806 <load_nls_default+c0e/1470>
Trace; c01675d7 <pagebuf_unpin+27/28>
Trace; c019386b <pagebuf_unlock+28927/6cdbc>
Trace; c01145a7 <iounmap+2f3/35c>
Trace; c01b5ff4 <pagebuf_unlock+4b0b0/6cdbc>
Trace; c01b609d <pagebuf_unlock+4b159/6cdbc>
Trace; f880758a <[scsi_mod]scsi_queue_next_request+52/124>
Trace; f88077c0 <[scsi_mod]__scsi_end_request+164/170>
Trace; f8807a9a <[scsi_mod]scsi_io_completion+1da/3bc>
Trace; f8817d86 <[sd_mod]rw_intr+1f6/204>
Trace; f880671d <[scsi_mod]scsi_old_done+3d/600>
Trace; f8806cc6 <[scsi_mod]scsi_old_done+5e6/600>
Trace; c011eb87 <del_timer_sync+5b7/e10>
Trace; c011eee0 <del_timer_sync+910/e10>
Trace; c011ef63 <del_timer_sync+993/e10>
Trace; c010c160 <do_settimeofday+3a8/438>
Trace; c01084d6 <__global_restore_flags+92/bc>
Trace; c01086c9 <enable_irq+14d/1a0>
Trace; c01086e9 <enable_irq+16d/1a0>
Trace; c0106ee4 <__read_lock_failed+1184/2510>
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c0114f82 <interruptible_sleep_on_timeout+52/7c>
Trace; c029476e <rwsem_up_write_wake+117b2/2ac44>
Trace; c012fbc9 <free_shortage+3d5/d54>
Trace; c0105643 <kernel_thread+23/17c>
Trace; c01001c8 Before first symbol
Trace; c01effa0 <batch_entropy_store+224/22c>
Trace; c01f04e2 <add_blkdev_randomness+4d6/504>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; f9c04e6a <END_OF_CODE+1215ca6/????>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; c0166599 <load_nls_default+9a1/1470>
Trace; f894b5b0 <[usbcore]proc_releaseinterface+30/40>
Trace; f8945a36 <[usbcore]usb_find_interface_driver+3a/190>
Trace; c02405e4 <qdisc_restart+14/258>
Trace; c0238d06 <dev_queue_xmit+136/2d8>
Trace; c0234312 <sk_alloc+12/5c>
Trace; c0262e2e <tcp_create_openreq_child+16/4c8>
Trace; c0263099 <tcp_create_openreq_child+281/4c8>
Trace; c02602cb <tcp_v4_syn_recv_sock+113/59c>
Trace; f894b5b0 <[usbcore]proc_releaseinterface+30/40>
Trace; f8945a36 <[usbcore]usb_find_interface_driver+3a/190>
Trace; c02405e4 <qdisc_restart+14/258>
Trace; c0238d06 <dev_queue_xmit+136/2d8>
Trace; c024cca3 <ip_finish_output+1d3/554>
Trace; c023ef54 <nf_hook_slow+128/17c>
Trace; c024b722 <ip_options_undo+a56/a60>
Trace; c024cc00 <ip_finish_output+130/554>
Trace; c024b5cc <ip_options_undo+900/a60>
Trace; c024bb0d <ip_queue_xmit+3e1/dd4>
Trace; c023ef54 <nf_hook_slow+128/17c>
Trace; c024b5cc <ip_options_undo+900/a60>
Trace; c025f81d <tcp_v4_send_check+6d/554>
Trace; c025a59f <tcp_transmit_skb+46b/5f0>
Trace; c025a662 <tcp_transmit_skb+52e/5f0>
Trace; c025b18b <tcp_write_xmit+21f/638>
Trace; c0258648 <tcp_cwnd_application_limited+15c/458>
Trace; c01fff66 <unregister_serial+122/34c>
Trace; c0200000 <unregister_serial+1bc/34c>
Trace; c011731b <panic+4db/694>
Trace; c011737f <panic+53f/694>
Trace; c011745b <panic+61b/694>
Trace; c0117696 <acquire_console_sem+5e/100>
Trace; c011761e <printk+14a/164>
Trace; c01070bf <__read_lock_failed+135f/2510>
Trace; c028cde0 <rwsem_up_write_wake+9e24/2ac44>
Trace; c0107170 <__read_lock_failed+1410/2510>
Trace; c0115783 <sleep_on_timeout+767/854>
Trace; c0115852 <sleep_on_timeout+836/854>
Trace; c0201b6d <handle_scancode+19dd/1a7c>
Trace; c0201d1c <__handle_sysrq_nolock+60/1440>
Trace; c02a3b80 <rwsem_up_write_wake+20bc4/2ac44>
Trace; c02a3af6 <rwsem_up_write_wake+20b3a/2ac44>
Trace; c0201cb3 <handle_sysrq+27/30>
Trace; c02002fc <handle_scancode+16c/1a7c>
Trace; c011eb87 <del_timer_sync+5b7/e10>
Trace; c0201286 <handle_scancode+10f6/1a7c>
Trace; c0201314 <handle_scancode+1184/1a7c>
Trace; c01084d6 <__global_restore_flags+92/bc>
Trace; c01086c9 <enable_irq+14d/1a0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0106ee4 <__read_lock_failed+1184/2510>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0100018 Before first symbol
Trace; c01052ad <enable_hlt+35/b0>
Trace; c0105312 <enable_hlt+9a/b0>
Trace; c011737f <panic+53f/694>
Trace; c028d37d <rwsem_up_write_wake+a3c1/2ac44>
Trace; c010832d <__global_cli+8d/12c>
Trace; c028d392 <rwsem_up_write_wake+a3d6/2ac44>
Trace; c01fb677 <paste_selection+b4f/50b4>
Trace; c01fb640 <paste_selection+b18/50b4>
Trace; c011e81c <del_timer_sync+24c/e10>
Trace; c011eda9 <del_timer_sync+7d9/e10>
Trace; c011b552 <tasklet_kill+ce/130>
Trace; c011b435 <get_fast_time+95d/988>
Trace; c011b2da <get_fast_time+802/988>
Trace; c010870a <enable_irq+18e/1a0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0106ee4 <__read_lock_failed+1184/2510>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0105280 <enable_hlt+8/b0>
Trace; c0100018 Before first symbol
Trace; c01052ad <enable_hlt+35/b0>
Trace; c0105312 <enable_hlt+9a/b0>
Trace; c0105000 <gdt+4d94/5004>
Trace; c01001cf Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c0105fdb <__read_lock_failed+27b/2510>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; symlink02
>>EIP; e4744000 <___strtok+24384db4/38440e14> <=====
Trace; c01305b2 <__alloc_pages+6a/294>
Trace; c01c2792 <pagebuf_unlock+5784e/6cdbc>
Trace; c01d877e <mrlock_init+fe/12c>
Trace; c01d881c <mraccessf+48/60>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01ad856 <pagebuf_unlock+42912/6cdbc>
Trace; c01ad873 <pagebuf_unlock+4292f/6cdbc>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01d2555 <pagebuf_unlock+67611/6cdbc>
Trace; c0141935 <permission+49/84>
Trace; c0142459 <path_walk+7c1/964>
Trace; c01428d8 <__user_walk+3c/58>
Trace; c01360ba <sys_chroot+326/3dc>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; rm
>>EIP; e42fa000 <___strtok+23f3adb4/38440e14> <=====
Trace; c01c312c <pagebuf_unlock+581e8/6cdbc>
Trace; c014b6da <__mark_inode_dirty+112/840>
Trace; c014c7bc <igrab+84/8c>
Trace; c01d7c22 <pagebuf_unlock+6ccde/6cdbc>
Trace; c01ca027 <pagebuf_unlock+5f0e3/6cdbc>
Trace; c01ad99b <pagebuf_unlock+42a57/6cdbc>
Trace; c01d1e67 <pagebuf_unlock+66f23/6cdbc>
Trace; c0143c2f <vfs_unlink+18b/304>
Trace; c0143d32 <vfs_unlink+28e/304>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; top
>>EIP; f4a23f0c <___strtok+34664cc0/38440e14> <=====
Trace; c01469ae <__pollwait+8e/11c8>
Trace; c0114691 <schedule_timeout+81/13c>
Trace; c01145b4 <iounmap+300/35c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; in.rlogind
>>EIP; 7fffffff Before first symbol <=====
Trace; c01ef51c <n_tty_ioctl+15fc/1a9c>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; rpciod
>>EIP; e5922000 <___strtok+25562db4/38440e14> <=====
Trace; f89c4912 <[raid1]__module_using_checksums+f2e0/18a2e>
Trace; f89bd2f0 <[raid1]__module_using_checksums+7cbe/18a2e>
Trace; f89bde2b <[raid1]__module_using_checksums+87f9/18a2e>
Trace; f89bdc4c <[raid1]__module_using_checksums+861a/18a2e>
Trace; f89c6d84 <[raid1]__module_using_checksums+11752/18a2e>
Trace; f89c6d7c <[raid1]__module_using_checksums+1174a/18a2e>
Trace; f89c6d7c <[raid1]__module_using_checksums+1174a/18a2e>
Trace; c0105643 <kernel_thread+23/17c>
Trace; f89c6d84 <[raid1]__module_using_checksums+11752/18a2e>
Trace; f89c6d84 <[raid1]__module_using_checksums+11752/18a2e>
Proc; lockd
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0136607 <filp_close+9b/a4>
Trace; f89c0cbe <[raid1]__module_using_checksums+b68c/18a2e>
Trace; f898bc44 <[ipchains]__module_using_checksums+2a10e/3352a>
Trace; c0105643 <kernel_thread+23/17c>
Proc; login
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Trace; c010002b Before first symbol
Proc; tcsh
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c0105fdb <__read_lock_failed+27b/2510>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; su
>>EIP; 00000000 Before first symbol
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; bash
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; strace
>>EIP; 00000014 Before first symbol <=====
Trace; c0106c37 <__read_lock_failed+ed7/2510>
Trace; c011a5c1 <sys_wait4+3f5/90c>
Trace; c0106e74 <__read_lock_failed+1114/2510>
Proc; pmd
>>EIP; e2e6c000 <___strtok+22aacdb4/38440e14> <=====
Trace; c029abdc <rwsem_up_write_wake+17c20/2ac44>
Trace; c029abdc <rwsem_up_write_wake+17c20/2ac44>
Trace; c01d877e <mrlock_init+fe/12c>
Trace; c01d881c <mraccessf+48/60>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01ad856 <pagebuf_unlock+42912/6cdbc>
Trace; c01ad873 <pagebuf_unlock+4292f/6cdbc>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01c74f2 <pagebuf_unlock+5c5ae/6cdbc>
Trace; c01d2555 <pagebuf_unlock+67611/6cdbc>
Trace; c0141935 <permission+49/84>
Trace; c0142459 <path_walk+7c1/964>
Trace; c013fbf1 <open_exec+19/c0>
Trace; c013fbff <open_exec+27/c0>
Trace; c01406c6 <do_execve+1e/1f0>
Trace; c01305b2 <__alloc_pages+6a/294>
Trace; c0124cf8 <remap_page_range+524/5b0>
Trace; c0124d0b <remap_page_range+537/5b0>
Trace; c01254db <vmtruncate+757/a80>
Trace; c0113c9f <__verify_write+3cf/8f4>
Trace; c0113b00 <__verify_write+230/8f4>
Trace; c01207c4 <notify_parent+7f8/cc0>
Trace; c01417cb <getname+5b/a0>
Trace; c0105a43 <dump_thread+2a7/2d8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; mingetty
>>EIP; 7fffffff Before first symbol <=====
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01f5dae <secure_tcp_sequence_number+4812/4c24>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; crond
>>EIP; e3c84000 <___strtok+238c4db4/38440e14> <=====
Trace; c012634a <do_munmap+56/2b0>
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; run-parts
>>EIP; 00000000 Before first symbol
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; sa1
>>EIP; 00000000 Before first symbol
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; awk
>>EIP; e4f86000 <___strtok+24bc6db4/38440e14> <=====
Trace; c014a2f1 <dput+21/16c>
Trace; c0140add <set_binfmt+245/b4c>
Trace; c0140bb9 <set_binfmt+321/b4c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; sadc
>>EIP; e5c92000 <___strtok+258d2db4/38440e14> <=====
Trace; c0137f62 <file_fsync+1d6/45c>
Trace; c010ca92 <get_cmos_time+8a2/8b0>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; in.rlogind
>>EIP; 7fffffff Before first symbol <=====
Trace; c01ef51c <n_tty_ioctl+15fc/1a9c>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c0146b28 <__pollwait+208/11c8>
Trace; c0147102 <__pollwait+7e2/11c8>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; login
>>EIP; 00000000 Before first symbol
Trace; c0114b12 <schedule+3c6/5c0>
Trace; c011a596 <sys_wait4+3ca/90c>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
Proc; tcsh
>>EIP; 7fffffff Before first symbol <=====
Trace; c01254db <vmtruncate+757/a80>
Trace; c0114627 <schedule_timeout+17/13c>
Trace; c01ed24d <tty_unregister_driver+1f01/2584>
Trace; c01e917c <tty_hung_up_p+2e4/200c>
Trace; c0136a1a <sys_read+8e/c4>
Trace; c0106e23 <__read_lock_failed+10c3/2510>
2 warnings and 6 errors issued. Results may not be reliable.
--
Nate Straz nstraz@xxxxxxx
sgi, inc http://www.sgi.com/
Linux Test Project http://ltp.sf.net/
|