Hi, Mark. I didn't hear from Tom, so I'll chime in. Did you grab
the latest lkcdutils RPM from the 3.1.1 directory? In addition,
I can build you a debug version that you can try out -- it might
get us a little more information. Also, if you have a place where
I can take a look at the crash dumps, I could probably answer the
issue even faster.
Typically this is either a problem with the Kernsyms file, or some
other symbol related issue. If your kernel/Kernksyms/map all match
up properly against your vmdump, then things should work great.
Let me know how I can help. Thanks, Mark.
--Matt
P.S. Sorry for the delay in response. :)
Mark Price wrote:
>
> Hi Folks,
>
> I recompiled my kernel (2.2.18) with the rawio and lckd patches,
> installed
> the lckd utils rpm, and rebooted.
>
> I can panic the system and the vmdump successfully retrieves the dump
> from my swap partition.
>
> However when I try and run lcrash on either the live system or against
> the dump it core dumps with a SIGSEGV.
>
> Nothing useful in the gdb output, so I grabbed the source and recompiled
> with -g flag, and made sure it wasn't being stripped. Still cores, and
> the debug output still gives me no clues.
>
> [root@w-mprice2 /root]# gdb --core=./core --exec=/sbin/lcrash
> --symbols=/sbin/lcrash
> GNU gdb 5.0
> Copyright 2000 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "i386-redhat-linux"...
> Core was generated by `lcrash'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x0 in ?? ()
> (gdb) bt
> #0 0x0 in ?? ()
> (gdb) [root@w-mprice2 /root]#
> [root@w-mprice2 /root]# exit
>
> So looks like lcrash stomped all over its own stack.
>
> truss gave me the following output (see below), but without a stack to
> give me an
> idea of where in the code we are I'm not sure it helps.
>
> It looks like it is in the process of reading the Kerntypes file.
>
> Has anyone here experienced these symptoms, can you offer any advice?
>
> Cheers, Mark.
>
> root wrote:
> >
> > execve("/sbin/lcrash", ["lcrash"], [/* 22 vars */]) = 0
> > fcntl64(0, F_GETFD) = -1 ENOSYS (Function not
> > implemented)
> > fcntl(0, F_GETFD) = 0
> > fcntl(1, F_GETFD) = 0
> > fcntl(2, F_GETFD) = 0
> > _sysctl({{CTL_KERN, KERN_OSRELEASE}, 2, "2.2.18", 6, NULL, 0}) = 0
> > geteuid32() = -1 ENOSYS (Function not
> > implemented)
> > geteuid() = 0
> > getuid() = 0
> > getegid() = 0
> > getgid() = 0
> > brk(0) = 0x8174b7c
> > brk(0x8174b9c) = 0x8174b9c
> > brk(0x8175000) = 0x8175000
> > getpid() = 2305
> > fstat64(1, 0xbffff280) = -1 ENOSYS (Function not
> > implemented)
> > fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(4, 1), ...}) = 0
> > old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > 0) = 0x40000000
> > ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
> > write(1, "map = /boot/System.map, vmdump ="..., 89) = 89
> > write(1, "\n", 1) = 1
> > write(1, "Please wait...", 14) = 14
> > rt_sigaction(SIGINT, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGPIPE, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGABRT, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGSEGV, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGBUS, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > open("/dev/mem", O_RDONLY) = 4
> > brk(0x8176000) = 0x8176000
> > open("/boot/System.map", O_RDONLY) = 5
> > fstat(5, {st_mode=S_IFREG|0644, st_size=216270, ...}) = 0
> > old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > 0) = 0x40001000
> > read(5, "c0100000 A _text\nc0100000 t star"..., 4096) = 4096
> > write(2, ".", 1) = 1
> > brk(0x8177000) = 0x8177000
> > brk(0x8178000) = 0x8178000
> > read(5, "16_interrupt\nc0108da0 t IRQ0x17_"..., 4096) = 4096
> > brk(0x8179000) = 0x8179000
> > brk(0x817a000) = 0x817a000
> > read(5, "rrupt\nc01093ac t IRQ0xa4_interru"..., 4096) = 4096
> > brk(0x817b000) = 0x817b000
> > brk(0x817c000) = 0x817c000
> > read(5, "nfig_dword\nc010de1c t pci_conf2_"..., 4096) = 4096
> > brk(0x817d000) = 0x817d000
> > brk(0x817e000) = 0x817e000
> > brk(0x817f000) = 0x817f000
> > read(5, "dma\nc0113900 T free_uid\nc0113970"..., 4096) = 4096
> > brk(0x8180000) = 0x8180000
> > brk(0x8181000) = 0x8181000
> > read(5, "550 t do_no_page\nc011d644 T hand"..., 4096) = 4096
> > brk(0x8182000) = 0x8182000
> > brk(0x8183000) = 0x8183000
> > brk(0x8184000) = 0x8184000
> > read(5, "94 T kdevname\nc0128fbc T bdevnam"..., 4096) = 4096
> > brk(0x8185000) = 0x8185000
> > write(2, ".", 1) = 1
> > brk(0x8186000) = 0x8186000
> > read(5, "_select\nc0132ffc t do_poll\nc0133"..., 4096) = 4096
> > brk(0x8187000) = 0x8187000
> > brk(0x8188000) = 0x8188000
> > brk(0x8189000) = 0x8189000
> > read(5, "ext2_permission\nc013e3b4 T ext2_"..., 4096) = 4096
> > brk(0x818a000) = 0x818a000
> > brk(0x818b000) = 0x818b000
> > read(5, " T de_put\nc014bca4 t proc_put_in"..., 4096) = 4096
> > brk(0x818c000) = 0x818c000
> > brk(0x818d000) = 0x818d000
> > read(5, "c\nc0155b78 t nfs_find_read\nc0155"..., 4096) = 4096
> > brk(0x818e000) = 0x818e000
> > brk(0x818f000) = 0x818f000
> > brk(0x8190000) = 0x8190000
> > read(5, "c_symlink\nc015dbb8 t nfs3_proc_m"..., 4096) = 4096
> > brk(0x8191000) = 0x8191000
> > brk(0x8192000) = 0x8192000
> > read(5, "0169370 T nfssvc_encode_readlink"..., 4096) = 4096
> > brk(0x8193000) = 0x8193000
> > brk(0x8194000) = 0x8194000
> > write(2, ".", 1) = 1
> > read(5, "t nlmsvc_proc_granted_msg\nc01700"..., 4096) = 4096
> > brk(0x8195000) = 0x8195000
> > brk(0x8196000) = 0x8196000
> > read(5, "de\nc0174130 t autofs_put_super\nc"..., 4096) = 4096
> > brk(0x8197000) = 0x8197000
> > brk(0x8198000) = 0x8198000
> > brk(0x8199000) = 0x8199000
> > read(5, "7a6a8 T dev_getbyhwaddr\nc017a6f4"..., 4096) = 4096
> > brk(0x819a000) = 0x819a000
> > brk(0x819b000) = 0x819b000
> > read(5, "181d70 t ip_masq_user_info\nc0181"..., 4096) = 4096
> > brk(0x819c000) = 0x819c000
> > brk(0x819d000) = 0x819d000
> > brk(0x819e000) = 0x819e000
> > read(5, "8 t tcp_ack\nc018d40c T tcp_timew"..., 4096) = 4096
> > brk(0x819f000) = 0x819f000
> > brk(0x81a0000) = 0x81a0000
> > read(5, "fa\nc019815c t inet_set_ifa\nc0198"..., 4096) = 4096
> > brk(0x81a1000) = 0x81a1000
> > brk(0x81a2000) = 0x81a2000
> > read(5, "1a0f0c t packet_create\nc01a1064 "..., 4096) = 4096
> > brk(0x81a3000) = 0x81a3000
> > brk(0x81a4000) = 0x81a4000
> > write(2, ".", 1) = 1
> > brk(0x81a5000) = 0x81a5000
> > read(5, "a8 T svc_release_buffer\nc01a84c4"..., 4096) = 4096
> > brk(0x81a6000) = 0x81a6000
> > brk(0x81a7000) = 0x81a7000
> > read(5, "ing_dma\nc01b05c0 t set_pio_mode\n"..., 4096) = 4096
> > brk(0x81a8000) = 0x81a8000
> > brk(0x81a9000) = 0x81a9000
> > read(5, "ary_port_responding\nc01ba97c t c"..., 4096) = 4096
> > brk(0x81aa000) = 0x81aa000
> > brk(0x81ab000) = 0x81ab000
> > brk(0x81ac000) = 0x81ac000
> > read(5, "_read_proc\nc01c0ec4 t misc_open\n"..., 4096) = 4096
> > brk(0x81ad000) = 0x81ad000
> > brk(0x81ae000) = 0x81ae000
> > read(5, "dstate\nc01cb06c T register_leds\n"..., 4096) = 4096
> > brk(0x81af000) = 0x81af000
> > brk(0x81b0000) = 0x81b0000
> > brk(0x81b1000) = 0x81b1000
> > read(5, "8 t keyboard_interrupt\nc01d644c "..., 4096) = 4096
> > brk(0x81b2000) = 0x81b2000
> > brk(0x81b3000) = 0x81b3000
> > write(2, ".", 1) = 1
> > read(5, " get_pci_list\nc01e3888 T pcibios"..., 4096) = 4096
> > brk(0x81b4000) = 0x81b4000
> > read(5, "02168c7 ? __kstrtab___global_sti"..., 4096) = 4096
> > read(5, "c ? __kstrtab_dput\nc0216f01 ? __"..., 4096) = 4096
> > read(5, "__kstrtab_disable_hlt\nc0217580 ?"..., 4096) = 4096
> > read(5, "ster_cvf_format\nc0217bf0 ? __kst"..., 4096) = 4096
> > read(5, "r\nc021829d ? __kstrtab_pneigh_lo"..., 4096) = 4096
> > read(5, "ll\nc021891a ? __kstrtab_call_in_"..., 4096) = 4096
> > read(5, " __kstrtab_loop_unregister_trans"..., 4096) = 4096
> > read(5, "1a418 ? __ksymtab_MCA_bus\nc021a4"..., 4096) = 4096
> > read(5, "021a7d8 ? __ksymtab_lookup_dentr"..., 4096) = 4096
> > read(5, "lip_buffer_push\nc021ab88 ? __ksy"..., 4096) = 4096
> > read(5, "ode\nc021af28 ? __ksymtab_global_"..., 4096) = 4096
> > read(5, "1b2c0 ? __ksymtab_sock_getsockop"..., 4096) = 4096
> > read(5, "_notifier\nc021b660 ? __ksymtab_u"..., 4096) = 4096
> > read(5, "oto\nc021b9d0 ? __ksymtab_xprt_de"..., 4096) = 4096
> > read(5, "b_set_selection\nc021bd48 ? __ksy"..., 4096) = 4096
> > brk(0x81b5000) = 0x81b5000
> > read(5, " cpucount\nc021f8c4 d smp_call_fu"..., 4096) = 4096
> > brk(0x81b6000) = 0x81b6000
> > brk(0x81b7000) = 0x81b7000
> > read(5, "_inode_operations\nc022a000 D def"..., 4096) = 4096
> > read(5, "\nc022bb40 d proc_root_ioports\nc0"..., 4096) = 4096
> > brk(0x81b8000) = 0x81b8000
> > read(5, "ion1\nc022e3f0 d nlm_version3\nc02"..., 4096) = 4096
> > brk(0x81b9000) = 0x81b9000
> > brk(0x81ba000) = 0x81ba000
> > read(5, "c0230990 D tcp_tw_death_row_slot"..., 4096) = 4096
> > brk(0x81bb000) = 0x81bb000
> > read(5, "\nc0232b94 d default_drive_params"..., 4096) = 4096
> > read(5, "e0 D plain_map\nc02363e0 D shift_"..., 4096) = 4096
> > brk(0x81bc000) = 0x81bc000
> > brk(0x81bd000) = 0x81bd000
> > read(5, "0241430 t pcibios_fixup_devices\n"..., 4096) = 4096
> > brk(0x81be000) = 0x81be000
> > brk(0x81bf000) = 0x81bf000
> > read(5, " T pckbd_init_hw\nc0249498 T aux_"..., 4096) = 4096
> > brk(0x81c0000) = 0x81c0000
> > brk(0x81c1000) = 0x81c1000
> > read(5, " b temparea.0\nc02796e0 b reply_b"..., 4096) = 4096
> > brk(0x81c2000) = 0x81c2000
> > write(2, ".", 1) = 1
> > brk(0x81c3000) = 0x81c3000
> > brk(0x81c4000) = 0x81c4000
> > read(5, "310 B proc_net\nc0282314 B proc_s"..., 4096) = 3278
> > brk(0x81c5000) = 0x81c5000
> > brk(0x81c6000) = 0x81c6000
> > read(5, "", 4096) = 0
> > old_mmap(NULL, 167936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > 0) = 0x40002000
> > write(2, ".", 1) = 1
> > write(2, ".", 1) = 1
> > write(2, ".", 1) = 1
> > write(2, ".", 1) = 1
> > write(2, ".", 1) = 1
> > lseek(4, 2221736, SEEK_SET) = 2221736
> > read(4, "\310\343(\300", 4) = 4
> > brk(0x81c8000) = 0x81c8000
> > open("/boot/Kerntypes", O_RDONLY) = 6
> > fstat(6, {st_mode=S_IFREG|0644, st_size=94920, ...}) = 0
> > old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> > 0) = 0x4002b000
> > _llseek(6, 0, [0], SEEK_SET) = 0
> > read(6, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\1\0\3\0\1\0\0\0\0\0\0\0"..., 4096)
> > = 4096
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 4096, [4096], SEEK_SET) = 0
> > _llseek(6, 94208, [94208], SEEK_SET) = 0
> > read(6, "omment\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096)
> > = 712
> > _llseek(6, 90112, [90112], SEEK_SET) = 0
> > read(6, "0,32;it_prof_value:(0,5),1312,32"..., 4096) = 4096
> > read(6, "omment\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096)
> > = 712
> > brk(0x81cb000) = 0x81cb000
> > brk(0x81e0000) = 0x81e0000
> > _llseek(6, 0, [0], SEEK_SET) = 0
> > read(6, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\1\0\3\0\1\0\0\0\0\0\0\0"..., 4096)
> > = 4096
> > read(6, "\0\0\0\0\242\0\0\0\0\0\0\0\303O\0\0\200\0\0\0\0\0\0\0\234"...,
> > 4096) = 4096
> > read(6, "\202\0\0\0\0\0\0\0d!\1\0\200\0\0\0\0\0\0\0\242!\1\0\200"..., 4096)
> > = 4096
> > read(6, "(10,5)\0int16_t:t(7,31)=(10,4)\0u_"..., 77824) = 77824
> > read(6, "0,32;it_prof_value:(0,5),1312,32"..., 4096) = 4096
> > brk(0x81e1000) = 0x81e1000
> > brk(0x81e2000) = 0x81e2000
> > brk(0x81e3000) = 0x81e3000
> > brk(0x81e4000) = 0x81e4000
> > brk(0x81e5000) = 0x81e5000
> > brk(0x81e6000) = 0x81e6000
> > brk(0x81e7000) = 0x81e7000
> > brk(0x81e8000) = 0x81e8000
> > brk(0x81e9000) = 0x81e9000
> > brk(0x81ea000) = 0x81ea000
> > brk(0x81eb000) = 0x81eb000
> > brk(0x81ec000) = 0x81ec000
> > brk(0x81ed000) = 0x81ed000
> > --- SIGSEGV (Segmentation fault) ---
> > rt_sigaction(SIGINT, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGPIPE, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGABRT, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGSEGV, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > rt_sigaction(SIGBUS, {0x8069200, [],
> > SA_NOMASK|SA_ONESHOT|SA_SIGINFO|0x4000000}, NULL, 8) = 0
> > --- SIGSEGV (Segmentation fault) ---
> > --- SIGSEGV (Segmentation fault) ---
> > +++ killed by SIGSEGV +++
|