I'd like to report that I have partially implemented our own kernel
modules support in lcrash. It is still "partial" because I have only
verified in live debug mode using /dev/kmem. But I believe it is
not too difficult to make this available also for crash dumps.
Having been watching the discussions here, I noticed most participants
assume to use LKCD in workstation type environments, which is different
from our embedded Linux application in a few key aspects. So, I decided
to develop our own way and to take the risk of code diversion from
the main line.
Well, here are the main things I've so far implemented:
- Cross debugging from SunOS to PPC Linux
LCRASH runs on a SunOS 5 (a.k.a. Solaris 2) workstation to analyze the
kernel memory of a PPC Linux target.
- Read-only live kernel debugging over TCP/IP
The target Linux system runs a modified version of GDBSERVER. LCRASH
on the host and GDBSERVER on the target communicate with each other
over a TCP/IP socket (serial line possible as well) for reading the
target's memory contents. If the vmdump file name is in the form of
"xxxx:yyyy," LCRASH will interpret this as "xxxx" is the target's
host name or IP address and that "yyyy" is the port number and will
open a socket. If the vmdump file is a tty, LCRASH will open the
serial port for communication. The protocol is based on the
GDB/GDBSERVER protocol with minor enhancements.
- Accessing virtual addresses
LCRASH (GDBSERVER) uses /dev/kmem, not /dev/mem, for accessing the
target's kernel memory for two reasons:
1) A simple port of x86 code for PPC could not correctly access
addresses above high_memory probably because of some MMU difference;
2) Due to our limited crash dump device size, we must implement
a selective dumping scheme which will save only the "interesting"
memory segments and will anyway need to work in the virtual address
- No use of System.map
To reduce the number of files to be archived for each release, I made
LCRASH use the ELF kernel executable with full debug information for
both kernel symbols and kernel structure information. LCRASH popen's
"nm vmlinux|grep xyz" internally. This eliminates the need of separate
"systypes" file creation (and archiving) as well.
- Modules support
Before going to show the prompt, LCRASH reads the target memory around
"module_list" to assemble an internal kernel module list for the target.
This has been the case for live kernel debugging. We may attach a
kernel module directory structure to the crash dump when we are ready
to deal with a real kernel crash dump.
The new "module" command takes one argument to tell it the "root"
directory location of the target file system image visible on the host's
file system. Under this directory, LCRASH reads each module file listed
in the internal module list, relocates its symbols according to the
relocation base addresses in the list, then adds those symbols to
LCRASH's internal symbol B-tree. For example, for "module /home/hiro/oss,"
the target's "/lib/modules/xyz.o" module is read from
"/home/hiro/oss/lib/modules/xyz.o" on the host.
So far my LCRASH is working fine as a read-only kernel debugger with
the desired modules support. I need to make the real crash dump analysis
available with the above features. Then I plan to implement a couple
of more enhancements like vmdump to ELF core file for using GDB for
source level analysis, etc.
BTW, I've noticed a couple of issues in the current LCRASH source code
- Non-stopping Makefiles
The whole make process from the top continues to the end even if a
make in a subdirectory fails. This pretends that the make process was
successful and thus often appears very misleading.
- Misleading naming
"vmdump" should be called "pmdump" because it is a physical memory;
"core_kmem" should be called "core_mem" because it accesses /dev/mem,
- No endian consideration
Probably because LCRASH was developed with only self-operation in mind,
there's no endian care: kl_int() and its relatives do not work in a
cross environment where the host and the target have different endians.