Glad to have you join the LKCD list. I'll address your questions
Moo Kim wrote:
> I recently joined the lkcd mailing and I have a newbie
> question. I am fairly new to Linux.
> I see there are two version of SVR4 UNIX crash-like
> utilities for Linux:
> 1. crash2.5 : http://www.missioncriticallinux.com/technology/crash/
> 2. lcrash 3.0 : http://oss.sgi.com/projects/lkcd/
> Is there any documentation/FAQ available that describes the
> difference between two, pros/cons, etc ? I tried to look
> thru the online documentation on each site and deja news,
> but could not find any relavent info. If someone had
> hands-on experience with both utilities, could you please
> share ?
I have only limited knowledge of the crash2.5 tool from Mission
Critical Linux, so I'll leave feedback on that tool for someone
who is more familiar with it. I am, however, very familiar with
lcrash, as I am its primary developer.
> We (in NCR Corp) is looking into porting one of the existing
> software product (Teradata RDBMS) on Linux and this product
> has the kernel device driver which will be implemented as
> loadable module. I would be interested in knowing whether
> lcrash3.0 currently provide the following capability:
> README file in crash2.5 stated as one needs to recompile
> the kernel with -g option (to get debug symbol info).
> Does lcrash3.0 require for kernel to compie with -g option
> as well ? Because -g option for kernel build may not be
> desireable for the kernel running on a production box.
> It would be nice to be able to use lcrash for analyzing Linux
> panic dumps (as well as live system) that have kernel compiled
> with/without -g option. This same question also goes to
> kernel module as well. I see lcrash3.0 provides the
> 'Kerntypes' target and perhaps this is the way to load the
> kernel symbol information for non-debug kernel (?).
You are correct in your assumption. Prior to the 3.0 release, lcrash
was built right along with the kernel. It knew the exact arrangement
of various kernel data structures because they were compiled in. The
downside to this was, that the copy of lcrash you used had to be the
one that matched the individual kernel you were examining. With the
3.0 release, almost all direct references to kernel header files
were removed. Instead, a facility was added that would allow lcrash
to determine the proper offsets to fields in kernel structures using
the type information contained in the stabs portion of the symbol table.
In order to get stabs data, you need to compile using the -gstabs
compiler option. This, of course, is not practical when you are
talking about a production kernel. What we did, however, was add a
dummy module to the kernel (init/kerntypes.c) that gets built with
the -gstabs flag set. The resulting object file never gets included
in the kernel build. Instead, it is renamed to be the Kerntypes target
you refer to above. This, along with the System.map target provide
lcrash with all the type and symbol mapping information it needs. Using
this mechanism, it is possible for a single lcrash binary to read core
dumps from multiple kernels (include 2.2.x and 2.4.x kernels) -- all
without recompiling. A side benefit to this new approach was that we
no longer needed to include lcrash in the kernel source tree. The lcrash
utility is now installed via the lkcdutils RPM (full source is also
available for download).
One other difference that I know of has to do with the generation of
kernel task backtraces. The lcrash backtrace facility does not require
that your i386 kernel be built using one of the registers as a frame
pointer. I believe that that this is a requirement with the MC Linux tool.
Also, lcrash, and LKCD, provide support for Intel ia64 boxes.
I might note that, in addition to the differences in crash dump analysis
tools, there are big differences in the kernel modifications necessary
to generate system dumps after a PANIC. If you are interested in a more
detailed comparison, you can take a look at the paper we presented at
LinuxWorld 2000 in New York, NY. The paper and presentation slides are
available for download from the following URL:
Hope this answers your questions.