lkcd
[Top] [All Lists]

Re: Module support

To: <lkcd@xxxxxxxxxxx>
Subject: Re: Module support
From: "Masashige Kotani" <m-kotani@xxxxxxxxxxxxxxx>
Date: Mon, 6 Aug 2001 19:25:43 +0900
References: <F13508319A1CD41187DE00508BACED6A020673AD@xxxxxxxxxxxxxxxxx> <20010717113618I.j-kondo@xxxxxxxxxxxxxxx> <3B5552DF.1040103@xxxxxxxxxxxxxxx>
Sender: owner-lkcd@xxxxxxxxxxx
----- Original Message -----
From: T.Ohno <ohno@xxxxxxxxxxxxxxx>
To: <lkcd@xxxxxxxxxxx>
Cc: <j-kondo@xxxxxxxxxxxxxxx>
Sent: Wednesday, July 18, 2001 6:11 PM
Subject: Re: Module support

> Dear all
>
> KONDO Jyunji wrote:
>
> > Hi, hiro.
> >
> >
> >> Has anyone tried to make lcrash support modules. If no one has,
> >> I guess I have to reverse-engineer the query_modules system call.
> >> Am I correct?
> >
> >
> > We are trying to do that by extending lcrash and using ksymoops.
> >
> >
> >> Since most of our proprietary kernel code (device drivers) is
> >> in module, LKCD is not very useful with module support...
> >
> >
> > I agree...
> >
> > At the moment, we are fixing our ideas and we'll be post it to this
> > group soon.
> > Please wait a moment.
>
> I am a Jyunji's co-worker. Now I will post our idea.
>
> I think Andreas's ideas are effective at debugging
> a specific module.
>
> Our idea is different from his one and aims at the
> investigation for detecting a system crash reason.
>
> Currently we are planning following procedure.
>
>   1. Extract symbols for debugging modules from
>      dump file by using a new lcrash option or a
>      dedicated command.
>
>   2. Create a System.map file including the all
>      symbols of loaded modules by passing the
>      result of #1 to ksymoops command.
>
> Using this System.map, the symbols of all modules
> which was loaded in kernel when system crash occurred
> are referable.
>
> In Andreas's idea, the modules installed at system
> crash or module map files must be saved and the map files
> must be loaded each lcrash.
>
> In our idea, as the System.map at the system crash is
> generated, module replacement causes no problem
> (of course after generating System.map)
> and the dump is analyzable at another machine.
>
> Any comment and suggestion are welcomed.
>
>
> Toshio Ohno
>

Hello.
I am a Jyunji's co-worker too.
I am making a new command for module support of lkcd as in the above mail.

I think Andreas's module subcommand is a general-purpose function and very
useful for bug investigation.
But we have to keep symbol information of modules which were loaded at
the time of system crash.
So, when some of modules are replaced, the module subcommand can't
deal with correct symbol information.
To avoid that situation, We should have a facility which saves symbol
information with saving memory dump simultaneously.

So I considered and implemented the facility described below:

  1) Extarct symbol information of modules which were loaded at the
     time of system crash from a dump file,
  2) Merge them to System.map,
  3) Save it as a map file which lcrash can read.

This consists of a command which extracts symbols of modules and a
patch of vmdump script.
This command merges symbol information of modules and System.map by
using ksymoops.
The map file which contains symbol information of modules is saved as
"map.bounds".
By using lcrash with this map file, we can investigate modules' issues.
Since original System.map is also saved, you can use Andreas's module
subcommand if you want.

The program is appended below. Comments are welcome.

--Masashige Kotani


The following is the patch of lkcdutils-1.0-7.tar.gz for our new command.

------------------------------------
diff -Naur lkcdutils-1.0/Makefile lkcdutils-1.0-1/Makefile
--- lkcdutils-1.0/Makefile Sun May  6 01:48:02 2001
+++ lkcdutils-1.0-1/Makefile Fri Aug  3 16:25:09 2001
@@ -27,7 +27,7 @@

 -include .config

-SUB_DIRS  = scripts libklib liballoc librl libsial lcrash
+SUB_DIRS  = scripts libklib liballoc librl libsial lcrash lcd_ksyms

 subdirs_make:
  for dir in $(SUB_DIRS) ; do \
diff -Naur lkcdutils-1.0/lcd_ksyms/Makefile
lkcdutils-1.0-1/lcd_ksyms/Makefile
--- lkcdutils-1.0/lcd_ksyms/Makefile Thu Jan  1 09:00:00 1970
+++ lkcdutils-1.0-1/lcd_ksyms/Makefile Fri Aug  3 16:26:00 2001
@@ -0,0 +1,55 @@
+#
+# Makefile for lcd_ksyms
+#
+# Copyright 1999 Silicon Graphics, Inc. All rights reserved.
+# COPYRIGHT(C) FUJITSU LIMITED 2001
+#
+DEPTH  = .
+include $(DEPTH)/commondefs
+
+TARGET    = lcd_ksyms
+CFILES    = main.c
+LIBS      = -lalloc -lklib -lbfd -liberty -lsial
+LFLAGS    = -static -rdynamic -L. -L$(TOPDIR) -L$(LKCDDIR)/libklib \
+     -L$(LKCDDIR)/liballoc -L$(LKCDDIR)/libsial
+OFILES    = $(CFILES:.c=.o)
+SUB_DIRS  = man
+
+default: $(TARGET)
+
+$(TARGET): subdirs_make
+ $(CC) -o $@ $(LDFLAGS) $(LFLAGS) $(OFILES) $(LIBS)
+
+subdirs_make: $(OFILES)
+ for dir in $(SUB_DIRS) ; do \
+  ( cd $$dir ; make TOPDIR=$(TOPDIR); cd ..) \
+ done
+
+clean:
+ /bin/rm -f *.o *.a
+ /bin/rm -f $(DEPTH)/include/arch
+ for dir in $(SUB_DIRS) ; do \
+  ( cd $$dir ; make TOPDIR=$(TOPDIR) ARCH=$(ARCH) clean; cd .. ); \
+ done
+
+clobber: clean
+ /bin/rm -f $(TARGET)
+
+$(OFILES): $(HEADERS)
+
+#
+# We avoid this for now -- you'll need to do a make clobber followed
+# by a make if you want to fix this later.
+#
+$(HEADERS): symlinks
+
+symlinks:
+ /bin/rm -f $(HEPTH)/arch
+ (cd $(HPATH) ; /bin/ln -sf arch-$(ARCH) arch)
+ (cd $(LKCDDIR)/libklib ; make ARCH=$(ARCH) symlinks)
+
+install: $(TARGET)
+ mkdir -p $(ROOT)/sbin
+ mkdir -p $(ROOT)/usr/man/man1
+ install -m 755 lcd_ksyms $(ROOT)/sbin/lcd_ksyms
+ install -m 644 man/lcd_ksyms.1 $(ROOT)/usr/man/man1
diff -Naur lkcdutils-1.0/lcd_ksyms/commondefs
lkcdutils-1.0-1/lcd_ksyms/commondefs
--- lkcdutils-1.0/lcd_ksyms/commondefs Thu Jan  1 09:00:00 1970
+++ lkcdutils-1.0-1/lcd_ksyms/commondefs Fri Aug  3 16:15:55 2001
@@ -0,0 +1,21 @@
+#
+# Common defines for lcrash makefiles
+#
+# Copyright 1999 Silicon Graphics, Inc. All rights reserved.
+#
+VERSION    = 1.5
+ifeq ($(ARCH),ia64)
+EXTRA_CFLAGS = -gstabs -DARCH=$(ARCH)
+else
+EXTRA_CFLAGS = -gstabs -DARCH=$(ARCH) -DALLOC_DEBUG
+endif
+
+EXTRA_CFLAGS += -Wall -Wstrict-prototypes -D_GNU_SOURCE
+
+HPATH      = $(DEPTH)/../lcrash/include
+LKCDDIR    = $(DEPTH)/..
+HEADERS    = $(HPATH)/command.h $(HPATH)/lcrash.h $(HPATH)/arch/trace.h
+EXTRA_CFLAGS += -I$(HPATH) -I$(LKCDDIR)/libklib/include -I$(TOPDIR)/include
\
+  -I$(LKCDDIR)/libsial
+
+include $(LKCDDIR)/Rules.make
diff -Naur lkcdutils-1.0/lcd_ksyms/main.c lkcdutils-1.0-1/lcd_ksyms/main.c
--- lkcdutils-1.0/lcd_ksyms/main.c Thu Jan  1 09:00:00 1970
+++ lkcdutils-1.0-1/lcd_ksyms/main.c Fri Aug  3 16:15:55 2001
@@ -0,0 +1,187 @@
+/*
+ * Copyright 1999 Silicon Graphics, Inc. All rights reserved.
+ * COPYRIGHT(C) FUJITSU LIMITED 2001
+ *
+ * $Id: main.c,v 1.7 2001/08/01 05:05:28 mkotani Exp $
+ */
+#include <lcrash.h>
+#include <string.h>
+#include <stdio.h>
+#include <linux/module.h>
+
+#define GK_LINE_LENGTH 255 /* ksyms' a line length */
+
+#if (ARCH == ia64)
+void
+*kl_get_ra(void)
+{
+ return(0);
+}
+#endif
+
+
+/*
+ * print_ksyms()
+ *
+ *  This function outputs the kernel symbols
+ *  of ksyms form got from a dump file.
+ */
+void
+print_ksyms(void)
+{
+ syment_t *module_list_sym; /* refer the symbol "module_list" */
+ struct module *next_module; /* modules are followed */
+ struct module temp_module; /* searching module */
+ struct module_symbol temp_syms; /* symbols are followed */
+ char symbol_name[GK_LINE_LENGTH];
+ char mod_name[GK_LINE_LENGTH]; /* symbol name, module name */
+        int i, j;   /* loop counter */
+ k_error_t gb_error;  /* GET_BLOCK result */
+
+
+ kl_reset_error();
+
+ module_list_sym = kl_lkup_symname("module_list");
+ if (module_list_sym == (syment_t *)NULL) {
+  KL_ERROR = KLE_NO_SYMBOLS;
+  return;
+ }
+
+ /* searching module_list */
+ gb_error = GET_BLOCK(module_list_sym->s_addr,
+  sizeof(struct module*), (void*)&next_module);
+ if (gb_error) {
+  KL_ERROR = gb_error;
+  return;
+ }
+
+ if (next_module == NULL) {
+  KL_ERROR = KLE_INVALID_VADDR;
+  return;
+ }
+
+ /* if each name longer than GK_LINE_LENGTH, can stop error*/
+ mod_name[GK_LINE_LENGTH - 1] = '\0';
+ symbol_name[GK_LINE_LENGTH - 1] = '\0';
+
+ /* searching all modules (and a kernel) */
+ for (;next_module != NULL; next_module = temp_module.next) {
+  gb_error = GET_BLOCK(next_module,
+  sizeof(struct module), &temp_module);
+  if (gb_error) {
+   KL_ERROR = gb_error;
+   return;
+  }
+  if (temp_module.name == NULL) {
+   KL_ERROR = KLE_INVALID_VADDR;
+   return;
+  }
+  gb_error = GET_BLOCK(temp_module.name,
+   sizeof(char) * (GK_LINE_LENGTH - 1), mod_name);
+  if (gb_error) {
+   KL_ERROR = gb_error;
+   return;
+  }
+
+  gb_error = GET_BLOCK(temp_module.syms,
+   sizeof(struct module_symbol),
+   (void*)&temp_syms);
+  if (gb_error) {
+   KL_ERROR = gb_error;
+   return;
+  }
+
+  /* searching all symbols */
+  for (i = 0; i < temp_module.nsyms; i++) {
+   gb_error = GET_BLOCK(&temp_module.syms[i],
+    sizeof(struct module_symbol), &temp_syms);
+   if (gb_error) {
+    KL_ERROR = gb_error;
+    return;
+   }
+   if (temp_syms.name == NULL)
+    continue;    // need ?
+
+   /* output */
+   printf("%08x ", (unsigned long*)temp_syms.value);
+   j = 0;
+   do {
+    gb_error = GET_BLOCK(&temp_syms.name[(GK_LINE_LENGTH - 1) * j],
+     GK_LINE_LENGTH - 1, symbol_name);
+    if (gb_error) {
+     KL_ERROR = gb_error;
+     return;
+    }
+    printf("%s", symbol_name);
+    j++;
+   } while (strlen(symbol_name) == (GK_LINE_LENGTH - 1));
+
+   if (mod_name[0] != '\0') {
+    /* case of a module */
+    printf("\t[%s", mod_name);
+    j = 1;
+    while (strlen(mod_name) == (GK_LINE_LENGTH - 1)) {
+     gb_error = GET_BLOCK(&temp_module.name[(GK_LINE_LENGTH - 1) * j],
+      GK_LINE_LENGTH - 1, mod_name);
+     if (gb_error) {
+      KL_ERROR = gb_error;
+      return;
+     }
+     printf("%s", mod_name);
+     j++;
+    }
+    printf("]");
+   }
+   printf("\n");
+  }
+ }
+}
+
+/*
+ * main()
+ */
+int
+main(int argc, char **argv)
+{
+ if (argc != 4) {
+  fprintf(stderr, "Usage: %s map vmdump kerntypes\n", argv[0]);
+  exit(1);
+ }
+
+ /* Set up signal handler to handle SIGINT and to prevent lcrash
+  * from dumping core on SEGV and BUSERR signals.
+  */
+ kl_sig_setup();
+
+ kl_init_klib(argv[1], argv[2], argv[3], 1);
+ if (KL_ERROR) {
+  if (KL_ERROR & (KLE_OPEN_ERROR|KLE_VMDUMP)) {
+   fprintf(KL_ERRORFP, "%s: ", argv[2]);
+  } else if (KL_ERROR &
+   (KLE_OPEN_ERROR|KLE_MAP_FILE|KLE_BAD_MAP_FILE)) {
+   fprintf(KL_ERRORFP, "%s: ", argv[1]);
+  }
+  kl_print_error();
+  exit(1);
+ }
+ if (!STP) {
+  exit(1);
+ }
+ kl_init_kern_info();
+
+ /* output ksyms here */
+ print_ksyms();
+ if (KL_ERROR) {
+  if(KL_ERROR & (KLE_NO_SYMBOLS | KLE_INVALID_VALUE))
+   fprintf(KL_ERRORFP, "module_list: ");
+  kl_print_error();
+  exit(1);
+ }
+
+ free_temp_blocks();
+ exit(0);
+}
+
+
+
+
diff -Naur lkcdutils-1.0/lcd_ksyms/man/Makefile
lkcdutils-1.0-1/lcd_ksyms/man/Makefile
--- lkcdutils-1.0/lcd_ksyms/man/Makefile Thu Jan  1 09:00:00 1970
+++ lkcdutils-1.0-1/lcd_ksyms/man/Makefile Fri Aug  3 16:15:55 2001
@@ -0,0 +1,12 @@
+#
+# Makefile for lcrash man library
+#
+# Copyright 1999 Silicon Graphics, Inc. All rights reserved.
+#
+DEPTH  = ..
+
+all clean clobber install: default
+
+default:
+
+include $(TOPDIR)/Rules.make
diff -Naur lkcdutils-1.0/lcd_ksyms/man/lcd_ksyms.1
lkcdutils-1.0-1/lcd_ksyms/man/lcd_ksyms.1
--- lkcdutils-1.0/lcd_ksyms/man/lcd_ksyms.1 Thu Jan  1 09:00:00 1970
+++ lkcdutils-1.0-1/lcd_ksyms/man/lcd_ksyms.1 Fri Aug  3 17:25:12 2001
@@ -0,0 +1,2 @@
+
+
diff -Naur lkcdutils-1.0/scripts/sbin.vmdump
lkcdutils-1.0-1/scripts/sbin.vmdump
--- lkcdutils-1.0/scripts/sbin.vmdump Wed Oct 18 05:58:50 2000
+++ lkcdutils-1.0-1/scripts/sbin.vmdump Fri Aug  3 18:06:08 2001
@@ -12,6 +12,7 @@
 LCRASH=/sbin/lcrash
 MAP=/boot/System.map
 KERNTYPES=/boot/Kerntypes
+LCD_KSYMS=/sbin/lcd_ksyms

 ###########################################################################
 # Functions
@@ -101,6 +102,19 @@
   /bin/cp -f $MAP map.$BOUNDS
   /bin/cp -f $LCRASH lcrash.$BOUNDS
   /bin/cp -f $KERNTYPES kerntypes.$BOUNDS
+
+  $LCD_KSYMS map.$BOUNDS vmdump.$BOUNDS kerntypes.$BOUNDS \
+                         > temp_ksyms 2>/dev/null
+                if [ $? -ne 0 ] ; then
+                        echo "Can not create a ksyms file." >&2
+                        exit 1
+                fi
+                echo | /usr/bin/ksymoops -s temp_map -k temp_ksyms -L -m
$MAP \
+                        -o /lib/modules/`uname -r`/ 1>/dev/null
+                awk '{print $1,$2,$3}' temp_map > map.$BOUNDS
+                /bin/rm -f temp_ksyms
+                /bin/rm -f temp_map
+
   $LCRASH -r map.$BOUNDS vmdump.$BOUNDS kerntypes.$BOUNDS \
    > analysis.$BOUNDS 2>&1
   if [ $? -ne 0 ] ; then
------------------------------------


And the following is module program for test.

------------------------------------
#define MODULE

#include <linux/kernel.h>
#include <linux/module.h>

static int *bomb;

int init_module(void)
{
 static void bomb_function(void);

 printk("Hello world.\n");

 bomb_function();

 return 0;
}

void cleanup_module(void)
{
 printk("Good bye.\n");
}

static void bomb_function(void)
{
 bomb = 0x0;
 *bomb = 999;
}
------------------------------------


The patch is applied as follows.

in lkcdutils top directory
$ tar zxvf lkcdutils-1.0-7.tar.gz
$ cd lkcdutils-1.0
$ patch -p1 < patchfile
$ ./configure --topdir=kernel directory
$ make install
$ insmod sys_crash.o

And the loading module crashes system.

A result.
(a part of analysis file)
$ lcrash -n 25
>> trace
================================================================
STACK TRACE FOR TASK: 0xc2a54000(insmod)

 0 bomb_function+18 [0xc48af0ae]
 1 init_module+22 [0xc48af076]
 2 sys_init_module+1347 [0xc01147d3]
 3 system_call+44 [0xc0106df4]
================================================================


A result of lcrash which didn't use the new map file.

$ lcrash /boot/System.map-2.4.4 vmdump.25 kerntypes.25
>> trace
================================================================
STACK TRACE FOR TASK: 0xc2a54000(insmod)

================================================================




<Prev in Thread] Current Thread [Next in Thread>