From ncunningham@cyclades.com Sat Apr 1 03:35:35 2006 Received: with ECARTIS (v1.0.0; list kdb); Sat, 01 Apr 2006 03:35:53 -0800 (PST) Received: from cust8446.nsw01.dataco.com.au (cust8446.nsw01.dataco.com.au [203.171.93.254]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id k31BZTgC019141 for ; Sat, 1 Apr 2006 03:35:33 -0800 Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by cust8446.nsw01.dataco.com.au (Postfix) with ESMTP id 23A1A64519; Sat, 1 Apr 2006 19:33:03 +1000 (EST) From: Nigel Cunningham Organization: Cyclades Corporation To: Peter Wainwright Subject: Re: KDB support for x86_64? Date: Sat, 1 Apr 2006 19:32:56 +1000 User-Agent: KMail/1.9.1 Cc: jfv@bluesong.net, kdb@oss.sgi.com References: <1143374303.4233.0.camel@localhost.localdomain> <20060327013234.GC7617@trane.bluesong.net> <1143877075.5788.8.camel@localhost.localdomain> In-Reply-To: <1143877075.5788.8.camel@localhost.localdomain> MIME-Version: 1.0 Content-type: text/plain Content-Transfer-Encoding: 8bit Message-Id: <200604011933.02334.ncunningham@cyclades.com> X-archive-position: 1179 X-ecartis-version: Ecartis v1.0.0 Sender: kdb-bounce@oss.sgi.com Errors-to: kdb-bounce@oss.sgi.com X-original-sender: ncunningham@cyclades.com Precedence: bulk X-list: kdb Hi. On Saturday 01 April 2006 17:37, Peter Wainwright wrote: > On Sun, 2006-03-26 at 17:32 -0800, Jack Vogel wrote: > > On Mon, Mar 27, 2006 at 08:00:46AM +1000, Nigel Cunningham wrote: > > > Hi. > > > > > > On Sunday 26 March 2006 21:58, Peter Wainwright wrote: > > > > Is KDB still maintained for the x86_64 architecture? oss.sgi.com has > > > > x86_64 patches for kernel 2.6.14 but not for more recent > > > > kernels. I tried to forward-port it to 2.6.16 myself but it is > > > > not giving a proper backtrace (%rsp seems to be wrong at > > > > breakpoints). > > > > > > > > Peter Wainwright > > > > > > I would find it useful too. I've forward ported it as well, and also > > > have the same backtrace problem. I guess it's probably simple to fix, > > > but it doesn't irk me enough that I've bothered to try to fix it. > > > > > > Regards, > > > > > > Nigel > > > > Did you have the problem with the last patch, there was > > some backtrace changes that I accepted that I did not have > > time to test. It would be useful if you could check a before > > and after of that code. > > > > Regards, > > > > Jack > > Sometime recently the following change occurred in > arch/x86_64/kernel/traps.c: > > - set_system_gate(3,&int3); > + set_system_gate_ist(3,&int3,DEBUG_STACK); /* int3 can be called > from all */ > > The int3 handler now runs in a special interrupt stack using > the "long-mode interrupt-stack table". > So calculating the process %rsp from the address of the pt_regs > structure no longer works. How to figure out the address of the > original stack is something I haven't figured out yet because I > have no time to read the fine "AMD64 Architecture Programmer's Manual". > > Peter Thanks for the info. Whenever you have a patch, I'll be happy to test it. I'm not about to complain though, because people don't always get immediate replies from me either. (And even if they did, your time is your time, not mine!). Thanks! Nigel -- Attached file included as plaintext by Ecartis -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQBELkjON0y+n1M3mo0RAke4AKChw3n/NrhWpA0UN6jzXQ2qZ5o/2QCcDL/1 4wiyfL5XYPPgzmz4joSoc18= =emZs -----END PGP SIGNATURE----- --------------------------- Use http://oss.sgi.com/ecartis to modify your settings or to unsubscribe. From ncunningham@cyclades.com Sat Apr 1 03:35:35 2006 Received: with ECARTIS (v1.0.0; list kdb); Sat, 01 Apr 2006 03:35:52 -0800 (PST) Received: from cust8446.nsw01.dataco.com.au (cust8446.nsw01.dataco.com.au [203.171.93.254]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id k31BZTgC019142 for ; Sat, 1 Apr 2006 03:35:33 -0800 Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by cust8446.nsw01.dataco.com.au (Postfix) with ESMTP id 552F26456D; Sat, 1 Apr 2006 19:38:12 +1000 (EST) From: Nigel Cunningham Organization: Cyclades Corporation To: Peter Wainwright Subject: Re: KDB support for x86_64? Date: Sat, 1 Apr 2006 19:38:05 +1000 User-Agent: KMail/1.9.1 Cc: jfv@bluesong.net, kdb@oss.sgi.com References: <1143374303.4233.0.camel@localhost.localdomain> <20060327013234.GC7617@trane.bluesong.net> <1143884167.3533.4.camel@localhost.localdomain> In-Reply-To: <1143884167.3533.4.camel@localhost.localdomain> MIME-Version: 1.0 Content-type: text/plain Content-Transfer-Encoding: 8bit Message-Id: <200604011938.11514.ncunningham@cyclades.com> X-archive-position: 1178 X-ecartis-version: Ecartis v1.0.0 Sender: kdb-bounce@oss.sgi.com Errors-to: kdb-bounce@oss.sgi.com X-original-sender: ncunningham@cyclades.com Precedence: bulk X-list: kdb Hi. On Saturday 01 April 2006 19:36, Peter Wainwright wrote: > On Sun, 2006-03-26 at 17:32 -0800, Jack Vogel wrote: > > On Mon, Mar 27, 2006 at 08:00:46AM +1000, Nigel Cunningham wrote: > > > Hi. > > > > > > On Sunday 26 March 2006 21:58, Peter Wainwright wrote: > > > > Is KDB still maintained for the x86_64 architecture? oss.sgi.com has > > > > x86_64 patches for kernel 2.6.14 but not for more recent > > > > kernels. I tried to forward-port it to 2.6.16 myself but it is > > > > not giving a proper backtrace (%rsp seems to be wrong at > > > > breakpoints). > > > > > > > > Peter Wainwright > > > > > > I would find it useful too. I've forward ported it as well, and also > > > have the same backtrace problem. I guess it's probably simple to fix, > > > but it doesn't irk me enough that I've bothered to try to fix it. > > > > > > Regards, > > > > > > Nigel > > > > Did you have the problem with the last patch, there was > > some backtrace changes that I accepted that I did not have > > time to test. It would be useful if you could check a before > > and after of that code. > > > > Regards, > > > > Jack > > The change I mentioned in the previous mail actually seems to > simplify things for the debugger, since SS/SP is now saved on > the interrupt stack at breakpoints. > This patch seems to work for me. Unless you can find some path > into kdba_getregcontents where SS/SP is NOT saved in regs? I'll give it a go. Regards, Nigel -- Attached file included as plaintext by Ecartis -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQBELkoDN0y+n1M3mo0RApAxAKDCgh8kLSZ+eLLogV961HVlzu91igCg6pDr vNQZpxqVTCEUs2RKQSZuhBI= =274P -----END PGP SIGNATURE----- --------------------------- Use http://oss.sgi.com/ecartis to modify your settings or to unsubscribe. From ncunningham@cyclades.com Fri Apr 7 14:18:29 2006 Received: with ECARTIS (v1.0.0; list kdb); Fri, 07 Apr 2006 14:19:12 -0700 (PDT) Received: from cust8446.nsw01.dataco.com.au ([203.171.93.254]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with ESMTP id k37LIOgC007171 for ; Fri, 7 Apr 2006 14:18:29 -0700 Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by cust8446.nsw01.dataco.com.au (Postfix) with ESMTP id AE7C2212F6E; Sat, 8 Apr 2006 07:17:43 +1000 (EST) From: Nigel Cunningham Organization: Cyclades Corporation To: Peter Wainwright Subject: Re: KDB support for x86_64? Date: Sat, 8 Apr 2006 07:17:38 +1000 User-Agent: KMail/1.9.1 Cc: jfv@bluesong.net, kdb@oss.sgi.com References: <1143374303.4233.0.camel@localhost.localdomain> <20060327013234.GC7617@trane.bluesong.net> <1143884167.3533.4.camel@localhost.localdomain> In-Reply-To: <1143884167.3533.4.camel@localhost.localdomain> MIME-Version: 1.0 Content-type: text/plain Content-Transfer-Encoding: 8bit Message-Id: <200604080717.42148.ncunningham@cyclades.com> X-archive-position: 1180 X-ecartis-version: Ecartis v1.0.0 Sender: kdb-bounce@oss.sgi.com Errors-to: kdb-bounce@oss.sgi.com X-original-sender: ncunningham@cyclades.com Precedence: bulk X-list: kdb Hi Peter. Sorry for the slow reply. With the patch you sent, it now works fine. Regards, Nigel -- Attached file included as plaintext by Ecartis -- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQBENtb2N0y+n1M3mo0RArZZAKDkNqQoTtwEoVRgnWJYulEgADntmwCeIxUa +UImRYG32EQsqtwXcW8NL2A= =6Bcz -----END PGP SIGNATURE----- --------------------------- Use http://oss.sgi.com/ecartis to modify your settings or to unsubscribe. From kaos@sgi.com Tue Apr 18 22:49:55 2006 Received: with ECARTIS (v1.0.0; list kdb); Tue, 18 Apr 2006 22:50:03 -0700 (PDT) Received: from larry.melbourne.sgi.com (mverd138.asia.info.net [61.14.31.138]) by oss.sgi.com (8.12.10/8.12.10/SuSE Linux 0.7) with SMTP id k3J5nrgC018309 for ; Tue, 18 Apr 2006 22:49:54 -0700 Received: from kao2.melbourne.sgi.com (kao2.melbourne.sgi.com [134.14.55.180]) by larry.melbourne.sgi.com (950413.SGI.8.6.12/950213.SGI.AUTOCF) via ESMTP id PAA28493 for ; Wed, 19 Apr 2006 15:49:51 +1000 Received: by kao2.melbourne.sgi.com (Postfix, from userid 16331) id 6964F2EAF; Wed, 19 Apr 2006 15:49:54 +1000 (EST) Received: from kao2.melbourne.sgi.com (localhost [127.0.0.1]) by kao2.melbourne.sgi.com (Postfix) with ESMTP id 5945680177 for ; Wed, 19 Apr 2006 15:49:54 +1000 (EST) X-Mailer: exmh version 2.7.0 06/18/2004 with nmh-1.1-RC1 From: Keith Owens To: kdb@oss.sgi.com Subject: [RFC] New backtrace algorithm for i386/x86_64 Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Date: Wed, 19 Apr 2006 15:49:54 +1000 Message-ID: <10039.1145425794@kao2.melbourne.sgi.com> Content-Transfer-Encoding: 8bit X-archive-position: 1181 X-ecartis-version: Ecartis v1.0.0 Sender: kdb-bounce@oss.sgi.com Errors-to: kdb-bounce@oss.sgi.com X-original-sender: kaos@sgi.com Precedence: bulk X-list: kdb One of the more useful features of kdb is its ability to print the arguments to functions, unlike the kernel which only prints an approximation of the function backtrace. The current kdb backtrace does not cope with i386 parameters in registers (i386 built with -mregparm=3), nor does it cope with x86_64 where the ABI mandates that the first 6 parameters are passed in registers. I will be rewriting the backtrace code for i386/x86_64 to cope with parameters in registers. kdb cannot rely on unwind data in the .eh_frame section. i386 has only just added that data (2.6.17-rc1), not all kernels are built with CONFIG_UNWIND_INFO but worst of all, CONFIG_UNWIND_INFO=y does not actually track where parameters are stored. gcc can copy arguments between registers, it can store arguments in local stack space and it can restore them to other registers later. gcc can even reuse argument registers when the argument is no longer required, printing such registers as arguments can lead to some very ambiguous backtraces and great user confusion. To track the parameters, you have to compile the entire kernel with -g, and load that data for kdb to use. Compiling with -g increases the kernel size by a factor of 5. Without -g, a reasonably small i386 vmlinux is 6760001, with -g it is 33209257, too much wasted space for my liking. Not to mention that there is no useful debugging data for assembler routines such as interrupt handlers, nor for the out of line spinlock contention paths. The current kdb backtrace routines already do partial code analysis at run time to work out stack sizes, to validate the guesses at where the return address is stored on stack and to handle all the special case hand crafted assembler code that we have in the kernel. The rewrite will go one stage further and do full code analysis, including tracking parameters as gcc moves them around. For each function in the backtrace, kdb will break it down into basic blocks and calculate the state on entry to and exit from each basic block. kdb will then use that state to locate the arguments, or to report that they have been overwritten. The state is also used to track the size of the stack, as a nice side effect this guarantees that we find the correct return address on stack and do not get fooled by stale kernel pointers on stack. A basic block (BB) starts at the beginning of the function or at any instruction that is the target of a branch instruction within the current function. A call to another function does not affect the current BB, except to invalidate some registers. A branch to an instruction outside the current function is ignored, these occur for tail recursion and for the out of line spinlock contention paths, neither of which affect the state of the current BB. A BB ends with an unconditional branch, a RET, LEAVE or UD2 instruction, or by dropping through to the next BB (the next line is the target of a branch instruction). A BB can also exit sideways via a conditional branch. A state record contains data such as :- The offset of the current stack pointer from the original value on entry. Where the original stack pointer was saved. This is preferred to the adjusted stack pointer, but only if the kernel is compiled with frame pointers and we have actually executed the instruction that saves the original stack pointer. For each argument, where it is currently held or a note that it has been overwritten. An argument can exist in multiple locations at the same time, the status of all of these locations must be tracked because kdb does not know which will be used later and which will be overwritten. Phase 1 runs the current function to identify the basic blocks and the branch instructions that link them together. This forms a directed graph, which may or may not have cycles. For each BB, count the number of ways that it can be entered and reserve enough space to point to the state records for each of those entries, plus one more. Phase 2 runs the directed graph, doing backtracking as cycles are detected. kdb starts off by assuming that i386 routines are compiled with -mregparm=3, and that x86_64 routines follow the ABI and pass 6 parameters in registers. Each BB is disassembled and the effect of each instruction is reflected in the state record, but only if the instruction changes any of the state that kdb cares about. At each exit point from the BB, the current state record is saved to form the input to the next BB in the graph. Phase 3 reconciles the input state when a BB has more than one entry point. One path to a BB may overwrite an argument, another path to the same BB may leave the argument alone. Reconciliation takes the lowest state for that argument and says that the argument is not valid on entry to the BB. Compiler theory 101, if two paths to a BB have an argument in an inconsistent state then the BB cannot be using the argument. It should be possible to fold the state reconciliation into the phase 2 processing, but treat it as a separate phase for clarity. Phase 4 calculates the number of arguments and works out how they are passed to this function. kdb does not know how many parameters are passed, it does not even know whether arguments to this function are in registers or on stack, some functions always get parameters on stack even when the kernel is compiled with -mregparm=3. If any BB reads from an argument register without first writing to that register then it must contain an argument, and all preceding argument registers are also in use. If a BB reads from the stack above the location containing the return address then it is reading an argument that was passed on stack, all locations between that address and the return address also contain arguments. This algorithm is not perfect, function(a, b, c, d) compiled with -mregparm=3 and where only d is used, will appear to have one argument that was passed on stack. Fortunately we do not write code that bad. Finally identify the starting state of the BB containing the current instruction pointer and calculate the argument state at the instruction pointer. Print the function and all arguments that kdb is confident about. Extract the return pointer from the stack. With full code analysis kdb knows exactly where this is stored, which removes any ambiguity caused by stale kernel pointers that are still on stack. This return address becomes the starting point for decoding the next function up the call tree, back to phase 1. Even with this algorithm, kdb will still need special case code for some of the assembler routines and for out of line code. No big deal, it is a Kernel DeBugger after all :). --------------------------- Use http://oss.sgi.com/ecartis to modify your settings or to unsubscribe.