[Top] [All Lists]

kdb problems with SMP systems

To: kdb@xxxxxxxxxxx
Subject: kdb problems with SMP systems
From: Ethan Solomita <ethan@xxxxxxxxxxxxxxx>
Date: Wed, 10 Apr 2002 15:45:36 -0700
Sender: owner-kdb@xxxxxxxxxxx
        It seems that kdb has some serious problems in SMP environments.
I get regular lockups when I set a breakpoint at a common location and a
second CPU hits the breakpoint before the first CPU has gathered up all
the other CPUs. Have others tried kdb in SMP situations and had
problems? I'm running on sparc64, which I realize isn't the most common
platform for kdb.

        If you look at the main kdb() function, it's clear that it won't
work reliably for SMP. There are a number of issues. One involves
kdb_state. There is no synchronization performed until after you hit the
DEBUG line "kdb 4". The code leading up to this point will set various
flags in kdb_state, for example KDB and KDB_CONTROL are set, SUPPRESS is
cleared, REENTRY is either set or cleared, etc.

        So let's say we enter kdb(), change kdb_state, then lose the
race after "kdb 4" to become kdb_initial_cpu. We will then receive the
IPI from the CPU that won the race, we'll re-enter kdb() but with state
flags set which shouldn't be, we'll then go through all of kdb() which
will do things like clear the KDB and KDB_CONTROL flags before it exits,
and then we'll return where we left off and grab kdb_initial_cpu. Except
all these state flags which we thought we'd set (or cleared) have now
been changed. This has to be wrong.

        Another problem comes from saying "go" after hitting a
breakpoint. This requires some magic to clear the breakpoint we just
hit, single step the CPU so as to execute the actual instruction at the
breakpoint, quickly re-enter kdb, clear single stepping, and then
resume. The catch is, kdb is releasing all the other processors during
this. So the breakpoint (which the other CPUs should still hit) is
temporarily missing, and for those of us using CPUs without hardware
single-step, we have to create a temporary breakpoint at the next
instruction, which any other CPU could hit. So this needs to be changed.

        Also, the very top of kdb() calls kdba_{bp|db}_trap() after
hitting a breakpoint. But the different CPUs have not yet synchronized.
These functions change state flags, set bp_delay in the breakpoint
structure, etc. So if two CPUS hit a breakpoint concurrently, both CPUs
will believe that they've set bp_delay, one of them will win the race to
own kdb_initial_cpu, but then afterwards the other CPU will enter kdb
thinking that it has correctly set bp_delay, when it's possible that the
breakpoint doesn't even exist, or worse the first CPU is playing its
single-step-after-breakpoint game and then we hit on the breakpoint that
temporarily doesn't exist. Bad stuff.

        I've been working on fixes to this, but it's complicated stuff.
I've gotten some feedback from Keith, but I was hoping to let others who
know the code voice their opinions too. It seems to me that the
synchronization step (ie. grabbing kdb_initial_cpu) has to come much
earlier in kdb(), before any state has been set. This seems simplest --
don't allow contention. I noticed one issue, and Keith pointed out

        One is that you may get recursion in kdb(), and sometimes this
is legal. So if kdb_initial_cpu == smp_processor_id() we need to
evaluate whether this is a legal recursion. This also implies that the
RECURSE env var needs to be evaluated this far up in the function.

        The other issue is on i386 systems (and possibly others) where
the user-land breakpoint instruction is the same as kdb's kernel-space
breakpoint instruction. If the user is in gdb and has set a breakpoint
with "ignore 100000" or with some condition that occurs rarely, we can't
afford to waste a lot of kernel resources (ie. synchronizing all the
CPUs) before we realize that the breakpoint wasn't for us. But the same
function (kdba_bp_trap()) that checks to see if the breakpoint was ours,
also sets a lot of state that shouldn't be done without all the CPUs
happily synchronized.

        My thought on this was to create a second function ala
kdba_bp_check() which scans the breakpoint list without making
assumptions about your state, and without changing state. If this says
the breakpoint is real, then you synchronize processors and call
kdba_bp_trap(). This would change arch-specific code, so I wanted to get
feedback on it.

        If you've read this far, thanks!
        -- Ethan

<Prev in Thread] Current Thread [Next in Thread>