Sorry, I got disconnected from lkcd and couldn't connect back in ...
>The dump configuration utility is checked in.
>It's in lkcdutils/lkcd_config.
>All the appropriate scripts/spec files have changed to use it.
>There's even a manual page, if you can believe that.
OK. Have pulled it over. Subodh is trying it out ...
>>Before moving to SMP, we decided to first merge in our changes to enable
>>system continuation after a dump, by making the other CPUs spin for the
>>duration of the dump and then release them, rather than making them stop.
>>(We are now using dprobes to trigger the dump from a probe point to test
>>our changes.)
>
>How's this working? I'd like to get this into the tree if at all possible
>so we can get rid of the current "stop" method and get rid of the SMP
bugs.
Hmm.. I guess what you are saying here is that if we have something that
works correctly for our case, we could simply use that for the panic dump
situation as well, and fall into the actual stop cpu + restart code after
the dump is through. Is that correct ?
As I mentioned, we aren't quite there yet in terms of perfect behaviour on
SMP in all cases. We have it a little easier for non-disruptive dumps in
some respect, because we can assume that the system is operational, yet
there still are a few things to think about. The changes that we've made
and tried out so far, simply involved issuing smp_call_function with
dump_spin() rather than with stop_this_cpu() in for the non-disruptive
case, where dump_spin() just spins in a busy loop as long as a flag is
enabled (that is, for the duration of the dump), and turning of this flag
after the dump is through. Well, we also make sure that NMI watchdog
doesn't complain about this spin, on those CPUs. I'm not sure if we need to
modify the cpu_online_map as yet, if we are stopping all scheduling - have
to check that out.
That's probably not enough.
When we tried this earlier (with your older patch) things seemed to work
(irrespective of which CPU triggered the dump).
But then, we just may not have hit some conditions.
It seems like we should change the irq affinities to make sure disk
interrupts don't go to the spinning CPUs. (i.e of course if we keep
interrupts disabled on those CPUs while they spin; Need to think of what
would happen if we keep interrupts enabled ) [If we do change irq
affinities, then for non-disruptive dump we would have to save the old
values and restore them after we are through; we can't assume which irq
this would be, so perhaps we could do this for all]. If the
smp_call_function_interrupt (or its equivalent) didn't ack the irq right
away, there is a possibility that the arbitration priority for apic of the
spinning cpus would be higher than the dumping cpu, so interrupts would
always go to the dumping cpu, but that's kind of a fragile approach, I
guess, and may lead to other problems. In any case the solution needs to be
more generic than that.
In the panic/forced dump case, could there be problems with
smp_call_function waiting for the other CPUs to enter the IPI interrupt
handler ? Using an NMI interrupt like kdb does was an option we'd
explored earlier, but this could abruptly interrupt a CPU while it holds
some locks or state which could be required by the dump i/o path, so
spinning inside the NMI interrupt doesn't seem suitable without any added
caution.
A part of getting this right for all cases, is ensuring that dump can be
triggered from any context, i.e. whatever each of the CPUs may be doing at
that instant, and that's an aspect that we are looking into next. How much
of that solution would be acceptable/usable in a panic type dump may be a
question, but for the moment, we could at least get some more cases
working before we have a complete solution there (with your block device
dump interface plus mcore).
>I believe so. I've just started communicating with Mike Keefe. He's
>sent me a patch (among other things), and I'm in the process of review
>and seeing how we can integrate it, and then mcore.
Good ! Hope we can start discussing this more soon.
>I've checked in almost everything you can imagine now:
>
> - all 2.4 code is checked in, all header mods done
Did you change anything in the 2.4 tree that isn't in the patch you'd sent
me earlier ?
Suparna Bhattacharya
IBM Software Lab, India
E-mail : bsuparna@xxxxxxxxxx
Phone : 91-80-5267117, Extn : 3961
|