This note covers some preliminary thoughts on ways to reliably get dumps
generated from a system that has crashed/is damaged / hung, so that a
reboot is necessary. We'd really like to get to work out a nice way for
doing this on Linux that can handle various kinds of environments
(architecture, firmware and device h/w) neatly and with minimal work, and
any ideas on that are welcome.
Standalone dumps:
-----------------------------
In the ideal world this would mean being able to take an *accurate*
snapshot dump of the system state without relying on the functioning of any
portion of the OS (*zero-system dependence*), and without executing
anything that could potentially worsen the system damage.
In some ways this might actually be a more critical requirement today, as
it comes into play for panic style dumps, where the system is damaged,
where a dump may be the only way to figure out what went wrong. Here the
system would be brought down (any further execution has a potential for
causing corruption of persistent data, which is unacceptable), so it is OK
to take drastic reset type actions that could result in loss of data.
>From a requirements perspective we need to characterise the extent of
damage that we must be able to tolerate and what information/data must be
dumped.
We listed some of the possibilities that we were aware of :
1. Having a special raw driver dump interface registered by dump devices
Issue: Range of devices / adapters to be supported on Linux
2. Make use of firmware/bios capabilities to trigger the dump (should be
possible for configuations that can be booted from as long as the
capabilities provided for boot extend to support for write operations)
Issue: Range of architectures to be supported and the need to check on
the kind of firmware support we need there
Issue: Potential loopholes or special cases to be handled on a case to
case basis ( there have been some tricky problems along this route
encountered with OS/2 standalone dump; e.g. some special case problems with
reseting adapters which disabled their bios support had to be addressed and
a few troublesome issues for certain h/w)
3. Reset (without clearing memory) to effect a soft boot of a reliable
linux kernel (built for this purpose with the necessary drivers for the
dump device), which performs the actual dumping to disk (before the reset,
save /mark the memory areas that could be overwritten during the process)
(Mission-critical mcore approach)
Issue: On systems where firmware support for a reset without clearing
memory is not available, as on many x86 systems, the reset, including reset
of the devices etc, has to be affected in software (e.g mcore has had
occasional problems with video resets; from what I've heard from some
people who've tried it, the bootimg capability which it relies on may not
have matured sufficiently to be considered as a reliable option for
production systems)
Issue: Can't capture entire physical memory state in the dump because
of the memory that gets used by the dump kernel (preloaded code plus
dynamic space during the softboot) (mcore reuses some of the user memory
pages to save the dump pages to make this space available)
It may be worthwhile to find out if BIOS extension standards could be
worked out to help with this (I hear of some attempts in this area),
because that would be the ideal way to solve this problem. Perhaps an
extension to soft reset without clearing memory coupled with a way to save
register/system state in some place, or in the longer run a way to dump
memory to disk (sort of, but not quite the way it happens with hibernation
on some systems). For example (2) seems to the way Linux 390 achieves
standalone dumps today, because of the h/w / firmware support existing in
that environment, so as general direction such a standard interface from
memory dump to disk would be nice to have. The390 capability of saving
register status is a useful h/w mechanism in the same context.
Having a raw device driver interface has its merits too as it is sort of
intermediate between (2) and (3), but there is the issue of getting all the
device drivers (which isn't small on Linux) to support this interface.
Should we try to get inputs from hard disk driver maintainers about the
kind of effort this entails for each driver and the acceptability of such
an approach ?
Maybe, in a hybrid solution, one could think of using (1) if the dump
device has registered a raw dumping interface, and if it doesn't then
moving to (3), i.e. the dormant kernel approach, maybe if even that fails
(e.g if there are memory limitations or some other restrictions; Ques: How
would we know that ?) then take up (2) as a last ditch effort only for
limited h/w set ? Or should we try (2) first and then (3) ?
Obviously, none of the options seem to be perfect.
So, enough of rambling :-)
As you can see, the floor is open for discussion !
Regards
Suparna
Suparna Bhattacharya
IBM Software Lab, India
E-mail : bsuparna@xxxxxxxxxx
Phone : 91-80-5267117, Extn : 2525
|