Frank replied ...
> > Please, if there is any choice, let's not add binary->ascii and
> > ascii->binary format conversions on the code path between the producer
> > and consumer of high volume and/or low latency performance data. I
> > hope the reasons are self-evident to this audience. [...]
>
> (Sure, generally speaking; but for lesser intensity, it could be fine.
> Must measure.)
Indeed it depends on the data rate ... I think of systemtap (and kernel
instrumentation in general) to be at the heavy lifting end of the spectrum
in terms of data rates and size, and expect that event trace data might be
even more so. But perhaps my "expectations" don't match the norm.
>
> > The mmv model has worked well in the past. [...]
>
> It would be nice if we used it ourselves (outside QA toys), so we could
> experience its robustness & utility right in the code we ship.
As Nathan has said ... "The MMV code in PCP has been around for quite some
time, and I know of a number of production environments using it 24x7 on
hundreds of machines, for many years - its not fragile at all. "
Because of the way the mmv pmda is written, it is likely that the most
common use case will be for instrumentation of applications that do not
already have a performance data export channel (IPC, log file, status file,
callback routine, system call) and are looking for an easy to implement and
low overhead option. So by definition, most of the uses will be outside the
PCP project's code base. And the cases we know about are in heavy duty
production environments. So it is not exclusively the plaything of QA toys,
and we don't use it more within the PCP code because the PMDAs we ship are
generally ones where the export API exists (in someone else's code) and
we're using their API.
The systemtap area looks to be one place where such an API does not already
exist and we're looking for options.
> ...
> No doubt. And yet, there are some problems visible to the skeptical eye.
>
> For example, the use of the g1/g2 generation numbers to delimit "in-flux"
> times -- it might not be quite enough to prevent TOCTOU errors. If an app
> mmv_stats_init's (maybe growing new instances or
> whatever) right after the pmda mmv starts a need_reload==1 processing
> cycle, some corruption will result.
>
> Similarly, an app that scrambles its mmv shared memory content due to
error
> or malevolence could likely also harm the shared systemwide pmda. The
> pmda seems to trust the data and performs little to no pointer arithmetic
> checking on the offsets coming therein. (Plus the mmv pmda could be used
> as a dso right inside pmcd, not even isolated into a process.)
Sceptical eyes are always welcome ... I'd be happy to take these issues on
board to inform a plan to improve the mmv pmda. Since these are your
observations, could you open a bz issue so we can track 'em?
The DSO pmda issue is however not really an mmv problem, it is a general
caveat on any PMDA that is added as a DSO, namely pmcd is exposed to coding
errors in the PMDA. For this reason we really don't recommend the DSO pmda
option except in cases where the pmcd<->pmda latency is an issue and we want
to minimize the code path and we have confidence in the quality of the pmda
code (e.g we know who wrote it, or it has been field tested in the confines
of a daemon PMDA before the config switch is made).
|