Thanks for the feedback Nathan.
I'll try responding insitu, but that is going to get pretty messy if we
go round this loop once more, so at that point I think we'll need to
pull out the remaining contentious points and deal with them without the
larger context of the original mail.
On Thu, 2009-11-05 at 14:36 +1100, Nathan Scott wrote:
> Hi Ken,
>
> ----- "Ken McDonell" <kenj@xxxxxxxxxxxxxxxx> wrote:
> > ... And finally the stateful computation of
> > delta(value) / delta(other_value) can only be sensibly done at the
> > per-process level on the client side.
> >
> > So one could consider a derived metrics module to be an old SysV-style
> > streams module that has a private configuration file and is inserted
> > between a PMAPI client and the source of PCP metrics.
> >
> > I'd like some initial feedback on the following set of initial
> > limitations and assumptions.
> > 1. Only works for platforms with ELF binaries. I'm planning to use
> > $LD_PRELOAD to optionally insert a DSO between the PCP client
> > and libpcp.so to intercept calls and rewrite them as needed.
>
> Hmm. That seems quite unwieldy (obviously, wont work on Windows too).
> What's the rationale for that instead of working in libpcp directly?
The real rationale is that for GNU/Linux the compiler/linker folk never
really understood the deeper issues of ABI compatibility and certainly
have not come close to the elegance of the IRIX approach to evolving
DSOs with optional symbols.
For this reason I'm hesitant to change the libpcp API/ABI.
But I'm willing to be persuaded on this point.
There is certainly no architectural obstacle to putting the
functionality into libpcp rather than as wrappers to the libpcp routines
as I was proposing.
> > 2. Only guaranteed to work for the synchronous PMAPI variants.
> > So for example I'll make pmLookupName() work, but invest no
> > effort in the asynchronous pair pmRequestNames() and
> > pmReceiveNames().
> > As an aside does any know of a living user of these asynchronous
> > interface extensions to the original PMAPI?
>
> (Not that I know of... perhaps these can die. They are not implemented on
> Win32 & AFAICT cannot be, so I would rather noone started using them).
Perhaps we should start a separate thread on this ... I'm with you and
have a philosophical objection to trying to provide an async API to a
library infrastructure that was never designed to be thread-safe
(deliberately, not by omission).
> > 3. Configuration file pathname comes via a new PCP environment
> > variable, probably $PCP_DERIVED_CONFIG.
>
> This seems unwieldy too.
>
> Would it be feasible to go more for something that can be specified
> all-at-once, and as a (easily identified) drop-in replacement for a
> metric name (I assume these will be given "made-up" PMIDs by Lookup
> Name?)? ... so maybe:
>
> $ pmval 'expr: delta(kernel.all.pswitch) * 1000'
>
> Or some other not-valid-as-metric-name keyword in place of "expr:" like
> 'derive:' or 'formula:' or 'infer:'. Or perhaps force them to be brace
> enclosed, e.g.
>
> $ pmval '(delta(kernel.all.pswitch) * 1000 + 42)'
>
> Or perhaps require something with an assignment, like the optional pmie
> syntax - "cputime = delta(simple.time.user) + delta(simple.time.sys)".
My goal here was to NOT require any application changes ... hence the
streams model analogy. The examples you give suggest that the
applications might be modified to understand about derived metrics.
On reflection I think we may be able to do both.
As an early initialization (this bit is a little tricky, but I think
pmLoadNameSpace or pmNewContext have to be called before anything we
care about could be called), I'll check for $PCP_DERIVED_CONFIG being
defined in the environment and if it use, register (see below) the
derived metrics from that configuration file. As a prop for platforms
that don't have environment variables the routine
pmLoadDerivedConfig(char *path)
may be used to explicitly register derived metrics from a config
file ... so this could be used with command line glue to load one (or
indeed more) configuration files as and when an application chooses.
To register a derived metric, the routine
pmRegisterDerived(char *name, char *expr)
is used. The name needs to come from the expression definer, because
this (a) acts as the handle for subsequent PMAPI calls, and
pmLookupName() in particular, and (b) defines where the derived metric
should be spliced into the PMNS.
By exposing this routine, ...
1. I am committing to the "change libpcp" path, so the earlier comments
about ELF and $LD_PRELOAD are not longer relevant
2. pmval could be modified to parse
pmval "cputime = simple.time.user + simple.time.sys" ...
(the delta() is not needed here, but would be needed in the
delta(v1) / delta(v2) class of expressions) and then pmval could call
pmRegisterDerived("cputime", "simple.time.user + simple.time.sys")
3. pmchart can go crazy with the definition of derived metrics via the
GUI (and possibly saving them for later reloading or auto reloading)
> I have plans to revisit the pmchart metric selection process for gadgets
> down the track, and it would be good to allow the metric selector dialog
> to allow derived metric syntax too (even just if its as free form text),
> whatever that syntax turns out to be... if it can be done "inline" like
> this, without needing environment variables, temporary files, etc - that
> would be great.
>
> > 4. No recursive definitions. Each derived metric is an
> > expression involving metrics that are NOT derived.
>
> That seems fair enough.
>
> cheers.
Is that closer to meeting your needs?
|