----- Original Message -----
> [...]
> A couple of hours of work, porting cockpit (cockpit-project.org) from
> its /proc-scraping C code to pcp PMAPI, have resulted in some code
> that I can't in good conscience recommend. This is mainly due to
> shortcomings in our code:
>
> - The API is really wordy for basic tasks such as extracting and
> scaling some metrics. It takes several setup steps, several arrays
> of data, several error-prone array/pointer arithmetic, with
> error-handling at each step. It adds up to way too much code - it's
*nod*
> >100 lines, and not done yet. We need a higher level API that
> combines the lookup/indom/fetch/extract/scale steps (doing internal
> caching/lookups as necessary), with the goal of making it easy to
> interface with an enclosing application.
Helper APIs would be good in general terms, bonus points for using
'em in existing tools (and extracting code used from tools wherever
possible/needed instead of starting from scratch).
Doing libpcp-internal caching is non-trivial. From a recent chat I
had with Ken though, he reckons alot of what we'd need is already in
place (the derived metrics code does some of this caching and auto-
metric modification for rate conversion, scaling, etc). If we can
indeed leverage that, it's certainly make me think more favourably
about tackling this within libpcp rather than a layer above.
Ken floated the idea of a new context type that would be able to do
caching (caching descriptors and other things is needed for all the
many types of live/archive/local contexts of course, and modifying
metrics on-the-fly could be part of a new context definition). And
that got me on to wondering about unified context mode again - but
that's probably just my pre-conceived ideas there and I may be mis-
representing his line of thinking. It'd be great to get your ideas
fleshed out a bit more there, Ken.
Having looked at the pmwebd code again recently, it's crying out for
help with caching - its not making ideal use of the PCP protocol at
the moment (doing descriptor lookups in a loop for every pmFetch &
that sort of thing). So, yep, I agree its worth tackling this now,
and am less inclined to want to layer above PMAPI now.
> - The linux-proc PMDA is not as efficient as it could be; see the bug
> filed last night. This is only a quality-of-impllementation matter,
> but as we trumpet light-weightness of the pmcd/agent system, we need
> to do better.
Sounds good.
> - Multithreaded support has had long-known bugs that preclude their
> heavy-duty use in a real application (pmwebd / graphite), even for
> just archives. It turns out that for light-duty use, there is not
> enough support either. For the cockpit application, we might like
> to use PM_CONTEXT_LOCAL, but that is precluded for multithreaded
> apps. PM_CONTEXT_HOST to local: is fine in theory, but due to the
> single-threadedness of pmcd, it snuffs out any possible advantage
Hmmm, there may be some leaps of logic there - I'd like to see more
measuring to verify the need for some of this (the "snuffs out any
advantage" bit, in particular, I could use more details/verification
there to be more convinced).
> anyway. We should at least permit multithreaded PM_CONTEXT_LOCAL,
> and do internal serialization, say on a per-PMDA-SO basis.
*nod*
> (PMCD could probably do the same.)
Hmm, maybe.
> - A small nit, it would be helpful if libpcp came with a pkgconfig
> file, so that non-system copies of libpcp could be found by
> applications.
*nod* - sounds good to me.
> Assuming no objections, I'll start assembling some more concrete
> proposals for these shortly.
Awesome, thanks.
cheers.
--
Nathan
|