[Top] [All Lists]

Re: [pcp] suitability of PCP for event tracing

To: "Frank Ch. Eigler" <fche@xxxxxxxxxx>
Subject: Re: [pcp] suitability of PCP for event tracing
From: Mark Goodwin <mgoodwin@xxxxxxxxxx>
Date: Wed, 01 Sep 2010 16:25:44 +1000
Cc: pcp@xxxxxxxxxxx, systemtap@xxxxxxxxxxxxxxxxxx
In-reply-to: <20100831194941.GC5762@xxxxxxxxxx>
References: <20100827153906.GD3185@xxxxxxxxxx> <4C7A7DFE.2040606@xxxxxxxxxxxxxxxx> <20100831194941.GC5762@xxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20100720 Fedora/3.0.6-1.fc12 Thunderbird/3.0.6
On 09/01/2010 05:49 AM, Frank Ch. Eigler wrote:
Hi -

Thanks, Nathan, Ken, Greg, Mark, for clarifying the status quo and
some of the history.

We understand that the two problem domains are traditionally handled
with the event-tracing -vs- stats-monitoring distinction.  We're trying
to see where best to focus efforts to make some small steps to bridge
the two, where plenty of compromises are possible.  We'd prefer to
help build on an existing project with a nice community than to do new

yes certainly :)

For the poll-based data gathering issue, a couple of approaches came up:

(1) bypassing pmcd and generating an pmarchive file directly from
     trace data This appears to imply continuing the archive-vs-live
     dichotomy that makes it difficult for clients to process both
     recent and current data seamlessly together.

one of the issues with the live vrs archive dichotomy is that live
data is always available (since you're requesting it explicitly from
a PMDA that is otherwise passive), whereas the archive data is not
available unless configured to be collected before-hand (see pmlogger).
There is too much data to collect everything all the time - it's too
impracticable and intrusive, so some form of filtering and/or aggregation
needs to be done (see pmlogsummary, and Greg's old project too).

Since using such
     files would probably also need a custom client, then we'd not be
     using much of the pcp infrastructure, only as a passive data
     encoding layer.  This may not be worthwhile.

(2) protocol extensions for live-push on pmda and pmcd-client interfaces
     This clearly larger effort is only worth undertaking with the
     community's sympathy and assistance.  It might have some
     interesting integration possibilities with the other tools,
     espectially pmie (the inference engine).

yep - I suspect Ken and maybe Nathan would have further comments on this

For the static-pmns issue, the possibility of dynamic instance
domains, metric subspaces is probably sufficient, if the event
parameters are limited to only 1-2 degrees of freedom.  (In contrast,
imagine browsing a trace of NFS or kernel VFS operations; these have
~5 parameters.)

PCP instance domains are traditionally single dimensional, though there
are a few exceptions such as kernel.percpu.interrupts. It's easy enough
to split multi-dimensional data structures out into multiple metrics with
a common instance domain.

For the scalar-payloads issue, the BLOB/STRING metric types are indeed
available but are opaque to other tools, so don't compose well.  Would
you consider one additional data type, something like a JSON[1]
string?  It would be self-describing, with pmie and general processing
opportunities, though those numbers would lack the PMDA_PMUNITS

this could work using string or binary blob data types in the
existing protocols - though there is a size limit. And one of
the blessed features of PCP is the client monitoring tools can
more or less monitor any metrics - so any solution here would
also need specially crafted client tools. Extensions to the perl
binding would probably work best, e.g. interface with perl-JASON-*

For the filtering issue, pmStore() is an interesting possibility,
allowing the PMDAs to bear the brunt.  OTOH, if pmcd evolved into a
data-push-capable widget, it could serve as a filtering proxy,
requiring separate API or interpretation of the pmStore data.

well pmcd is already data-push capable using the pmstore interface,
allowing clients to store values for certain metrics in some of
the PMDAs. Filtering and parsing is done by the PMDA itself and
pmcd just acts as a proxy passthru (kind of a back-channel to
the pull interface).

pmstore hasn't really been used in anger like this though - more just
for setting config & control options and the like. The same (or similar)
protocol has also been used for a data source to open a socket directly
to a PMDA and tie into the PMDA's select loop, rather than going via pmcd.

For the web-based frontend issue, yeah, javascript+svg+etc. sounds
most promising, especially if it can be made to speak the native wire
protocol to pmdc.  This would seem to argue for a stateful
archive-serving pmdc, or perhaps a archive-serving proxy, as in Greg's
old project.

Time averaging, aggregation and filtering were all ambitious aims
of the project Greg's talking about - I wonder if that code could
ever be resurrected and open sourced? One abomination here was
that a PMDA could also be a client - and potentially query itself
for metrics(!)

Is this sounding reasonable?

it's going to take a lot more discussion, but enthusiasm seems to
be on our side :)

-- Mark Goodwin

<Prev in Thread] Current Thread [Next in Thread>