Re: [pcp] high frequency metrics collection and trace

[Jun Wang <junwang123@xxxxxxxxx>]

Hi Ken,

Thanks a lot for the information. Any information on the event record service?
Is it related to the "Event Logging" on the following link at all?
http://oss.sgi.com/projects/pcp/pcp-gui.git/man/html/howto.systemlog.html

Thanks,
Jun

On Fri, Dec 14, 2012 at 2:10 PM, Ken McDonell <kenj@xxxxxxxxxxxxxxxx> wrote:

On Fri, 2012-12-14 at 01:53 -0800, Jun Wang wrote:
> Folks,
>
> In terms of the add-hoc high frequency
> stats/metrics collection, do we have a known upper limit on the
> frequency to collect and display the metrics? Can we sample every 100
> milliseconds or with even higher frequency?

This is a "length of the piece of string" question ... not all metrics
have equal cost to instantiate (collect), not all fetches return the
same number of metrics and values and not all CPUs are the same speed.
Sampling in the 100's of msec range is certainly feasible for many
combinations.

> What about tracing with trace PMDA? How fast can an instrumented
> application push data through libpcp_trace? Is the PDU buffer size on
> the trace PMDA side configurable?

The trace PMDA was never intended for high frequency sampling
(annotation of bash scripts is a common use), so I would not consider
this for high throughput.  More interesting would be the relatively
recent PMAPI extensions for event records ... Nathan can probably expand
on this better than I.

> Does the PMAPI have to use fetch/pull to collect data with certain
> fixed frequency? I assume that PMCD cannot push data to the monitoring
> side via PMAPI.

There is very little client state maintained at pmcd, so clients are
free to fetch at whatever frequency they desire and indeed pmlogger and
pmie collect different groups of metrics at different frequencies.

Pushing from pmcd is nver going to happen (this is not an SNMP trap
model!).

> However, for the trace PMDA case, does it make sense to collect
> multiple samples in the buffer, each with a different timestamp, and
> have the monitoring tools on the other side, such as pmchart, to
> display multiple samples, per every fetch/pull, based on the
> timestamp? Just a thought on one option to increase the sample
> frequency without increasing the fetch/pull frequency.

The event record services address just this issue.  Individual event
records are timestamped, even though a group of them are delivered in a
batch to the PMAPI client.