Re: PCP and flamegraphs/Heatmaps

[Brendan Gregg <bgregg@xxxxxxxxxxx>]

G'Day Frank,

On Tue, Sep 9, 2014 at 2:15 PM, Frank Ch. Eigler <fche@xxxxxxxxxx> wrote:

Hi, Amer -

> [...]Â Our plan is to integrate flamegraph with PCP. This allows our
> web tool to trigger profiling on-demand basis on the cloud
> instance. PCP agent (python) will then collect 15-20 seconds worth
> of perf data, process it and save svg file in S3 bucket. PCP agent
> returns success/failure and path of S3 bucket to the web
> client. [...]

It sounds a little bit like the hypothetical pcp agent (pmda?) is
being used only as sort of a remote-process-execution widget, not
feeding metric data in or out through pmcd to pmapi clients. Do I
have that right?

Yes. Just trigger a script. This is version 1, since we already have the script to trigger.

Â

Likely one can make that work, but one can probably come up with more
pcp-ish ways.

And that can be version 2. We'd want to do this so that we can make these visualizations dynamic: plotting data immediately, and refreshing, rather than waiting 30 seconds for a script to run.

For example, if it's sampled backtrace-based profiling that's being
sought here as the data source, a pmda that returns events or strings
for kernel / userspace backtraces, is a fine fit. (A pmapi client can
poll for these backtrace samples, and run it through the
flamegraph-building chain to make eye candy out of it, and plop it
somewhere the web server can see, like a pmwebd -R resource directory,
or some S3 volume.)

Sure, although I suspect it'll be easier to get heat maps to work this way, as it'd need to transfer an array of ~100 integers each second. It may be prohibitive to send aggregated stack traces every second (could be 1 Mbyte of data) - I guess we can find out.

Â

This new PMDA could be hard-coded (based on invoking perf commands,
consuming their output after a while, etc.).

Or perhaps could be a client of the general systemtap/json PMDA that
dsmith's starting to build. In this alternative, the only target-side
application-specific part would be a systemtap script/.ko? that
computes backtraces on demand. The new general PMDA would relay those
(and whatever other stats the stap script shares) as pcp event-or-atom
metric values. The rest of the process would be as above (pmapi
client gathering the backtraces, drawing svg's).

Right, just running a script and consuming the real time output in the browser for svg drawing should make a lot of things possible, like real time heat maps. And maybe real time-ish flame graphs.

Brendan

--

Brendan Gregg, Senior Performance Architect, Netflix