PCP and flamegraphs/Heatmaps

[Amer Ather <aather@xxxxxxxxxxx>]

Hi Frank and Ken,

Thanks for all your help with pcp. With your help and support, we now have PCP 3.9.10-1Ârunning on both Ubuntu Precise and Trusty. Our web tool that consumes basic PCP metrics is in beta and our plan is to start testing on cloud instances soon.Â

Next phase of our project is to extend PCP to support Brendan's flamegraph. flamegraphs is a great tool for visualizing profiling data containing application and kernel stack traces. It helps with isolating hot functions and stack traces that may be contributing to performance regression.Â

flamegraph is easy to setup and use:

$ git cloneÂÂhttps://github.com/brendangregg/FlameGraph.git

# perf record -a -g -F 97 sleep 60ÂÂ
# perf scrpt > out.stackÂ
# stackcollapse-perf.pl < out.stacks > out.folded
# flamegraph.pl <out.folded >out.svg

Where out.svg contains interactive flamegraph of all stack samples captured using "perf" tool. In Flame Graphs each box represents a function: y-axis shows the stack depth and x-axis show the sample count. Box width is proportional to the the total time a function or its children (called functions) were on the CPU.Â

Our plan is to integrate flamegraph with PCP. This allows our web tool to trigger profiling on-demand basis on the cloud instance. PCP agent (python) will then collect 15-20 seconds worth of perf data, process it and save svg file in S3 bucket. PCP agent returns success/failure and path of S3 bucket to the web client. Web client polls the S3 bucket and displays the svg file containing flamegraph in browser.

Please let us know if there is a better way to design this feature in PCP.Â

--

Thanks,

Amer Ather

Cloud Performance Engineering

My Location