Nathan,
Thanks for the great and detailed explanation on the multi-host support.
I assume that pmchart can graph individual metrics, collected from multiple
hosts separately, over time series, though it isn't designed to graph the
average/sum/etc of those metrics, collected on a multiple host, over time
series. Is there any other way, or workaround, that can be used for PCP to
graph the average/sum/etc of a metrics across multiple, in 100s for example, of
hosts?
Can summary PMDA be used for that?
I noticed a summary PMDA process, together with a pmie daemon, that can do
expression of multiple metrics.
At first, for a single collector host, this means the following steps, all on
the collector host except for the last step, based on my understanding.
* metrics foo.X is collected by PMDA foo;
* metrics bar.Y is collected by PMDA bar;
* the pmie daemon talks to PMCD and get metrics foo.X and metrics bar.Y via
the corresponding PMDAs, then use both of them to calculate metrics summay.Z.
* the summary PMDA talks to pmie daemon periodically and fetch summary.Z;
* the monitoring tools retrieve summary.Z from PMCD, and summary PMDA, via
PMAPI and graph it.
Does PMCD cache the metrics foo.X and bar.Y? or does PMCD run completely
stateless without caching any metrics? I think that PMCD shouldn't cache any
metrics. That, however, means that time-wise the summary metrics is always
running a step, or a sampling period, behind the foo.X and bar.Y as it uses the
values of X and Y at the last time when the pmie talks to PMCD.
Secondly, can summary PMDA be used to do sum/avg across multiple hosts? Say we
run a pmie process on a server and have it talk to PMCDs on multiple hosts to
collect h1:foo.X, h2:foo.X, h3:foo.X and ... and calculate the value of the
expression. Then we also run summary PMDA and a PMCD on this server and have
the summary PMDA talks to the pmie process to collect the the of summary.FOO-X
metrics. Now we can have any PCP monitor tools to, either remotely or locally,
graph or analyze the sum/avg/etc of foo.X across multiple hosts.
Does this make sense?
Thanks,
Jun
On Dec 6, 2012, at 6:27 PM, Nathan Scott <nathans@xxxxxxxxxx> wrote:
> Hi Jun,
>
> ----- Original Message -----
>>
>> Hi PCP Folks,
>>
>> Being a newbie on PCP, I have questions on monitoring metrics from
>> multiple collector hosts. On the monitoring side, with tools like
>> pmchart, pmie and pmstat, what's the strategy to monitor metrics
>> from multiple collector nodes? To my understanding, each pmcd,
>> together with the corresponding pmdas, runs and manages one
>> collector node.
>
> That's correct.
>
>> I'm reading PCP on Linux user's guide on
>> http://www.damtp.cam.ac.uk/cosmos/private/documentation/SGI/propack/PCP_LinuxUAG/pdf/PCP_LinuxUAG.pdf
>> .
>>
>> pmstat has an option -H which allows users to specify multiple hosts.
>> I verifies that it does print out stats from multiple hosts, one
>> after the other when hosts are specified with -H.
>
> *nod*. You can also use multiple -h options on the command line to
> pmstat, which results in an extra initial column showing the node:
>
> $ pmstat -h smash -h verge -t 1
> @ Fri Dec 7 13:13:04 2012
> node loadavg memory swap io system cpu
> 1 min swpd buff cache pi po bi bo in cs us sy id
> smash 1.45 0 4015m 8746m 0 0 0 0 493 1246 0 1 99
> verge 0.30 0 404512 2213m 0 0 0 0 291 358 1 1 98
> smash 1.45 0 4025m 8746m 0 0 0 0 575 1471 1 0 99
> verge 0.30 0 404648 2213m 0 0 0 0 264 296 2 1 97
>
>> pmie has an -h option which allow users to specify only one host to
>> monitor.
>
> Strictly speaking, that specifies the default host. Others can
> be specified directly, in the pmie language, for individual rules
> or parts of rules. For rules with no host explicitly specified,
> the default is used. If no default is specified, localhost will
> be used.
>
>> However, it appears that hosts can also be specified in the
>> pmie specification language to create per-host and cross-host rules.
>
> That's right.
>
>> Does this implicitly trigger the communication between pmie and the
>> pmcd(s) on the corresponding hosts that are specified in the pmie
>> spec language?
>
> Yes.
>
>> On the other hand, there appears to be discussions on
>> starting one pmie, on the monitor side, for each collector host.
>> What's the recommended approach?
>
> Whatever works best for you. I've seen production deployments where
> either, and both strategies have been used.
>
>> pmchart also has an -h option allowing users to specify only one host
>
> Like with pmie before, that's for the default host(s). And like with
> pmstat, you can give multiple -h options.
>
>> to monitor. However, there are discussions on hosts in the pmchart
>> man page and it is claimed that pmchart can display metrics from
>> multiple hosts. How do I specify those hosts? Is it available on the
>> menu of the GUI? What's the relation between those hosts and the
>> ONLY host specified with -h option?
>
> Using this as a guide:
> http://oss.sgi.com/projects/pcp/pcp-gui.git/man/html/lab.pmchart.html
> (from the PCP Manual link on the bottm left of the PCP project page)
>
> On the third screenshot (Open View) there's a "Host:" drop down, and
> a new host button between Open and Cancel. The drop down menu will
> contain all hosts seen so far (so, all -h options, and any hostnames
> entered via the UI). The new host button pops up another window that
> will allow you to enter additional host names.
>
> FWIW, when running in "archive mode" (historical data), the UI is all
> the same, except instead of "Hosts:" the user can enter additional PCP
> archives from which historical data is sourced. The archives have a
> one-to-one mapping with hosts (i.e. an archive records data from one
> host and one host only).
>
> If you scroll further down the above URL to the sixth screenshot, you
> will see another Host button on the New Chart window - that has the
> same effect as the new host button on the Open View window earlier.
>
>> Also, if I define a new metrics domain on a collector, a new PMDA and
>> new configuration line(s) in pmcd to enable it, how do I make it
>> available to be selected from pmchart menu? What about multiple
>
> As soon as the PMDA is installed (via its Install script, usually),
> its metrics become immediately available in the client tools like
> pmchart, with no other intervention required (client tools do auto-
> discovery via the metric names and their associated metadata).
>
>> metrics domains on multiple collector hosts? Can they be displayed
>> with one pmchart application but multiple views on the monitor side?
>
> Yes. It's even possible in pmchart to have a single chart that has
> individual plots from different hosts.
>
> Good questions! :) The PCP man pages are extensive too - so on any
> host that you've installed pcp and pcp-gui on, you will also be able
> to browse through:
>
> $ man pmstat
> $ man pmie
> $ man pmchart
> $ man pmcd
>
> Several of the command line option (e.g. time window specification, for
> archives) are also documented on the PCPIntro man page, might be of use
> too. Enjoy!
>
> cheers.
>
> --
> Nathan
|