Some comments in-line below - I think this is evolving in all the
right directions, if I'm any judge :)
On 01/08/2014 12:39 PM, Frank Ch. Eigler wrote:
Hi -
Here are some notes related our earlier unified-context ideas [1][2].
As a recap, it's desirable to teach pcp clients to locate pcp data for
arbitrary hosts, and then to easily seek between live & historical
metrics for it.
[1] http://oss.sgi.com/pipermail/pcp/2013-September/003963.html
[2] http://oss.sgi.com/pipermail/pcp/2013-November/004090.html
My proposed approach focuses on archive files and a new server for the
archive files. It might make do without a new PM_CONTEXT_* mode, and
at the PMAPI level just generalize PM_CONTEXT_ARCHIVE & _HOST a little.
1) We'd extend archive files to be usable as a source of "live" data,
so that clients can sort-of-"tail -f" the files to get current info,
or can seek along time with pmSetMode().
1.1) pmlogger needs to learn to write its output with what IIRC kenj
has referred to as "semantic units", ie., proper use sequencing of
write(2), fdatasync(), to put interdependent data on disk correctly.
the old pmchart used to be able to replay from the tail of a growing
archive - I think pmlogger was modified to update and sync() the archive
label with new time bounds whenever new data was available (and had been
sync'd). By polling the archive label the tool never tried to read
partial records and if it tried to get ahead of real-time it would
just stop.
1.2) libpcp needs to learn to read archives that are being written-to.
It should not freak out when the end-of-file is reached, and for
PM_MODE_LIVE, just return the then-freshest measurements and/or
trigger PM_ERR_PMDANOTREADY if clients are fetching faster than the
logger is recording. This could be done without a PM_CONTEXT_UNIFIED
extension, just permitting PM_MODE_LIVE for PM_CONTEXT_ARCHIVE, and
giving clients an option (like the -f for tail) to use that flag.
yes agree CONTEXT_UNIFIED isn't needed if CONTEXT_ARCHIVE and CONTEXT_HOST
can be made synonymous.
2) Because these archive files may not be local to the clients, or
because they may not already contain every metric a client might like,
we need a network server to offer them such tasty treats. Elsewhere
and elsewhen, nathans has ably argued that this shouldn't be pmcd, but
a new server or an extended pmlogger/pmproxy/pmmgr.
2.1) A new server needs to be written, which would monitor some local
archive files, and serve an extended pcp wire protocol for it (one
that includes archive-like pmSetMode operations). It would advertise
the pcp hostname (or pmmgr-style hostid) to the network, so clients
can find the right host data (probably one tcp port per archive, or
else multiplexed over a single tcp port and identifying the host
during startup negotiation). The clients could keep using
PM_CONTEXT_HOST but permit PM_MODE_BACK etc. to forward -S/-T times -
ie. no requirement for a new PM_CONTEXT_UNIFIED.
Or alternatively, PM_CONTEXT_ARCHIVE that supports live mode with
-S 'now' or whatever.. This might be less work since archive contexts
already support all of the PM_MODE_* options, etc. I guess we could
end up with -a and -h options being synonyms.
2.2) Clients would be extended with enough discovery logic to find the
network server that has data for the interested pcp hostname /
pmmgr-hostid. Or, just rely on an extended pmfind:
"pmstat -h `pmfind --hostid HOSTID`" would attach to a server that
has data for that HOSTID. (A bonus complication is having multiple
archives/servers for the same HOSTID, such as with different subsets
of metrics, or for different times: perhaps extra pmfind filtering
params.)
OK, so "pmsometool -a hostname" should invoke the mentioned discovery
mechanism to find the relevant archive(s) containing the metrics for
the requested host and timestamps, without the user or client having to
do anything special. That data may be spread across multiple archives
(for the same host but for different timestamps and/or metrics), but
this can and should be completely transparent. It's a configuration
task to nominate the set of directories and/or ports to search, perhaps
aided by a $PCP_ARCHIVE_SEARCH_PATH envar or something.
For live monitoring, it's up to the PMAPI infrastructure whether it gets
the data from the tail of one or more archives or from pmcd on the named
host, or the proposed new server .., or a combination depending on
what's being logged and where - the client tool shouldn't care.
For back-compat, an explicit archive path/file can still be named and
-h hostname is just synonymous with "-a hostname -S live" (or whatever
designates 'now').
2.3) That server needs to be extended: merged into pmlogger, or
interfaced with pmlc, so as to arrange logging of newly requested live
data that wasn't already set up in the pre-configured set of metrics.
It could heuristically control their logging interval and duration for
multiple clients.
This clearly doesn't work for the case where a client decides to rewind
and replay metrics from befoe the time they started being logged when
they were first requested. ENOTIMEMACHINE :) But it should be ok for
the common use cases and would be quite innovative.
Cheers
-- Mark
|