Paste events
============
Goal: Capture arbitrary information from an active performance
investigation. Allow other performance tools, or configuration
state from the system under observation to be stored inside the
daily PCP archives (or personal archives) for later reference and
subsequent playback or deeper analysis.
Examples:
During a production application outage / crisis, sysadmins would
have a mechanism whereby ps(1), top(1), jstat(1) or other reports
could be stored into the regular PCP recording system for post-
mortem analysis. To support this it needs to be *extremely easy*
and obvious as to how to store such information, require no change
that might result in a pmcd(1) or pmlogger(1) restart, and so on.
It also needs to be simple to retrieve that information later, and
would be helpful to associate accurate timestamps with it. Being
able to search on the stored reports would be mighty helpful also.
When benchmarking or load testing, it is often helpful to store
system configuration information at the start of the test - things
like operating system version, application versions, list of all
the installed packages and so on. We should make it easy to do
this (both adding new information and subsequently retrieving it).
Approach:
So a system with the level of simplicity of pastebin.com would be
ideal (except of course, the data should not leave the system and
be stored alongside other PCP data). In PCP terms, the following
components would seem to solve most of these needs:
- A new pmdapaste(1) with the following metrics:
paste.bin (a storable event metric, type aggregate)
paste.text (a storable event metric, type string)
paste.params.uid (event parameter: user ID of the storer)
paste.params.gid (event parameter: group ID of the storer)
paste.params.size (event parameter: size of this event)
paste.params.name (event parameter: optional user-supplied ID)
paste.params.format (event parameter: format of this event)
paste.total.count (cumulative counter of all events stored)
paste.total.size (cumulative counter of all event sizes)
So, when one wishes to add freeform text/data to a PCP archive,
the idea would be to store into either paste.bin or paste.text.
We would cleverly have in place a pmlogconf template to ensure
the local primary logger is recording any events that arrive for
these metrics, with no manual configuration required.
Only pastes from authenticated users would be permitted, but
thanks to the wonders of AF_UNIX and SO_PEERCRED this wouldn't
unduly restrict the make-it-easy-by-default goal.
- A new command line utility pmpaste(1), with possible usage below,
that handles both sides of the coin - inserting and extracting:
Usage: pmpaste [options] [text]
Options:
-a archive retrieve pastes from the named PCP log archive
-b format paste contains binary data in specified format
-d delay, pause between updates for archive replay
-f file paste contents of specified file ("-f-" => stdin)
-g start in GUI mode with new time control
-h host send paste to PMCD on host
-N name an optional user-supplied paste identifier (string)
-O offset initial offset into the time window for extract
-p port port number for connection to existing time control
-S starttime start of the time window for extract
-s samples terminate extraction after this many samples
-T endtime end of the time window for extract
-t interval sample interval for extract [default 1 second]
-Z timezone set reporting timezone for extract
-z set reporting timezone to local time of metrics source
It might be possible to make this command a shell script which wraps
pmstore(1) and pmevent(1) - either way thats basically what it would
be doing (i.e. pmstore -> insert, pmevent -> extract). Or, maybe it
is a bit confusing rolling both modes into one command? Seems OK to
me, for now anyway.
Notes:
- Cannot allow arbitrary size store actions, for pmcd DoS reasons.
Initially, we should see whether the default pmcd in-coming PDU
size limit suffices as a guard. The binary metric, allowing the
compression of large reports, may help to alleviate this. If it
later turns out this does not suffice in practice, a pagination
mechanism would be one angle of attack, perhaps using instance
identifiers for page numbers.
- Allow only authenticated connections to perform stores, recording
the credentials of the storer with each event as described above.
- pmstore(1) should acquire an option allowing data to be streamed
to it, and a read-data-from-file option. Restriction preventing
aggregate data being stored (in pmstore) can then be removed.
- An optional format specifier (to pmpaste, above) for binary pastes
would be used to label the binary data - in particular compression
format would be highly useful to have stored, possibly allowing us
to automatically uncompress during extraction.
Alternatives:
Some discussion around the use of a more pmdashping-alike model, where
the commands are run by the PMDA and output captured for export, all
on the server side.
This has the upside that we could dodge the size limitation restriction
as the data doesn't need to be imported via pmStore(3) PDUs. It follows
that there is likely to be less import traffic too, since the command
line is probably going to be shorter than its resulting output, which is
advantageous for pasting over the network.
Downsides are that its less simple for the user - need to pass a command
into the PMDA, rather than the data, and less control on the environment
is available than from a regular user shell. Also, the PMDA would need
to have multiple threads of control, like pmdashping, so code complexity
goes up relative to the earlier solution. Would we go for a one-child-
process-per-command model? Or a fixed-size pool of processes (then have
to deal with pool-filling, & stall new paste activity?). It also forces
a requirement that the data originates from a command too, rather than
somewhere else (e.g. literally a paste into the terminal, by the user).
Clearly, I'm less favourable of this approach, however it might well be
preferable in some situations and certainly worth further investigation.
It is an almost complete superset of the earlier approach, however, in
terms of coding, so I'm advocating we do that first then extend into the
alternative solution space as needed.
Issues:
We should tackle the pmlogger-recording-error-codes-for-event-metrics
"problem", from some time back, as part of this work. Without this we
would end up with needlessly large archives in the presence of no paste
(event) data.
Free-form search within event metrics (or any metrics for that matter)
is not something we have really started considered tackling yet. It's
something for which a generic solution would be desirable (beyond the
scope here) - worth considering how to tackle searching in archives,
beyond use of the temporal index though.
|