Hi All,
I've been quietly adding code to my git://git.pcp.io/brolley/pcp.git
multi-archive branch over the past few months and I believe that it
is getting close to the point where a merge into the pcp master
branch will be possible.
There have been discussions about various technical issues with
regard to this work on the list. However, I think that now would be
a good time to summarize what is on the branch and how the
command-line and API interfaces work. If this work is to be merged
in time for the next release (PCP 3.11.1, scheduled for March 18),
then these details need to be solid by that time.
Overview
The purpose of this work is to allow libpcp clients to treat sets of
compatible PCP archives as though they are one. A typical use case
would be the archives in the pmlogger(1) output directory.
pmlogger(1) restarts and daily log management cause archives to be
ended and new archives to be started within that directory. In
reality, tools often need to query data which spans these archives.
This forces manual intervention via tools like pmlogextract(1) in
order to consolidate the separate archives into one.
It would be much easier if libpcp contexts could automatically treat
these sets of archives as one, transitioning among them seamlessly
as though pmlogextract(1) had been used to consolidate them. This
initial implementation aims to provide that support for static sets
of archives which are:
- created on the same host
- using the same time zone
- temporally distinct (no overlaps)
- with no PMNS conflicts
Future enhancements will allow for reduced restrictions, where
possible, and for supporting the possibility that archives within
the set may still be actively being written to.
pmNewContext(3)
In order to support specifying sets of archives without breaking API
compatibility with older clients, pmNewContext(3) now accepts a
comma-separated list of names for it's name argument, when
the type is PM_CONTEXT_ARCHIVE. Each element in the list may
be the name of an archive, as currently supported, or the name of a
directory. When a directory is specified, it is searched for PCP
archives and the list is expanded as if those archives had been
specified directly. Directories are not recursively searched for
archives. Duplicate archive names in the list are ignored. The final
list of archives forms the set of archives over which the context
will operate.
There is precedent for the use of a comma-separated lists in libpcp.
See __pmAddOptHostList(). However, since this is a list of files
and/or directories, perhaps a colon-separated list in the style of
PATH might be more appropriate.
pmGetOptions(3)
At the command line level, pmGetOptions(3) has been enhanced in the
following ways:
- The argument to -a may now be a comma-separated list of names,
as described above
- The argument to -a may now be the name of a directory.
- Multiple -a options are now supported.
The arguments to each -a option are consolidated into a single
comma-separated list which is then presented to pmNewContext(3).
The exception to the above are tools which use PM_OPTFLAG_MULTI
(e.g. pmstat(1)). These tools use a separate PCP context for each -a
option specified. However, each -a option may still specify a
comma-separated list of names, each of which may be a directory or
the name of an archive.
Using this approach, many PCP tools have gained the ability to
support multi-archive contexts without any code changes.
The implication is that older versions of these tools also gain this
ability when paired with a multi-archive-enabled version of libpcp.
As discussed on this mailing list, there are some tools for which
older versions will be broken with respect to the handling of time
indices when paired with a multi-archive-enabled version of libpcp.
These tools will still work correctly when a single archive is
specified. The tools in question are pmdumplog(1) and pmlogcheck(1).
It has been suggested that perhaps these tools should be restricted
to working against single archives going forward. I disagree. I
believe that these tools could be updated to work with multi-archive
contexts and I see no reason to penalize the users of these tools if
that is the case.
Other APIs
Once a multi-archive context has been established, it is the intent
that all other PMAPI calls which operate on the context behave
seamlessly as if the set of archives are one, with the same result
as if pmLogExtract(1) had been used to consolidate the archives.
This includes queries and traversals against the PMNS of the set of
archives as well as reading forward and backward across archive
boundaries. In particular, <mark> records are generated as
archive boundaries are crossed at the __pmLogRead() level, allowing
the existing mechanisms to handle archive boundaries appropriately.
Remaining Tasks
- PMNS consistency checks (archive label consistency checks are
complete)
- libpcp features update (__pmGetAPIConfig())
- qa (some qa-style scripts have been written but not committed)
- Enhancements to individual tools
- man pages/books
Issues to Resolve
Based on the above, there remain a couple of issues for which a
decision needs to be made:
- Whether to use a comma or a colon (or either?) to separate the
names in the list passed to pmNewContext(3).
- Whether to restrict pmdumplog(1) and pmlogcheck(1) to single
archive contexts.
There are many, more experienced members of this team than I. I am
open to arguments and even edicts on these matters and anything else
I may have missed for this initial implementation.
Dave
|
|