Proposal for handling dynamic metric names (and hence dynamic metrics)

[Ken McDonell <kenj@xxxxxxxxxxxxxxxx>]

I've been threatening to get this out for sometime now.

There is no code to back any of this up (yet), it really is a
proposal ... so please let me know if you think this is a good or bad
idea, and holes being picked in the issues covered would be most
welcome, as would better ideas.

The motivation here is to get pmcd out of the way for cases where it is
the PMDA that knows what metrics are available, not the PMNS loaded by
pmcd, e.g. an mmv-like PMDA where the available metrics are discovered
from mmap'd files when the PMDA starts.

Proposal for Supporting Dynamic PCP Performance Metric Namespaces

Ken McDonell
kenj@xxxxxxxxxxxxxxxx

Initially in PCP, the Performance Metrics Namespace (PMNS) was local to each machine where PCP was being used. This made it difficult to co-ordinate the PMNS versions on multiple monitoring machines with the PMDAs installed on the collector machines and was quickly identified as a weakness and replaced by the "Distributed PMNS" we have today where the PMNS is maintained on the PCP collector machine or within the PCP archive. Monitoring applications ship their namespace requests to the relevant source of metrics, namely a pmcd or a PCP archive.

Aside from the rare use of a local PMNS (with the –n option) by PCP monitoring applications, the principal use of the PMNS is to be loaded (or reloaded) by pmcd and then used by pmcd to respond directly to remote requests from PCP monitoring applications using pmLookupName() (or the asynchronous equivalent pair pmRequestNames() and pmReceiveNames()), pmNameID() (or the asynchronous equivalent pair pmRequestNameID() and pmRecieveNameID()), pmNameAll() (or the asynchronous equivalent pair pmRequestNameAll() and pmReceiveNameAll() &mdash although the former is defined, documented but not implemented!), pmGetChildren(), pmGetChildrenStatus() (or the asynchronous equivalent pair pmRequestNamesOfChildern() and pmReceiveNamesOfChildren()) and pmTraversePMNS() (or the asynchronous equivalent pair pmRequestTraversePMNS() and pmReceiveTraversePMNS()).

The PMNS on a collector machine is maintained as a single file with entries added and deleted as a part of the installation and removal of a PMDA.

While this regime has served PCP well for most PMDAs, there have been a small number of cases where the static nature of the PMNS has not been appropriate, e.g.

• PCP's one-dimensional instance domain data model imposes the need for remapping when data naturally occurs across more than one dimension, e.g. histogram bins of service times for a set of similar operation types. This is usually addressed by either linearizing the instance domain and constructing composite instance names, or mapping one of the dimensions onto the PMNS. When both dimensions are variable this can become quite messy.

  All of the DBMS PMDAs had issues in this area.

• Some PMDAs are data-driven, e.g. the memory mapped class of PMDAs, and for these the names of available metrics maybe "discovered" when the memory mapped files are opened and so cannot be part of the PMDA's PMNS at the time the PMDA is installed.

Existing methods for handling a dynamic aspect of the PMNS are all ugly and error-prone, e.g. make a new PMNS, update the global PMNS and sent pmcd a SIGHUP signal.

Proposal Overview

The existing PMNS will be extended (no backwards compatibility issues) to introduce a new "non-terminal" node that will be used to indicate that the PMNS below this point is dynamic and defined by the associated PMDA.

As an example to be used throughout this proposal, the foo PMDA (domain 44) supports dynamic names below the foo.count node in the PMNS. The relevant fragment of the ASCII PMNS would be as follows:

root {
	...
	foo
	...
}
...
foo {
	version	44:0:1
	count		44:*:*
	memory	44:0:2
}

The foo PMDA is willing to export metadata and metric values for
the following additional (dynamic) metrics:
      
foo.count.ops (PMID 44:1:0)
      
foo.count.errs (PMID 44:1:1)
      
foo.count.numcount (PMID 44:0:27)

Changes to pmcd and new interactions with the foo PMDA would mean that attempts to look up the PMIDs for metrics with names beginning foo.count. would be passed from pmcd to the foo PMDA, and similarly requests to find the names of metrics given their PMID would also be passed from pmcd to the foo PMDA if they are not resolved in the PMNS loaded into pmcd.

Detailed Changes Required

Changes to the ASCII PMNS Format

As forshadowed, the syntax :*:* after a domain number would flag a PMNS node as the root of a subtree of names to be resolved in the associated PMDA.

The only place where the ASCII PMNS format is known at this level of detail is in the internal routine loadascii() of libpcp which is called from pmLoadNameSpace(), pmLoadASCIINameSpace() and pmGetPMNSLocation(). So extending the parser here is simple.

Changes to the Binary PMNS Format

The binary format of the PMNS is what is loaded into the address space after the ASCII PMNS has been parsed (it is also the format generated by pmnscomp and read by the libpcp routines, but this is just a performance short cut — pmnscomp will need almost no change as it simply writes out the binary PMNS after it has been loaded).

The relevant data structure is __pmnsNode (defined in <pcp/impl.h>). Now this structure is sufficiently public that we cannot change it in any way that would break binary compatibility, and the only field avalable to encode both the PMDA's domain number and the dynamic nature of the node in the PMNS is the pmid field. Internally a pmid is structured thus (ignoring the endian alternative form):

typedef struct {
	int			pad : 2;
	unsigned int	domain : 8;
	unsigned int	cluster : 12;
	unsigned int	item : 10;
} __pmID_int;

So the domain field must be used to encode the domain of the PMDA providing the dynamic names, but unfortunately there are no values for cluster and/or item that could be used to mark the node as the root of a subtree of dynamic names. By good fortune we have spare bits hiding in the pad field, so the proposal is to extend the __pmID_int struct to allocate one of the bits from pad to the new field dynamic, as follows:

typedef struct {
	unsigned int	dynamic : 1;
	int			pad : 1;
	unsigned int	domain : 8;
	unsigned int	cluster : 12;
	unsigned int	item : 10;
} __pmID_int;

A value of 1 for
dynamic
encodes the fact that
this PMNS node is the root of a dynamic subtree.
Leaving
pad
between
domain
and
dynamic
would allow the
domain field of
a PMID to expand to 9 bits if that becomes necessary at some point in the
future.
This change does make a dynamic PMID negative when treated as a 32-bit integer
but this should not be a problem as code of the form:
	if (pmid < 0) ...
is just plain wrong, and should probably be
	if (pmid == PMID_NULL) ...

Internally, the PMID for a node at the base of a dynamic subtree would be encoded as 1::4095:1023, i.e. the pad field set to 1 and all ones in the cluster and item fields to minimize the chance of any "false" matching.

Changes for pmlogger and PCP Archives

No changes are needed here as pmlogger tolerates missing metrics and only adds PMNS and metadata information into the PCP archives for those metrics that can be found, so the PMID for the the root of a subtree of dynamic names will never appear in an archive, although the descendent nodes (with their associated names and PMIDs) may appear in an archive.

Changes for libpcp_pmda

To support the additional interactions between pmcd and the PMDAs the pmdaInterface structure needs to be extended. This will be PMDA_INTERFACE_4, and involves adding struct { } three; to the union, with all of the fields from struct { } two; plus the following:

	int	(*pmns_pmid)(char *, pmID *);
	int	(*pmns_name)(pmID, char **);
	int	(*pmns_children)(char *, char ***, int **);

The standard implementation of these routines should suffice for the majority of cases, but they are exposed in the interface to allow an over-riding implementation should that be necessary (this also makes them consistent with all other PDU handling routines in the PMDA library).

Changes for libpcp

The table below describes the changes that are needed in various libpcp routines that are used once a PMNS is loaded (for simplicity we've omitted the asynchronous versions of these synchronous routines, but the same semantics would apply to the asynchronous versions).

libpcp Routine	PMNS_LOCAL	PMNS_REMOTE
pmLookupName	If namelist[i] prefix matches the path to a dynamic node in the PMNS, then pmidlist[i] is set to a dynamic PMID (with the domain domain of the PMDA from the root of the dynamic subtree, the cluster and item serial fields are set to all ones and the dynamic field set to 1) and pmLookupName() returns success. So foo.count would return PM_ERR_NONLEAF (this is a special case that needs to be checked for), while foo.count.ops and foo.count.anything.else would both return the PMID 1:44:4095:1023.	Ship to pmcd, where pmLookupName() is called as in the PMNS_LOCAL case. Scan the resulting pmidlist[] and if a dynamic PMID is found and if the associated PMDA is using PMDA_INTERFACE_4 ship the original metric name to the matching PMDA and use the PMID and status that is returned from the pmns_pmid() method, otherwise do nothing. So foo.count would return PM_ERR_NONLEAF, foo.count.ops would return PMID 0:44:1:0 and foo.count.anything.else would PM_ERR_NAME.
pmNameID	Nothing really special is required. Any PMID that could be associated with dynamic names will fail to match in the PMNS and PM_ERR_PMID will be returned. If the PMID has the dynamic field set to 1, then it might be possible to find and return the name at the root of a dynamic subtree (e.g. foo.count for PMID 1:44:4095:1023), but since this is potentially ambiguous and of no apparent use, PM_ERR_PMID will be returned in this case also.	Ship to pmcd, where pmNameID() is called as in the PMNS_LOCAL case. If PM_ERR_PMID is returned, then some extra processing is required. If PMID has the dynamic field set to 1 or the associated PMDA (from the domain field of the PMID) is not using PMDA_INTERFACE_4 then return PM_ERR_PMID as this cannot match any valid metric. If the domain field in the PMID matches any node in the PMNS that is the root of a dynamic subtree then ship the original PMID to the matching PMDA where the pmns_name() method is called and use the metric name and status that is returned.
pmNameAll	Same handling as for pmNameID().	Same handling as for pmNameID().
pmGetChildren	If name matches the path to a dynamic node in the PMNS then return zero (no descendents), otherwise there is no change in behaviour. So the descendents of foo will be version, count and memory, but there are no descendents of foo.count.	Ship to pmcd, where pmGetChildren() is called as in the PMNS_LOCAL case. If there are no descendents of name, and name is the root of a dynamic subtree in the PMNS, and the associated PMDA (from the domain field of the matching PMID) is using PMDA_INTERFACE_4 then ship the original name to the matching PMDA where the pmns_children() method is called and use the list of offspring and status that is returned. Otherwise (not PMDA_INTERFACE_4 or no matching PMDA) if name is the root of a dynamic subtree in the PMNS then it is skipped. So, for example calling pmGetChildren with foo as the name argument will return foo.version, foo.count.ops, foo.count.errs, foo.count.numcount and foo.memory.
pmGetChildrenStatus	Similar handling as for pmGetChildren(), except that the status for a descendent node that is itself the root of a dynamic subtree in the PMNS is set to PMNS_NONLEAF_STATUS.	Similar handling as for pmGetChildren(), except the status may also be returned from the PMDA if pmcd ships the request to a PMDA.
pmTraversePMNS	Any node that is at the root of a dynamic subtree of the PMNS is never returned to the dometric() method.	Ship to pmcd, where pmTraversePMNS() is called as in the PMNS_LOCAL case, however whenever a node is encountered that is the root of a dynamic subtree in the PMNS (i.e. the matching PMID has the dynamic field set to 1 and the associated PMDA (from the domain field of the PMID) is using PMDA_INTERFACE_4) then the path is shipped to the associated PMDA where the pmns_children() method is called and all of the decendent names are returned to pmcd which inserts these into the list of names to be returned to the caller for subsequent callbacks to the dometric() method.
pmLoadNameSpace	Parser changes as noted above, e.g. for 44:*:*. Do not check for duplicate PMIDs in the case of a node that is the root of a dynamic subtree, as there could be more than one dynamic subtree associated with the same PMDA.	Not relevant.

A small change is needed in pmIDStr() to detect a dynamic PMID (checking if the dynamic field is 1), and then output an asterisk in place of the numeric cluster and item fields.

Some Limitations

The "dynamic" nature of the PMNS only applies to the PMNS at the time it is explored. For most monitoring tools this is at start up (typically after a configuration file has been read), so any changes to the PMNS after that point in time will not be noticed. Specifically this means:

• pmchart explores the PMNS each time a new plot is added, so metrics missing at the time a view is populated are dropped with a warning, but the metric selector is re-populated each time it is launched
• pmlogger will only log those metrics defined at the time it is started
• pmie will return "unknown" as the value of any _expression_ involving metrics not defined at the time it is started
• pmdumptext will only report on those metrics defined at the time it is started
• pmval reports an error and exits if the metric name is not defined at the time it is started

This behaviour is no different to other existing interactions, e.g. when a PMDA is installed or removed, so is not a new issue.