On 02/19/2016 01:38 AM, Ken McDonell wrote:
-----Original Message-----
From: pcp-bounces@xxxxxxxxxxx [mailto:pcp-bounces@xxxxxxxxxxx] On
Behalf Of Mark Goodwin
Sent: Friday, 19 February 2016 2:54 PM
To: pcp <pcp@xxxxxxxxxxx>
Subject: [pcp] log import needs ability to write mark records?
collectl2pcp can take multiple input logs from collectl and create a
single PCP archive. The input logs are often not temporally
contiguous, i.e. there may be a time interval between one log and the
next.
Replaying the PCP log produces some unexpected results, and I suspect
this may be due to missing mark records. Is there any way to write out
a <mark> record in the PCP log, between processing input files? I
don't see any pmiFoo API function in LOGIMPORT for this ...?
maybe it's better to write individual PCP logs, and then merge them?
Mark,
If you're sure about the temporal discontinuity (a real gap between the end
of one archive and the start of the next archive in temporal order), then I
could see the case for adding pmiPutMark() or similar to the Perl LogImport
module (I don't think this is a libpcp candidate, so the implementation will
need to be entirely in the perl wrapper).
The code behind pmiPutMark() is a little tricky ... see putmark() in
pmlogger/src/callback.c
On the other hand, if collectl2pcp can't be sure of the temporal semantics,
it would be better to process one archive at a time and then let
pmlogextract merge 'em all if you need a single output archive.
Dave,
Please note this brings up a case where pmlogextract should _not_ follow the
new multiple-archives semantics ... it _really_ needs to know where each
archive ends, so transparently stitching multiple archives together under
the covers in libpcp is going to be an issue. I think we need a discussion
on some sort of new API to allow tools to discover how many archives are
really behind an archive context ... pmlogextract needs the answer to be 1
for all of the input archive "names" on the command line (I think).
Looks like I picked a bad day for a day off! ....
All of this worry about <mark> records for multi-archive contexts may be
unnecessary. You may all recall a few weeks back when I implemented the
automatic generation of <mark> records when transitioning between the
archives of a multi-archive context on my branch. These <mark> records
are returned to the caller of __pmLogRead() and so are returned to all
users of our internal APIs. As Ken had recommended, the effect is the
same as if pmlogextract(1) has already been used to combine the archives
of a multi-archive context into a single archive.
Now, tools which manually transition between archives or archive
contexts, of course, still will need to generate their own <mark>
records, so it looks like there is still a need for an API for this.
pmlogextract(1) already does this but it sounds like pmcollectl2pcp(1)
does not yet do this. __pmLogGenerateMark() in logutil.c on my branch
may be a suitable place to start, once my branch has been merged ---
hopefully very shortly after I return from vacation on March 8.
Dave
|