On Thu, 2010-07-08 at 04:09 +1000, nathans@xxxxxxxxxx wrote:
> ...
> > so a Perl pmimport program might look like:
> >
> > use PCP::IMPORT;
>
> Can we call it PCP::LogImport please? IMPORT doesn't really
> describe it that well IMO. Happy to keep the libpcp_import
> name for the C API though.
Done.
> > use vars qw( $import $myindom $inst $ctr $now $time );
> > $import = PCP::IMPORT->new('');
>
> That seems like maybe it should take more (optional?) args?
> (log name, host name, timezone).
We've moved on a bit from this earlier discussion ... the equivalent
Perl code fragment would now be ...
$ctx1 = pmiStart("myarchive", 0); # do not inherit metadata from current
context
$_ = pmiSetHostname("somehost.com"); # optional
$_ = pmiSetTimezone("GMT-12"); # optional
> > $myindom = 1;
> >
> > $import->add_metric('myown.counter', PM_ID_NULL, PM_TYPE_U32,
> > PM_INDOM_NULL, PM_SEM_COUNTER, units(0,0,1,0,0,PM_COUNT_ONE));
> > $import->add_metric('myown.time', PM_ID_NULL, PM_TYPE_FLOAT,
> > $myindom, PM_SEM_INSTANT, units(0,1,0,0,PM_TIME_SEC,0));
> > ...
>
> (hmm, whats the algorithm for auto-picking PMIDs going to be like?
> esp, how will the domain# be chosen & guaranteed not to conflict?
> I like the idea of auto selecting IDs, just need to make sure its
> deterministic & non-conflicting with existing agents I guess).
We already had domain 245 reserved (was COUNTD used only by the example
old pmimport plugin) ... I just renamed this to be LOGIMPORT. I've
added helper functions in the Perl API to build pmUnit, pmID and pmInDom
values, as in these fragments from LogImport.xs ...
pmUnits
pmiUnits(dimSpace, dimTime, dimCount, scaleSpace, scaleTime, scaleCount)
pmID
pmiPmid(domain, cluster, item)
pmInDom
pmiIndom(domain, serial)
All return a value that looks like an integer to Perl.
It turns out that autogenerating the pmIDs works just fine (is
deterministic for a given Perl program and avoids clashes with other
PMDAs), but the pmInDoms are a bit of a problem ... you cannot
autogenerate these as you _must_ use the matching pmInDom value in the
calls to pmiAddMetric() and pmiAddInstance(). So even if I autogenerate
the pmIDs, the other calls end up with the sort of messy specifications
like pmIndom(245,3) for the calls to pmiAddMetric() and
pmiAddInstance(). I cannot see a clever way around this.
> ...
>
> > I've toyed with a pmiConfig() routine as a generic metadata parser
> > ... I cannot see how to marry up the resultant metadata with
> > the data stream.
>
> I'm not sure I understand the problem ...
>
> > Consider a 5 column spreadsheet and the config (in Nathan's syntax):
> > ...
>
> But the call to pmiConfig would effectively replace all the calls to
> $import->add_metric and $import->add_instance, no? Those don't give
> any hint as to which column in the spreadsheet they match up to either
> ... thats done via "put_value('metric name', ...)" or a handle, which
> would still be the case with a pmiConfig call. I'm missing something
> here I think...?
What I was trying to develop was a totally data driven solution for a
particular input data format, e.g. a spreadsheet so that you could take
the input data file, create a config file and run a shipped generic
(Perl) application.
I think this may still be possible but it requires one more section of
metadata to map the metric-instance pairs onto the sequence of data
items in each record of the data set (so the column -> metric-instance
pair mapping for a spreadsheet).
If I was to do this, I'd probably make the config file syntax XML (does
that seem sensible to others?).
This is very much a second stage to be built on the first version of the
libpcp_import library and Perl wrappers.
|