On Wed, 2010-06-23 at 08:52 +1000, nathans@xxxxxxxxxx wrote:
> ----- "Ken McDonell" <kenj@xxxxxxxxxxxxxxxx> wrote:
> ...
> > I've attached the two man pages for the parts of pmimport (which have
> > been released by SGI, but are not in the PCP tree on oss.sgi.com yet,
> > pending a decision on what to do with pmimport).
>
> We should put them in the tree, just not installed (like the code).
Done in my local tree ... I'll commit to oss once 3.3.0 is out of the
way.
> > If there was consensus on exactly how this meta data was to be
> > encoded in the file, then it would be possible to write a generic
> > "csv" plugin for pmimport.
Leaving aside the implementation on the pmimport side for the moment
(I'll return to that further down this mail), I think there is a case
for a "standard" CSV (or spreadsheet, I don't really care) format that
will work in a data-driven way with a "standard" PCP import plugin. So
the task remains to define this "standard" so the plugin can extract the
necessary metadata from the first few lines of the spreadsheet ahead of
the data values. I don't have a proposal for this at the moment (beyond
some scratches on a scrap of paper), but it would be nice to agree that
this is something that should be done.
>
> The plan I had in mind for the pmimport tool and API was to make
> a Perl wrapper around a plugin API so that script plugins could
> be written. With that kind of approach we'd be giving punters
> access to CPAN for their data import situations (databases, a CSV
> parser that handles the corner cases, any date formats, and even
> direct spreadsheet parsing) and it's alot simpler and quicker to
> write scripts than C code.
[warning gross Perl ignorance area ahead]
I can sort of see how
pmimport (C) <-->
API wrapper (C) <-->
API wrapper (Perl) <-->
punter's plugin (Perl)
might work, except that the punter's plugn has to be a package or
somesuch (the main program is still pmimport).
Alternatively I could see an implementation architecture like
punter's main prog (Perl) with callback routines <-->
call pmimport_setup() (may not be needed, as primeImportFile() sort of
gets done in the punter's main prog ... perhaps some API to deal with
setting the host and timezone
calls to pmimport_register to register the callbacks
call pmimport_main() ... which is really the existing pmimport main
program hidden behinds a Perl wrapper
<--> call backs into the punter's Perl for getPmResult(),
getPmDesc()
and getIndom()
Is this even close to what you had in mind?
> Then an example Perl script or two for importing specific formats
> (using http://search.cpan.org/~hmbrand/DBD-CSV-0.29/lib/DBD/CSV.pm
> for CSV, perhaps, to give a DBD example importer - and/or using
> http://search.cpan.org/~hmbrand/Spreadsheet-Read-0.40/Read.pm for
> a direct spreadsheet->pcp archive export example).
>
> The PCP::PMDA module would be a good starting point - it wraps up
> the libpcp_pmda API, in much the same way as we'd want to capture
> the pmimport plugin API. Some pmimport code refactoring would be
> needed to make it more amenable to use from scripts though (perhaps
> the pmimport binary should become a libpcp_import with a Perl
> front-end driver script, which can use either shared lib or script
> plugins, for example) ... its a fair bit of work, but I think the
> end result would be alot better than just a CSV parser.
Is the front-end driver you're describing here similar to the first
architecture I explained above (with the main program in Perl rather
than C)?
> Also, if libpcp_import refactoring done specifically for scripts,
> it should be relatively straightfwd to do Python bindings, etc.
I have no clue what the guidelines for such a refactoring might be, so
I'll need some help here.
|