On 07/15/2015 01:55 AM, Nathan Scott wrote:
> Hi David,
>
> ----- Original Message -----
>> [...]
>> I've scratched my head over this for a good bit of today, and I finally
>> figured it out. RHEL6 has python 2.6. The json.load() function in python
>> 2.6 doesn't support the 'object_pairs_hook' which the json pmda uses. I
>> haven't found a fix or workaround yet.
>
> Any thoughts on this one since then? (not sure it was communicated, but
> end result of this was Dave needed to switch pmdajson off in EL6 builds).
Right, I figured that switching pmdajson off would be the case for EL6.
For the record, we've got 3 (known) problems using the JSON PMDA with
python 2.6.
1) "json.load" syntax error. This one is fairly minor, and might be
fixed by switching to another one of the python json implementations or
just removing the use of OrderedDicts. They were very helpful during
development, since I was always processing JSON items in the order they
were present in the input file, instead of in some semi-random order.
2) "six" module missing in RHEL6. This one is actually fairly minor. The
"six" module is for python 2-to-3 compatibility. We could fix this by
copying the fairly small parts of the six module we actually use into
the python pmda. Another option would be to have 2 different versions of
the python JSON pmda available, one for 2.6 and one for 2.7+.
3) "jsonpointer" module missing in RHEL6. This is the major one, and the
reason I didn't bother working on 1) and 2). We could fix this at least
3 different ways I can think of: a) get the python-jsonpointer package
into RHEL6, b) bundle the python-jsonpointer module with pcp for python
2.6, or c) get pcp and the python-jsonpointer module into DTS and ask RH
customers who really want to use the JSON support use the DTS versions.
This one is really a distro problem, anyone building from source
themselves can obviously install anything they want.
> Given this, and the other niggling issues we're seeing from the python
> modules pmdajson uses, maybe we should start thinking outside the square.
> An alternative way to tackle this class of issue (incl. not having json
> pointer packages at all on some platforms, and so on) could be to provide
> a lower-level implementation of json parsing & pointing in libpcp_pmda.
>
> We've already seen need for non-python PMDAs (pmdaroot, pmdaelasticsearch)
> to parse JSON and we already have a teensy embedded json parser in C (~300
> LOC IIRC). Given the increasing prevalence of json instrumentation in the
> wild, perhaps we should promote its use as a first class PMDA concept, and
> make use of (new) underlying API services in pmdajson and other PMDAs.
>
> ( Obviously, we need short-term stop-gap fixes for the python packaging
> issues for some of our builds - possibly embedding the jsonpointer code
> into the python pmdajson code as Frank suggested, or something like that;
> but I'm thinking of longer-term, cross-platform (where "cross" includes
> RHEL5, RHEL6, as well as Mac OS X, Solaris, Windows, etc), and not-just-
> for-python-code solutions here. )
>
> So, in this model we might:
>
> - move JSON parsing into libpcp_pmda or libpcp_http from pmdaroot/jsmn.c
> and make the pmdaroot Docker code use the shared library code
>
> - implement jsonpointers in C (the spec looks quite simple, so I'd not
> expect alot of C code there? - I'd be v. interested in your take on
> that, David, since you've worked with it alot now)
I really don't have a "take" here. I was pretty focused on using
jsonpointer, not looking at how they were implemented internally in the
python module.
> - provide some clean PMDA-appropriate C API above 'em. Maybe not just
> "parse this arbitrary-sized memory blob" (probably too simple - e.g.
> we may want to be able to handle streaming json from a socket?), and
> possibly using jsonpointers only at the API level? - that's all that
> pmdaroot would need from an API, anyway
>
> - interface this new PCP library code to python too, of course; also
> to perl, and oh also for pcp-pmda-cpp C++ PMDAs if Pauls keen on that
>
> - switch pmdajson over to using these new interfaces, removing those
> problem "jsonpointer", "json.load", "six" module/version dependencies
> entirely.
>
> Again, this is all longer-term planning, its not going to be a quick fix.
>
> It would resolve all the current packaging problems though, and bring the
> wonders of JSON instrumentation to all platforms we support PCP on (which
> is clearly not going to happen with a python-only solution - some of the
> platforms do not have python at all, others do not have jsonpointers at all
> (and are in stages of their life, like RHEL5/6, where this will not happen).
As far as python goes, at some point you'll have to decide if python is
a good way to write a PMDA or not. If it isn't, you should probably
think about removing the python PMDA support. Ditto for perl.
> Thoughts? We'd need to flesh out the API a whole lot more, I've not given
> a whole lot of thought to that yet (suggestions welcome!), and we may want
> to pick a more general purpose embedded C JSON library to start from.
Wow, this is far outside the square. Resolving the current packaging
problems seems like a problem that is quite doable in several different
ways, and what you are describing above is several magnitudes of effort
above fixing the packaging problems.
Putting JSON parsing functionality into libpcp_pmda or libpcp_http seems
a bit odd to me. It doesn't seem like it fits in either, especially
libpcp_pmda. Perhaps you'd need a new 'libpcp_json' library.
I'm not sure I see the value in the short-to-medium term for rewriting
everything in C/C++. You'd certainly have a better idea of the long term
benefits.
If you did decide to go this route, I'd *certainly* pick a
well-supported general purpose C JSON library to start from.
> Incidentally, this may help to resolve some of the other pmdajson worries
> still in the back of my mind (which I still owe you some mail on, sorry
> 'bout the tardiness there - will follow up soon).
As far as I know all the worries that you've mentioned to me in the past
have been addressed. If you've got some new ones, send them on and I'll
look at them.
--
David Smith
dsmith@xxxxxxxxxx
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
|