On 01/13/2015 10:41 AM, Frank Ch. Eigler wrote:
> Hi -
>
> With the recent python-pmda pmns-flexibility improvements, we can
> revisit the old thread [1] about the metadata/schema format for the
> json-pmda. When we left off, the prototype was based on a slightly
> enlarged json-schema [2] format. It's kind of attractive because it
> defines a simple mapping from json payload to metadata, by mirroring its
> nesting structure to some extent.
>
> Since then, I've come across a widget called jsonpointers. This is a
> standardized notation for identifying parts of a json payload via a
> string syntax; python libraries for it are easily available. The nice
> thing about this is that it would allow our pcp-json metadata to be
> focused on just what's needed to extract pcp metrics from an arbitrary
> json document: no fluff.
>
> Here's a hypothetical rewriting of the metadata [4] of dsmith's original
> prototype. (The payload [5] could be unmodified, but let's imagine
> that the "generation":1 and "data": { } wrappers are removed, and
> write a metadata for that variant.) It should give the same pminfo
> output (except for extra units included here).
... stuff deleted ...
I finally got a good bit of this working (I haven't implemented units
support yet).
I did make a few changes to the metadata format when it comes to arrays.
Your original proposal looked like this:
====
{ "pcp-metrics":[
{"pmns": "json.dummy_array.dummy2",
"indom-str": "/dummy_array/-/__id", # identify indom field
"pointer": "/dummy_array/-/dummy2", # use - as array-index
jsonpointer
"type": "string"},
{"pmns": "json.dummy_array.dummy1",
"indom-str": "/dummy_array/-/__id",
"pointer": "/dummy_array/-/dummy1",
"type": "int64",
"semantics": "counter"},
...
====
I changed things around a bit to make it easier to implement and perhaps
be a bit more readable:
====
{
"metrics": [
{
"name": "dummy_array",
"pointer": "/dummy_array",
"type": "array",
"index": "/__id", # indom field, jsonpointer is relative to the
array entry itself
"metrics": [
{
"name": "dummy1",
"pointer": "/dummy1", # jsonpointer is relative to the array
entry itself
"type": "integer"
},
{
"name": "dummy2",
"pointer": "/dummy2",
"type": "string"
}
]
},
...
====
In my version, the metrics contained by an array are stored within a
"metrics" array inside an item for the array itself. In addition, all
the jsonpointers to fields of an array are relative to the array item
itself. This made implementing things easier since I no longer have to
parse jsonpointer strings. As I handle each array item in the data the
jsonpointers work as is. In addition, I don't have to specify the indom
field for every item, just once for the array.
I've attached 4 files:
- net_xmit_json7.stp: systemtap script
- data: json data produced by the systemtap script
- metadata: metadata file produced by the systemtap script
- pmdastap_json.python: PCP pmda (also checked into the dsmith/dev
branch of pcpfans.git)
Comments welcome.
--
David Smith
dsmith@xxxxxxxxxx
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)
data
Description: Text document
metadata
Description: Text document
net_xmit_json7.stp
Description: Text document
pmdastap_json.python
Description: Text document
|