pcp
[Top] [All Lists]

Re: JSON PMDA (was Re: [pcp] PCP JMX PMDA)

To: David Smith <dsmith@xxxxxxxxxx>, Nathan Scott <nathans@xxxxxxxxxx>
Subject: Re: JSON PMDA (was Re: [pcp] PCP JMX PMDA)
From: Marko Myllynen <myllynen@xxxxxxxxxx>
Date: Tue, 26 Apr 2016 22:09:26 +0300
Cc: pcp developers <pcp@xxxxxxxxxxx>
Delivered-to: pcp@xxxxxxxxxxx
In-reply-to: <571FB5F1.6000502@xxxxxxxxxx>
Organization: Red Hat
References: <56D8858A.3020407@xxxxxxxxxx> <56F940C7.2080909@xxxxxxxxxx> <CF911C98-E061-46AF-A8DB-10A5361C413B@xxxxxxxxxx> <56FC0E5B.3040708@xxxxxxxxxx> <1847433648.36205882.1459557172304.JavaMail.zimbra@xxxxxxxxxx> <570AD655.7020108@xxxxxxxxxx> <1799198159.39292885.1460350959108.JavaMail.zimbra@xxxxxxxxxx> <570BC712.1080905@xxxxxxxxxx> <880763790.39521507.1460426029264.JavaMail.zimbra@xxxxxxxxxx> <5710CC97.5020209@xxxxxxxxxx> <571E4C9D.2070708@xxxxxxxxxx> <571F0AFA.3080005@xxxxxxxxxx> <571FB5F1.6000502@xxxxxxxxxx>
Reply-to: Marko Myllynen <myllynen@xxxxxxxxxx>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0
Hi,

On 2016-04-26 21:39, David Smith wrote:
> On 04/26/2016 01:30 AM, Marko Myllynen wrote:
>> On 2016-04-25 19:58, David Smith wrote:
>>> On 04/15/2016 06:12 AM, Marko Myllynen wrote:
>>>> On 2016-04-12 04:53, Nathan Scott wrote:
>>>>> ----- Original Message -----
>>>
>>>> Oh, well, I think I was being overly optimistic here.. But perhaps (or
>>>> even hopefully) some of the issues I'm listing below are due to my
>>>> misunderstanding (see the previous README patch - the current README
>>>> wasn't enough even to get a basic example working):
>>>>
>>>> * (When *testing* metadata, deleting /var/lib/pcp/config/pmda/137*
>>>> before each round may help to prevent some mysterious error messages)
>>>
>>> Without seeing the "mysterious error messages", I'm afraid I don't
>>> really know what the JSON PMDA was doing here.
>>
>> Actually, I was a bit confused what was going on but nevertheless there
>> are few surprising things. With metadata.json and data.json looking like:
>>
>> {
>>   "prefix": "jsontest",
>>   "metrics": [
>>     {
>>       "name": "a_b_c",
>>       "pointer": "/a_b_c",
>>       "type": "integer"
>>     },
>>     {
>>       "name": "a_d_e",
>>       "pointer": "/a_d_e",
>>       "type": "string"
>>     },
>>     {
>>       "name": "a_b_f",
>>       "pointer": "/a_b_f",
>>       "type": "integer"
>>     }
>>   ]
>> }
>>
>> {
>>   "a_b_c": 1,
>>   "a_d_e": "2",
>>   "a_b_f": 3
>> }
>>
>> First clear the cache and install the PMDA to see all is ok. Then change
>> 1 to "1" in data.json, confusingly there's error printed for both a_b_c
>> and a_b_f (I'd expect error at most for a_b_c). Now revert "1" back to 1
>> and then verify things are ok again. Then change 3 to "3" and see how
>> the PMDA crashes.
>>
>> Of course, the user is responsible for providing correctly formatted
>> data but when testing (possibly with hundreds of metrics) such
>> inconsistencies in error cases can be a bit confusing.
> 
> The PMDA certainly shouldn't crash there, that's a bug.
> 
> As far as "providing correctly formatted data" goes, I'd expect that for
> most cases of hundreds of metrics, you wouldn't write the metadata by
> hand, but generate it from a script (as the generate_ceph_metadata
> script does).

Sure, nobody hopefully writes down hundreds of definitions/values by
hand. But when developing/testing any error slipping in might be hard to
find (although still possible).

>>>> * In the non-array case metrics a.b.c and a.d.e work but a.b.f is
>>>> silently omitted (so using a_b_c seems currently the only option)
>>>
>>> I'm afraid you lost me here. Can you explain a bit more?
>>
>> The above metrics still work if you do s,_,.,g but in a simalar simple
>> array case the PMDA log is being filled with errors and tracebacks.
> 
> Without seeing the array case I couldn't guess at what is going on.

Ok, here are example files:

{
  "prefix": "json",
  "metrics": [
    {
      "name": "test",
      "pointer": "/test",
      "type": "array",
      "index": "/__id",
      "metrics": [
        {
          "name": "a.b.c",
          "pointer": "/a.b.c",
          "type": "integer"
        },
        {
          "name": "a.d.e",
          "pointer": "/a.d.e",
          "type": "string"
        },
        {
          "name": "a.b.f",
          "pointer": "/a.b.f",
          "type": "integer"
        }
      ]
    }
  ]
}

{
  "test": [
    {
      "__id": "123",
      "a.b.c": 1,
      "a.d.e": "2",
      "a.b.f": 3
    }
  ]
}

With these you should see a failure but if you then change . to _
everywhere, everything starts to work ok.

>>>> * metadata.json updates are not picked up during JSON PMDA lifetime
>>>
>>> Yes, that is true. As far as I know, that's fairly standard for PMDAs,
>>> especially python ones.
>>
>> Ok, Nathan mentioned earlier that the Python PMDA API was extended to
>> allow adding metrics dynamically so if metadata.json updates are not
>> picked up then on the fly then perhaps he meant something else.
> 
> OK, I looked into this one a bit more. I was a bit mistaken.
> 
> If you add a new metadata source, the JSON PMDA will pick up the new
> source. If you change an existing source during the PMDAs lifetime, it
> will not notice the change. You'd have to restart the PMDA for it to
> reread the metadata.

Hmm, a new metadata source is added in pmdas/json/config.json, right? I
don't see the new metrics to appear in that case. Or is there another
mechanism to add a new source?

Thanks,

-- 
Marko Myllynen

<Prev in Thread] Current Thread [Next in Thread>