On 07/24/2014 08:43 PM, Paul Smith wrote:
> I _love_ JSON,It's a useful format for clients to receive post data in a lot
> of cases, but for this sort of functionality I would not think this is the
> best internal format for capturing the events. Perhaps there's some clients
> that want to post that data easily via JSON, but make that an optional case,
> behind the scenes it's stored as binary. If you wanted to keep the format
> close to JSON, you could consider SMILE:
>
> http://wiki.fasterxml.com/SmileFormat
>
> For what it's worth, the clustered search service ElasticSearch has 2
> connectors available, a JSON friendly text based connector which allows
> simple REST services to plugin via this text protocol, and a second connector
> mainly aimed at the companion ElasticSearch Java client layer that talks
> binary SMILE for efficiency.
>
> The text encoding/decoding can be expensive, so I would be careful about
> using it for internal storage/querying, but perhaps SMILE gives you something
> close.
>
> cheers,
>
> Paul Smith
Hi Paul,
I have noticed this problem of binary->ascii->binary conversion overhead with
high volume trace points in particular the netdev-times script that is
available with perf. The script spent a lot of time in the code
reading/translating the data.
I know it is going to shades of gray rather than a black and white decision,
but are there thoughts on guidelines about when it JSON would be bad choice for
use? Or when JSON would be a reasonable choice? JSON is human readable and is
one of the reasons that people might prefer to use it. Is there some kind of
reader that we can point people at to make it easier to read the SMILE data and
verify that their pmda is generating reasonable data? That might tip things in
SMILE's favor.
-Will
>
> On 25 Jul 2014, at 8:38 am, Ken McDonell <kenj@xxxxxxxxxxxxxxxx
> <mailto:kenj@xxxxxxxxxxxxxxxx>> wrote:
>
>> On 23/07/14 07:01, David Smith wrote:
>>> On 07/22/2014 10:21 AM, Frank Ch. Eigler wrote:
>>>> Hi, David -
>>>>
>>>>
>>>>> [...]
>>>>> I think a JSON fetcher/parser is a good idea.
>>>>
>>>> Good enough to redirect from MMV?
>>>
>>> I think MMV and a JSON fetcher/parser aren't mutually exclusive. I'd say
>>> they are complementary.
>>>
>>
>> Joining a bit late here ... 8^)>
>>
>> Please, if there is any choice, let's not add binary->ascii and
>> ascii->binary format conversions on the code path between the producer and
>> consumer of high volume and/or low latency performance data. I hope the
>> reasons are self-evident to this audience.
>>
>> Pushing data from PCP via some JSON plumbing seems to be addressing a
>> different, but important use case. Pushing data into PCP via JSON may be
>> required if the producer of the data provides no more efficient API, and is
>> again a different use case.
>>
>> The mmv model has worked well in the past. If it is not adequate in some
>> respect (e.g. for event records), then I'd advocate making a variant of the
>> mmv model that is refined to address the inadequacies (or even extend the
>> existing mmv model). There is a lot of pain that has been invested in
>> maturing the operational and semantic robustness of the mmv model, so don't
>> discount the effort that would be required to get a completely new model up
>> to the same production standard.
>>
>> _______________________________________________
>> pcp mailing list
>> pcp@xxxxxxxxxxx <mailto:pcp@xxxxxxxxxxx>
>> http://oss.sgi.com/mailman/listinfo/pcp
>
>
>
> _______________________________________________
> pcp mailing list
> pcp@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/pcp
>
|