Hi,
On 2016-03-30 02:03, Paul Smith wrote:
>>
>> Yes, I think none of the other PMDAs come close to JSON/JMX PMDAs
>> in terms of their flexibility and generic nature. But at least with
>> pmdajmx this is all pretty well hidden at the most important level,
>> the user interface. For comparison, how would the corresponding
>> diagram look like with the Parfait agent approach you suggest
>> below? Would it be something like:
>>
>> pmdammv <-> parfait{-agent,} <-...-> multiple-java-apps
>
> Not quite, the Parfait Agent+Java app is a pair. The app uses
> Parfait internally to the process to write to the MMV file, and then
> the pmdammv is incorporating it into the PCP metric space. Since
> it's an Memory Mapped File, PMCD is never blocked from getting
> metrics.
>
> I would probably write it as:
>
> pmdammv <-> { parfait-agent<->java app}, {parfait-agent<->java app},
> ...
Ok, thanks for the clarification. Do we already know how this would look
like with multiple Java apps? PCP User/Admin guides don't give any
examples how to actually use MMV, it's of course easy to decipher how to
try it out with something simple like the acme.c demo app but there are
not much hints how it all would play out with several Java apps, each
possibly using different Java components.
> While the Parfait agent may be stalled, the PMCD->pmdammv is not. If
> I understand the pmdajmx correctly, it connects to the application
> process via the JMX socket communication protocol. Inside the JVM
> the JMX socket connector has a matching set of threads that provide
> the mechanism to respond to these requests.
The threads are per-JVM so the JVMs are being queried in parallel. But
like PMCD<->MMV<->Parfait, old data is served until the new data is
available.
> In the case of a stalled JVM (heavy GC pause) these threads are also
> blocked, and so the socket communication stalls, which I believe
> would flow on back up to the pmdajmx and stall that. This is of
> course, unless the pmdajmx decouples this already, so that PMCD can
> always get the last values that pmdajmx has received (even if a
> receiving thread within pmdajmx is blocked waiting on values over the
> wire).
It is decoupled, yes. But even so, one thing I'm wondering in general
is, how crucial is this in the first place - I mean that some other
well-established projects in this area (like Jolokia) use the standard
JMX even with its own JSON/HTTP on top and it doesn't look like a
show-stopper for them? IOW, do you see MMV first and foremost needed to
keep PMCD happy or due its minimal overhead?
>> Great to hear that you didn't find any memory leaks when testing!
>> ;) But yes, Java applications tend to consume more memory than C
>> programs. Have you already measured how much the Parfait agent +
>> Parfait (+ perhaps Spring, if it's also used?) will consume extra
>> memory?
>
> I'm not sure in the days of even VM's with minimum RAM sizes in GB
> that this memory discussion goes very far, however...
>
> IT's significantly less than the overhead of another JVM. My
> WetFinger is Parfait is several MB's compared with the overhead of
> the full JVM of at least 10's of MBs. While I personally don't think
> the overhead of another JVM is that big a deal, I think the
> _perception_ of having a a second JVM for monitoring being 'heavy' is
> definitely real.
Ok, I guess we can do some more precise measurements later once the code
matures (if anyone cares that much, I agree that we're not seeing
anything anywhere near the show-stopper category here).
>>> - That external tools.jar dependency is unfortunate for some
>>> users; all external dependencies cause pain for users and pain
>>> for PCP developers getting requests for help when bits aren't
>>> installed. (minor issue, we like talking to users really - but
>>> not everyone will followup).
>>
>> This one I certainly don't see as a problem at all. First of all,
>> tools.jar is part of the standard JDK every Java developer on the
>> planet already have installed on their system. Also, since it's
>> part of the standard JDK it's easier to have it accepted into use
>> in some environments/organizations than an alien component that
>> hasn't been used ever before (e.g., Parfait).
>
> The issue with tools.jar is that it contains among other things a way
> to compile java code, so I believe one of the reasons to separate the
> JDK (for developing and compiling the Java application) and the JRE
> (runtime, to execute the code) is to reduce the security footprint of
> the running java process. "Evil processes" could detect the
> availability of the tools.jar and use it for leverage for code
> execution. The JDK is a heftier install than a JRE.
As said, we could in theory drop the tools.jar dependency but I'm not
fond of the idea - we have had tools like apt/yum for ages to deal with
basics like dependencies so it can't be an issue. For the security
concerned it could be evaluated would it help to offer two versions for
connectivity so that users would have the choice and flexibility.
> I'm not sure I understood the Parfait-never-been-used-before bit.
I meant that when introducing Java monitoring with PCP to someone who
has no experience on this front at all, I'd think it's easier to get a
buy-in from them to enable tools.jar than something they've never heard
of before, i.e., Parfait. (Again, tools.jar could be made
optional/unneeded so then there would be no external requirements for
pmdajmx at all.)
But since Nathan mentioned that the mere installation of tools.jar would
be somehow problematic (as in getting the jar in place), isn't it then
even more of a problem for Parfait since Parfait isn't available for
some/most major distributions (e.g., Fedora / CentOS/EPEL / RHEL)? As a
Fedora user I expect that if PCP is to support Java monitoring, then all
I need to do is something along the lines "yum install ... ; cd ... ;
./Install". Or is the plan to embed Parfait in PCP so that the
parfait-agent.jar possibly coming with PCP would contain all of Parfait
and its dependencies?
>> No code changes is definitely good. Having to configure Parfait
>> agent for each JVM is perhaps not ideal but still reasonable.
>> However, could you please describe the configuration scheme you've
>> envisioned? Let's say I'm interested in java.lang and jboss.org
>> metrics, with pmdajmx I'd do:
>>
>> pcpjmxconnector.attrfilter = java.lang:*!jboss.org:*
>>
>> Also, how does the agent deal with "unknown" components, e.g.,
>> something I've developed in-house and it provides a thousand or two
>> metrics over JMX, will the agent be able to pick them all up
>> without any/much
>
> I think Nathan has in place a plan to automatically scan standard
> Java metrics. If the model of following what other agents like
> NewRelic do, I would bet there's a 'scanner' module that is looking
> for known patterns of standard frameworks/contains like JBoss, Tomcat
> etc, they all emit standard JMX namespaces, so I can envisage doing
> something similar, scanning for JMX patterns and auto-registering
> ones you find. This is similar to the pattern Parfait is already
> doing with the Java memory space, looking for a few patterns of JMX
> with some optionality for some differences in JVM memory
> configurations.
This sounds good - as said, nobody is going to manually maintain metrics
information for tens or even hundreds of thousands of metrics so
automation / dynamic approach is a must. However, I can't comment much
more as the agent current code int git looks pretty static.
Makes me actually wonder could we re-use the some pmdajmx code snippets
for this..
> If PCP is looking to create something that behaves like modern Java
> tracing facilities, the JVMTI mechanism is generally the pattern that
> NewRelic, AppDynamics etc follow.
Do we have any estimates on the needed effort for this, are we talking
about weeks or months?
Wrt others, for example AppDynamic lists quite a long list of supported
components in their web pages:
https://www.appdynamics.com/solutions/appdynamics-java-monitoring/free-java-monitoring-tools/
Do you think such a list could be covered with the 'scanner' approach
you mentioned?
> * I believe but could be wrong that Java processes do not expose JMX
> connector unless configured
Correct, Attach API enables/exposes it if not yet exposed.
> * Exposing JMX is a potential security risk
JMX and pmdajmx support authentication so it's up to user whether to
accept the risk of unauthenticated access or not.
> _immediately_ present in the MMV, an so if you need to sample through
> the local PMCD at a higher frequency for some metrics you can. This
> may be an advanced use case, but there's certainly circumstances when
> we've valued being able to look into metric values more frequently.
We talked about something like this on IRC with Frank I did quick
testing, with ~20 lines of code I was ably to "dynamically" to change
the update interval (parts of it was hard-coded so it's not there yet)
but I didn't pursue further as I also consider it somewhat advanced
case, not critical at this point.
> didn't really want to commit to trying that. We thought the
> Perl->Jolokia REST would be easy, but it was more annoying than we
> thought it was.
Yeah, I looked at Jolokia as well and I don't think it'd be a good fit
for PCP at least as a general solution. But seems that some real-world
products use it successfully (but I think they don't have such strict
requirements for response times as PCP PMDAs):
http://developers.redhat.com/blog/2016/03/30/jolokia-jvm-monitoring-in-openshift-2/
> I'l try and follow this conversation as much as I can, and contribute
> my thoughts when time permits.
Thanks, much appreciated. I think I need to clarify that I'm not in
principle against the Parfait agent approach (would be less work for
me!) but I'm concerned that it will take a very long time before we have
anything that'd be at the level of pmdajmx is already today and, perhaps
even more importantly, since there is nothing concrete available about
the planned user interface and usage in general, it's completely unclear
how it would look like for users? I failed to build it and the current
configuration looks static so it seems to be way too early to make any
conclusions at this point. Whatever the end result is, I think the user
interface would at least need to match the ease of pmdajmx configuration.
So while parts of the agent approach look good, it's somewhat
hypothetical at the moment.
Cheers,
--
Marko Myllynen
|