Hi Marko,
----- Original Message -----
> Hi,
>
> Sorry for the late reply, I've been busy with something else recently.
>
No problem at all.
> [...]
> As said, we can eliminate the Perl thread once the fix for the libpcp
> issue is available. So sounds like we're talking about N threads in 1
> Java helper (NÃ1) vs 1 additional thread in N JVMs (1ÃN).
Just to check we're on the same page here - at this point, it seems clear
to me we should be able to eliminate all of pmdajmx.pl too, right? (we'll
definitely want the JMX-connector Java code in some form but the perl code
I'm not seeing as a useful long term proposition - the problems its trying
to solve have already been solved in PCP via pmdammv &| pmdajson, right?).
> >> [...] the mere installation of tools.jar would
> If you think it's a show-stopper then I guess we can drop the tools.jar
> dependency or use it alternatively/optionally.
The latter sounds encouraging - any thoughts on how that'd be implemented?
As an alternative, it might be worth looking into the code that is needed
from tools.jar by the JMX-Connector, and seeing how involved/complex it is.
We might find its pretty straight forward to implement/carry ourselves.
I noticed tools.jar seems to be quite a large jar, I wonder if that's why
it's JDK-only and not part of the standard runtime. Probably there's lots
in there we don't use/need.
I'd naively expect the tools.jar classes used by the JMX connector to not
be too involved though (?), so maybe the above is an option - if so, we'd
not have to specify JDK vs JRE, which would be a good usability win.
> Ah, ok, so you will be packaging Parfait, that solves it then.
Oh absolutely. And also helping to maintain and develop it if that wasn't
clear earlier too. But, working with Java experts instead of us trying to
go it alone in core PCP - I don't expect we would ever want to include Java
code in the PCP git tree FWIW (again, see Vector/JavaScript/browser example
from earlier mail for a more practical model).
> > OK, so, good - it sounds like we're all agreeing there's no reason not
> > to make use of a -java-agent based approach at some level.
>
> It's certainly a valid approach for some cases, however I don't think we
> should make it the only available approach.
Yep, I agree - thanks for making me see the light there. I had hoped we
could do what NewRelic and others do (agent-only AIUI) but now I think you
are right - this option is more important than I'd realised.
> Ok, so you're basically saying a PCP-specific definition is a must-have
> for each and every metric to be used from any FOSS/proprietary/in-house
> component?
Yes, like every other PCP metric ever. The metadata is an important part
of the value PCP adds to the performance analysis process, and trying to
retrofit it later is full of pitholes.
> Ideally that'd be nice of course but I'm not sure who's going to scan
> and do this for all those hundreds of thousands of metrics.
Hmm, that number keeps going up every time I hear it. ;) Again, it is
"values" not "metrics". And the more I look the more I find management
info in JMX beans that is not related in any way to performance analysis
and that would probably not be appropriate for PCP to be exporting.
So we're collectively going to have to solve this, just like we have for
every other performance domain - there is no silver bullet. Fortunately,
some of it has already been done by the Parfait folk over the years, many
others we'll need to do with help from various other Java communities and
users I expect.
As mentioned in earlier mail, we'll need to write tools to help ourselves
- so code like your JMX-Connector code that can expand all beans, but then
compare 'em to what we know about (with a blacklist, probably) and report.
> Ok, so what's the plan on this front?
>
> >> Makes me actually wonder could we re-use the some pmdajmx code snippets
> >> for this..
> >
> > *nod*, I suspect so too Marko.
>
> FWIW, I made some trivial changes to PCPJMXConnector so that there's now
> a stub method for JSON output support so if anyone wants to try out
> PCPJMXConnector with the JSON PMDA it should be a bit easier to get
> started with that now (the Perl part of course still fully working).
>
I realized since last we spoke that my suggestion to use pmdajson might
not be the optimal path long term - instead maybe utilizing Parfait to
go directly to MMV in the JMXConnector code (i.e. JMXConnector starts
using Parfait, as a proxy though, instead of as a directly instrumented
program).
We could also do both (i.e. allow choice of MMV/JSON) if people want that.
I would recommend reading through pmdajson(1) and src/pmdas/JSON/README
to see if you think one or the other is better suited here. In the JSON
model, a JMXConnector process (child of pmdajson) would feed data into
pmdajson (like the pmdajmx.pl model now).
In the MMV model, a JMXConnector daemon (standalone daemon? child of
pmmgr perhaps? i.e. not a PMDA itself at all - more like a JMX proxy)
would generate MMV format instead of JSON, leveraging existing Parfait
code.
I'm not fussed either way - I suspect the latter will involve less code
in the end, but not sure. MMV is also supported on all platforms while
JSON is not. MMV is also more efficient. *shrug*, I'm not fussed - we
do need to see some actual real-world pmdajson users beyond systemtap,
and this could make a good one.
> Just to clarify, the options even worth mentioning are Parfait which
> uses JVMTI and pmdajmx which uses socket-based JMX? Or are there still
> some additional alternatives worth checking / discussing?
That's correct - those are the only two options for JMX AFAIK. There are
variations on the theme creeping into things now, like the above MMV idea
- that'd be Parfait code able to use either JVMTI or socket-based JMX all
from one code base.
> >> Do we have any estimates on the needed effort for this, are we talking
> >> about weeks or months?
> >
> > I'm 100% convinced parfait-agent is the right approach for us to take now,
>
> We need to remember the agent based approach might not possible in 100%
> of the use cases.
+1 - I'll amend that statement to "I'm 110% convinced Parfait is the right
approach for us to take now", offering both agent & socket-based mechanisms.
> > so it's my main development priority atm and I'm actively seeking helpers.
>
> To be honest it's been pretty hard even trying to follow the overall
> plan around this, let alone to help in some concrete way.
The plan has certainly been evolving as we explore the problem space - so the
plan keeps changing as we find new information / uncover issues. That's OK -
it's all very fluid but we're definitely closing in on the best solutions for
the long-term I think.
> However, one aspect I could do already (regardless of the status of the
> current code and such) is to check few real-world environments for the
> non-JMX metrics considered relevant there [...]
That would be great. The ones that I know of immediately are those already
being handled / bridged by Parfait, that the Parfait developers came across
over the years that they've been doing this:
- access to APIs designed for performance instrumentation like DropWizard
Metrics (see parfait-dropwizard)
- access to low-level instrumented JDBC driver metrics (see parfait-jdbc)
> com.custardsource.agentparfait:parfait-agent:jar:0.3.8-SNAPSHOT: The
> following artifacts could not be resolved:
> com.custardsource.parfait:dxm:jar:0.3.8-SNAPSHOT,
> com.custardsource.parfait:parfait-core:jar:0.3.8-SNAPSHOT,
> com.custardsource.parfait:parfait-pcp:jar:0.3.8-SNAPSHOT,
> com.custardsource.parfait:parfait-jmx:jar:0.3.8-SNAPSHOT,
> com.custardsource.parfait:parfait-spring:jar:0.3.8-SNAPSHOT: Could not
Oh! Sorry, I really need to document this for us non-Java-background folk.
Parfait is a maven project, and uses several sub-projects - this is a bit
like the way we have src/* in the PCP tree and some of those src SUBDIRS
have build dependencies on each other.
That build.sh is only for builds of the agent component & relies on a maven
build of the rest of Parfait beforehand. Strictly speaking, I should just
git-rm "build.sh" - its a one-line sh script that was just for convenience,
and to point others trying this out in the right direction - unsuccessfully!
Lukas pointed me toward this book the other day, it might help you here too:
https://books.sonatype.com/mvnref-book/reference/
Experienced Java developers don't need to be told this stuff though (eyes
were rolled at my build.sh :) - its bread-and-butter to them, alot like
"./configure && make" is to us - so I'd not worried about documenting it at
all just yet.
Anyway to resolve those jars you'll need to build from the top-level first,
much like for a PCP build you'd need to get src/libpcp in place first via a
top-level "make". With maven, that's a "mvn clean package install".
> So instead of doing ./Install for a Java PMDA it would be ./Install for
> the MMV PMDA? Or do you mean something else?
Something else - MMV is default-installed (enabled out-of-the-box for every
PCP install for many years now) & it's supported on every PCP platform. We
still have a little work to do to reach those goals I mentioned before, but
we are getting quite close now.
cheers.
--
Nathan
|