Hi,
On 2016-04-11 08:02, Nathan Scott wrote:
> ----- Original Message -----
>
> Just to check we're on the same page here - at this point, it seems clear
> to me we should be able to eliminate all of pmdajmx.pl too, right? (we'll
> definitely want the JMX-connector Java code in some form but the perl code
> I'm not seeing as a useful long term proposition - the problems its trying
> to solve have already been solved in PCP via pmdammv &| pmdajson, right?).
Yes, there's no need to stick with the pmdajmx.pl, the only reason I
started with it was that I had some earlier custom PMDAs at hand which
parsed and registered metrics for similar type CSVs so it gave a quick
start on that side. But I now see a way to achieve almost the same also
with the JSON PMDA so we can jettison pmdajmx.pl.
>>>> [...] the mere installation of tools.jar would
>> If you think it's a show-stopper then I guess we can drop the tools.jar
>> dependency or use it alternatively/optionally.
>
> The latter sounds encouraging - any thoughts on how that'd be implemented?
By adding one extra if-statement :-) So now when compiling you'll still
need to provide tools.jar in classpath but thanks to lazy loading if
using direct connections to JVMs (instead of automatic Attach API based
connections) the classes from tools.jar are never used so no need for
tools.jar in classpath in that case.
The code in question is the exception handling section at the bottom of
the run() method of the JVMWorker class.
> As an alternative, it might be worth looking into the code that is needed
> from tools.jar by the JMX-Connector, and seeing how involved/complex it is.
> We might find its pretty straight forward to implement/carry ourselves.
No, there's platform-specific code involved so not feasible (but luckily
we don't have to worry about that).
> I noticed tools.jar seems to be quite a large jar, I wonder if that's why
> it's JDK-only and not part of the standard runtime. Probably there's lots
> in there we don't use/need.
>
> I'd naively expect the tools.jar classes used by the JMX connector to not
> be too involved though (?), so maybe the above is an option - if so, we'd
> not have to specify JDK vs JRE, which would be a good usability win.
To recap, what is used from tools.jar is the Attach API [1,2] which
provides means to automagically enable and connect JMX on local JVMs run
by the same user so it's a nice zero-config alternative. (And sounds
like starting with JDK 9 root should be able to connect to any JVM [3,4]
so the same-user requirement might go away in the future.)
1) http://docs.oracle.com/javase/8/docs/technotes/guides/attach/
2) https://blogs.oracle.com/CoreJavaTechTips/entry/the_attach_api
3) https://bugzilla.redhat.com/show_bug.cgi?id=1311638
4)
http://mail.openjdk.java.net/pipermail/serviceability-dev/2016-March/019108.html
>> Ok, so you're basically saying a PCP-specific definition is a must-have
>> for each and every metric to be used from any FOSS/proprietary/in-house
>> component?
>
> Yes, like every other PCP metric ever. The metadata is an important part
> of the value PCP adds to the performance analysis process, and trying to
> retrofit it later is full of pitholes.
With the JSON PMDA I think we can also have an acceptable compromise
here (do it Right but still leave some rope on the floor for those who
are tempted to shoot themselves in the foot or can't / don't want to
generate complete metadata e.g. for in-house/proprietary components).
>> Ideally that'd be nice of course but I'm not sure who's going to scan
>> and do this for all those hundreds of thousands of metrics.
>
> Hmm, that number keeps going up every time I hear it. ;) Again, it is
> "values" not "metrics". And the more I look the more I find management
> info in JMX beans that is not related in any way to performance analysis
> and that would probably not be appropriate for PCP to be exporting.
Yeah, can't think of much value with config/mgmt data exporting but OTOH
someone still needs to identify the relevant ones - below is one of the
first hits when searching around this subject, it shows that the list of
components supported by some well-known solutions is quite long so
there's lots of items in play:
https://www.appdynamics.com/solutions/appdynamics-java-monitoring/free-java-monitoring-tools/
> So we're collectively going to have to solve this, just like we have for
> every other performance domain - there is no silver bullet. Fortunately,
> some of it has already been done by the Parfait folk over the years, many
> others we'll need to do with help from various other Java communities and
> users I expect.
>
> As mentioned in earlier mail, we'll need to write tools to help ourselves
> - so code like your JMX-Connector code that can expand all beans, but then
> compare 'em to what we know about (with a blacklist, probably) and report.
I'm thinking that we should (of course) aim for as complete and precise
support in upstream as possible and in case there's no information
available err on the safe side (i.e., ignore/skip the rest). But since
there are in-house / proprietary components which we will never be aware
of, provide some basic help/tools for users for creating the needed
metadata files for them. For example, if using the JSON PMDA,
PCPJMXConnector could easily spit out basic metadata for JMX metric it
encountered which would then need to be updated to match reality. (And
in case the need is urgent or they want to take a risk of things going
wrong in the long term they can do it - but dealing with the fallout is
completely up to them, nothing upstream needs to be even aware of.)
> We could also do both (i.e. allow choice of MMV/JSON) if people want that.
> I would recommend reading through pmdajson(1) and src/pmdas/JSON/README
> to see if you think one or the other is better suited here. In the JSON
> model, a JMXConnector process (child of pmdajson) would feed data into
> pmdajson (like the pmdajmx.pl model now).
>
> In the MMV model, a JMXConnector daemon (standalone daemon? child of
> pmmgr perhaps? i.e. not a PMDA itself at all - more like a JMX proxy)
> would generate MMV format instead of JSON, leveraging existing Parfait
> code.
>
> I'm not fussed either way - I suspect the latter will involve less code
> in the end, but not sure. MMV is also supported on all platforms while
> JSON is not. MMV is also more efficient. *shrug*, I'm not fussed - we
> do need to see some actual real-world pmdajson users beyond systemtap,
> and this could make a good one.
Since we already have data flowing thru pmdajmx.pl I'd assume it'd be
easier to switch to the somewhat similar JSON PMDA than the MMV PMDA.
However, at some point it might indeed be worth checking out the
estimated effort for MMV support - if JSON support is in the range of
~20 lines of code and MMV would not be radically different then it'd be
insignificant cost for allowing additional flexibility for users.
>> However, one aspect I could do already (regardless of the status of the
>> current code and such) is to check few real-world environments for the
>> non-JMX metrics considered relevant there [...]
>
> That would be great. The ones that I know of immediately are those already
> being handled / bridged by Parfait, that the Parfait developers came across
> over the years that they've been doing this:
>
> - access to APIs designed for performance instrumentation like DropWizard
> Metrics (see parfait-dropwizard)
>
> - access to low-level instrumented JDBC driver metrics (see parfait-jdbc)
Yes, this would be definitely interesting. Asking around a bit,
something like the examples below might be good candidates as future
additions where feasible:
-EJB statistics (response times, call counts)
-Servlet statistics (response times, call counts)
-JDBC connection pools (usage percentages)
-J2C connection pools (usage percentages)
-Thread pools (usage percentages)
-Transactions (active transactions, total transactions vs. JVM uptime)
-Servlet sessions (live count, active count)
> Oh! Sorry, I really need to document this for us non-Java-background folk.
> Parfait is a maven project, and uses several sub-projects - this is a bit
> like the way we have src/* in the PCP tree and some of those src SUBDIRS
> have build dependencies on each other.
>
> That build.sh is only for builds of the agent component & relies on a maven
> build of the rest of Parfait beforehand. Strictly speaking, I should just
> git-rm "build.sh" - its a one-line sh script that was just for convenience,
> and to point others trying this out in the right direction - unsuccessfully!
Yeah nuking it is probably best for the best - I was looking at the
initial commit 276348b which looked like build documentation and wasn't
sure am I or the script expected to do something else.
> Anyway to resolve those jars you'll need to build from the top-level first,
> much like for a PCP build you'd need to get src/libpcp in place first via a
> top-level "make". With maven, that's a "mvn clean package install".
Thanks, now I have parfait-agent.jar built.
I see that it certainly leverages other Java components:
$ unzip -l parfait-agent.jar | grep -c class
5810
(With tools.jar dependency optional, PCPJMXConnector is 4 classes.)
>> So instead of doing ./Install for a Java PMDA it would be ./Install for
>> the MMV PMDA? Or do you mean something else?
>
> Something else - MMV is default-installed (enabled out-of-the-box for every
> PCP install for many years now) & it's supported on every PCP platform.
Ok.
Thanks,
--
Marko Myllynen
|