Hi,
Sorry for the late reply, I've been busy with something else recently.
On 2016-04-02 03:32, Nathan Scott wrote:
> ----- Original Message -----
>
>>>> Yes, I think none of the other PMDAs come close to JSON/JMX PMDAs
>>>> in terms of their flexibility and generic nature.
>
> Hmm, well, I would consider the MMV PMDA to go beyond both in terms of
> being flexible and generic. Its light years ahead on the performance
> curve and has the many-years-of-actual-production-use factor too.
JMX PMDA is of course about JMX but not sure how the MMV PMDA is more
flexible/generic than the JSON PMDA?
> The point was, there are threads *everywhere* in one approach - pmdajmx
> perl code is multi-threaded, the java helper is multi-threaded, and the
> java apps themselves are always multi-threaded (for anything beyond the
> basic "hello world" kind of java app). In the other approach, we piggy
> back in via an additional thread in the (already-threaded) java app but
> nothing more than that (no new PCP code, and MMV doesn't need threads).
As said, we can eliminate the Perl thread once the fix for the libpcp
issue is available. So sounds like we're talking about N threads in 1
Java helper (NÃ1) vs 1 additional thread in N JVMs (1ÃN).
Also remember that since the beginning PCPJMXConnector has had on its
TODO list "add JSON output support" in case it would be preferable to
use PMDA JSON instead of the Perl part - I've now done some minor
initial changes to that direction (see below).
>>> pmdammv <-> { parfait-agent<->java app}, {parfait-agent<->java app},
>>
>> Ok, thanks for the clarification. Do we already know how this would look
>> like with multiple Java apps?
>
> Yes, the ASCII art from Paul is showing multiple Java apps. If we consider
> the "<->" to be IPC (shared memory for pmdammv, sockets/pipes for pmdajmx),
> and "+" to show things being in-one-process, it's:
>
> pmcd+pmdammv.so <-> {java-app+parfait-agent},{java-app+parfait-agent},...
>
> (The Programmers Guide is where this is hiding, and I'm sure it could be
> further improved. That would help all programmers, for all languages,
> not only Java.)
I meant the end-user configuration aspect, I can't spot such
documentation even from The Programmers Guide, Java or non-Java.
>> [...] the mere installation of tools.jar would
>> be somehow problematic (as in getting the jar in place)
>
> This is more an issue about not forcing people to install the full JDK, when
> the JRE would do & perhaps is what they use today - telling prodops folk they
> must now install the JDK as well - and e.g. into a minimal app container - so
> that they can monitor an app? Its a problem, not to be lightly dismissed IMO.
If you think it's a show-stopper then I guess we can drop the tools.jar
dependency or use it alternatively/optionally.
> OTOH, providing a package with a standalone parfait-agent jar in it ... works
> for both JRE and JDK installs, has no dependency chain (not even on PCP), and
> is easy to fit into that hypothetical app's container build.
But it's not always straightforward or perhaps even possible to
introduce such an agent either.
>> even more of a problem for Parfait since Parfait isn't available for
>> some/most major distributions (e.g., Fedora / CentOS/EPEL / RHEL)?
>
> Hmm, this logic doesn't really make sense to me - we will be packaging either
> one or the other or both - pcp-pmda-jmx (pulling in a tools.jar & jdk rpm dep
> & perl dep & pcp-libs dep) and/or a pcp-parfait noarch Java-only package (with
> no deps).
Ah, ok, so you will be packaging Parfait, that solves it then.
>> Again, this is of course a tempting approach if you merely look at
>> pcp.git but who's going to fix the issues users will eventually find
>> out? Users don't care whether it's a Spring issue or an MMV PMDA issue
>> or something in between, they either get the metrics from PCP or not.
>
> I was thinking of "supported" more in the PCP upstream developer sense not
> the Linux-distribution / Red-Hat-as-a-company / any-other-end-user sense.
>
> That involves pcp/qa test writing, running, maintaining for a rather vast
> matrix of java versions & having PCP maintainers fixing QA failures there.
> It involves different build/install toolchains, different static analysis
> tools, different ... well, different everything really. Different people
> with different backgrounds and skill sets, too, for the most part.
>
> This model has been shown to work extremely well with Vector. The PCP /
> Vector relationship makes for a good analogy - theres different languages
> (javascript), different target platforms (browsers), different developers
> (working together when needed though), different build/test/release model
> and so on.
Ok, I was thinking more of the downstream perspective: at least for the
time being Red Hat does not officially support PCP + Vector (since the
related packages live in the Optional channel which is unsupported [1]).
1) https://access.redhat.com/articles/1150793
>>>> No code changes is definitely good. Having to configure Parfait
>>>> agent for each JVM is perhaps not ideal but still reasonable.
>
> By "configure Parfait agent" we're talking about "-java-agent=parfait"
> being added to the java command line, right? Or to a java properties
> file - both are widely-used well-understood ways of doing things.
>
> OK, so, good - it sounds like we're all agreeing there's no reason not
> to make use of a -java-agent based approach at some level.
It's certainly a valid approach for some cases, however I don't think we
should make it the only available approach.
>>>> However, could you please describe the configuration scheme [...]
>>>> how does the agent deal with "unknown" components, e.g.,
>
> Oh, I need to make this clearer I think. IMO pmdajmx doesn't deal with
> "unknown" components in a viable way. There is no viable approach for
> automating this at PMDA runtime with no involvement from a human, in
> practice. The information needed simply doesn't exist from JMX, and
> it requires domain knowledge to provide it.
>
> We cannot simply make up PCP metadata about JMX values for which we have
> no idea what the metric is. Saying "great, we have 15000 new metrics!"
> but having no idea which are counters, which ones are measures of time,
> what the time and/or size units are - this is a big maintenance problem
> (from historical experience, not just IMO).
Ok, so you're basically saying a PCP-specific definition is a must-have
for each and every metric to be used from any FOSS/proprietary/in-house
component?
>> So far I've tested OpenJDK JVM 1.6/1.7/1.8 and IBM JVM 1.7.1/1.8 and I
>> didn't see any differences there. Do you have concrete examples or was
>> this a hypothetical scenario?
>
> This referred to the case in the existing metrics in the parfait-agent
> code where different (mutually exclusive) garbage collection algorithms
> present the same metric(s) with differently named JMX values - these can
> be reduced to a single metric in the Parfait+MMV world.
>
> Its one of many possible examples though - scanning through the first few
> hundred lines of that Cassandra sample JMX data showed this pattern to be
> pervasive - many individual values there would more ideally be modeled as
> instances of one metric for example.
Ideally that'd be nice of course but I'm not sure who's going to scan
and do this for all those hundreds of thousands of metrics.
>> Mandating so would just limit the usability of PCP on this front and
>> drive potential users away.
>
> Providing correct PCP metric metadata is very important, and it is not
> something we'll be discarding because its difficult or inconvenient. I
> think we can solve this perceived usability limit though.
Ok, so what's the plan on this front?
>> Makes me actually wonder could we re-use the some pmdajmx code snippets
>> for this..
>
> *nod*, I suspect so too Marko.
FWIW, I made some trivial changes to PCPJMXConnector so that there's now
a stub method for JSON output support so if anyone wants to try out
PCPJMXConnector with the JSON PMDA it should be a bit easier to get
started with that now (the Perl part of course still fully working).
>>> If PCP is looking to create something that behaves like modern Java
>>> tracing facilities, the JVMTI mechanism is generally the pattern [...]
>
> *nod*.
Just to clarify, the options even worth mentioning are Parfait which
uses JVMTI and pmdajmx which uses socket-based JMX? Or are there still
some additional alternatives worth checking / discussing?
>> Do we have any estimates on the needed effort for this, are we talking
>> about weeks or months?
>
> I'm 100% convinced parfait-agent is the right approach for us to take now,
We need to remember the agent based approach might not possible in 100%
of the use cases.
> so it's my main development priority atm and I'm actively seeking helpers.
To be honest it's been pretty hard even trying to follow the overall
plan around this, let alone to help in some concrete way.
However, one aspect I could do already (regardless of the status of the
current code and such) is to check few real-world environments for the
non-JMX metrics considered relevant there (obviously, if you have
something planned here, please do share - I could then check how those
plans sound like for those involved with the aforementioned environments).
>> even more importantly, since there is nothing concrete available about
>> the planned user interface and usage in general,
>
> Sorry, here's what I intended the parfait-agent code to express so far:
>
> This initial parfait-agent uses Spring configuration (XML) to tackle the
> "specify-metrics-correctly-yet-dynamically-too" aspect. We'll ship a jar
> with the pre-classified set, and add more metrics and configs over time as
> the metric classification process proceeds. There's also ways to pass in
> command line properties and/or system definitions we can use to manually
> configure parts of the agent, should that be needed. The early code shows
> the use of both JMX and non-JMX metrics as well. Finally, the code builds
> a single standalone jar with no external runtime dependencies.
>
> We could certainly add a way for people to pass configs to the -java-agent
> to add in more metrics without building the agent themselves ... there are
> lots of options and possibilities.
I'm sure there is.. but at least I'm currently in the "show me the code"
mode (having done so myself with pmdajmx already).
And by code I mean something that easily allows me to do "pminfo -f
java" or such to list at least all the JVM JMX metrics.
>> I failed to build it [...]
>
> Oh OK - what was the build failure? Which platform? Which version of Java?
> And how was the build invoked? Could you fpaste all of that, or open a new
> parfait github issue & I'll take a look (though Paul will probably spot the
> problem more quickly that me).
RHEL 7 + OpenJDK 1.8 using the build.sh script:
$ git clone git://git.pcp.io/nathans/parfait.git parfait.git
$ cd parfait.git/parfait-agent
$ sh build.sh
After lots of initial downloading I see:
[ERROR] Failed to execute goal on project parfait-agent: Could not
resolve dependencies for project
com.custardsource.agentparfait:parfait-agent:jar:0.3.8-SNAPSHOT: The
following artifacts could not be resolved:
com.custardsource.parfait:dxm:jar:0.3.8-SNAPSHOT,
com.custardsource.parfait:parfait-core:jar:0.3.8-SNAPSHOT,
com.custardsource.parfait:parfait-pcp:jar:0.3.8-SNAPSHOT,
com.custardsource.parfait:parfait-jmx:jar:0.3.8-SNAPSHOT,
com.custardsource.parfait:parfait-spring:jar:0.3.8-SNAPSHOT: Could not
find artifact com.custardsource.parfait:dxm:jar:0.3.8-SNAPSHOT in
sonatype-nexus-snapshots
(http://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]
On the second run I see warnings like:
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 46,
column 15
(maven-compiler-plugin etc. have been installed with Yum earlier.)
>> [...] Whatever the end result is, I think the user
>> interface would at least need to match the ease of pmdajmx configuration.
>
> From an end user point of view, I'd like to see this work "out of the box".
> i.e. install PCP, install parfait-agent - start a java app with -java-agent
> option. Then metrics are immediately visible live, and logging auto-starts
> if pmlogger has been enabled.
>
> No new PMDA, no ./Install, no perl/jdk-installation-needed-for-java-metrics,
> no external jar dependencies, & all the classified JMX (and non-JMX) metrics
> available as soon as the java process starts.
So instead of doing ./Install for a Java PMDA it would be ./Install for
the MMV PMDA? Or do you mean something else?
Thanks,
--
Marko Myllynen
|