Hi!
PCP has very complete coverage for system and supporting applications /
infrastructure metrics (like containers, 389 Directory Server, KVM,
Oracle, PostgreSQL, etc.) but there are lots of places where Java
performance metrics would be essential to have in the mix as well.
To fill this gap for PCP, I've written PCP JMX PMDA, below is a quick
illustration what kind of metrics this PMDA plugin provides:
$ pminfo jmx | wc -l
274
$ pminfo jmx | grep -Ei '(garbagecollector|heap|memory)' | wc -l
150
$ pminfo jmx.threading
jmx.threading.threadallocatedmemorysupported
jmx.threading.peakthreadcount
jmx.threading.threadcputimeenabled
jmx.threading.totalstartedthreadcount
jmx.threading.threadcputimesupported
jmx.threading.currentthreadcputimesupported
jmx.threading.threadcount
jmx.threading.allthreadids
jmx.threading.threadcontentionmonitoringsupported
jmx.threading.currentthreadcputime
jmx.threading.threadcontentionmonitoringenabled
jmx.threading.synchronizerusagesupported
jmx.threading.currentthreadusertime
jmx.threading.daemonthreadcount
jmx.threading.objectmonitorusagesupported
jmx.threading.threadallocatedmemoryenabled
$ pminfo -dfm jmx.threading.threadcount
jmx.threading.threadcount PMID: 498.0.153
Data Type: 32-bit int InDom: 498.0 0x7c800000
Semantics: instant Units: count
inst [0 or "localhost:9010"] value 11
inst [1 or "localhost:9012"] value 14
$ pminfo -dfm jmx.runtime.systemproperties.sun_java_command
jmx.runtime.systemproperties.sun_java_command PMID: 498.0.106
Data Type: string InDom: 498.0 0x7c800000
Semantics: instant Units: none
inst [0 or "localhost:9010"] value "JavaTest"
inst [1 or "localhost:9012"] value "JavaTest"
$
Below are some design principles and implementation notes, let me know
what you think, could this be considered to merged to PCP?
The design goals were to have zero Java code changes for monitored
applications, preferably also zero configuration changes. Target was
both local Java applications and remote Java apps over the network.
Since there's a wide range of metrics available and use cases are wildly
different, the user should have full control on what metrics to collect,
if possible also from custom performance metrics providers. Lastly, the
implementation itself should also have minimum number of dependencies to
external libraries to keep it as portable as possible.
On the implementation side JMX is the standard API for management and
monitoring resources [1] so it was the natural choice to be used for
collection the metrics. Since we're talking Java/JMX, it's best to be
done from a Java application. I checked some existing alternatives but
they were either non-native / not available for many distributions (like
jmx4perl) or not actively maintained and/or somewhat limited (various
command line JMX utilities), especially when it comes to attribute
filtering and output formatting results. So writing a new one from
scratch specifically for PCP was IMHO warranted. This is now called
PCPJMXConnector.
OTOH, PCP does not have Java PMDA API so a counterpart component in a
supported language would be needed. I chose to use Perl as it's pretty
good in extraction and reporting and also because the Perl PMDA API is
slightly more complete than the Python PMDA API (e.g., pipe support is
available only with Perl).
More practically speaking, one the issues to take into consideration
from early on was the fact that it's a known fact that under extreme
load the JMX requests might take several seconds to complete - otherwise
no such a big deal but since PMCD is quite trigger happy with slowly
responding PMDAs the JMX PMDA must allow JMX requests to complete while
still keeping PMCD at bay (the worst case scenario to avoid is that when
under extreme load, perhaps due to application issues, either the PMDA
gets killed or is unable to collect metrics when they are most needed).
To achieve this, the Java side uses connection caching and threading to
be as efficient as possible and the Perl side reads the results from a
pipe (or a tailed file, if configured) as they come and only give them
to PMCD once a consistent set of fresh metrics are available, otherwise
providing instant replies with the most recent complete results.
The Perl implementation should be pretty easy to read for anyone
familiar with PCP PMDAs, the one thing worth noting is that it support
three different modes of communication with the Java application (with
the cost of only few extra lines): in-pmda piping (the default, works
well), libpcp piping (hopefully works after the yesterday's fix from
Ken), and tailing from a file (works ok, uses a local file which can be
later discarded/stored as needed).
The Java side outputs CSV - it would be easy to extend it to support
other formats like JSON as well (but I'm not sure how well it would work
with the JSON PMDA then, perhaps something that could be tested in the
future).
The Java side has two alternatives for collecting the metrics over JMX.
The first one is to use the Attach API [2] which will also automatically
start to management agent on the target JVM if needed to collect data
from MBeans [3]. This mode can be used to monitor any running JVM on the
local system, filtering for local JVMs is supported as well. Due to this
mode, one external requirement for the Java code is present, namely the
standard tools.jar which is part of Java SDK (included in the -devel
package on many distributions).
The second alternative is to use host:port type specifications to
monitor only a selected list of JVMs, local or remote. In this case the
JVM needs to be started with the management already enabled [4].
The way to define the metrics to be collected is fully dynamic, I've
tested the filtering configurations against the standard MBeans but any
custom MBean providing supported data types should work as well. The
standard JMX filtering [5] can be used but that might be a bit
inconvenient for some, thus there's a also a PCP JMX Connector specific
filtering alternative (which can even be combined with the standard JMX
filtering to certain extent):
# Question mark (?) separated list of beans. Beans may optionally be
# followed by a list attributes as a pipe (|) separated list. List mode
# lists matching attributes. Note that the attribute filter matches
# innermost attributes only, like the Vm* ones in the example below.
#
# Example:
java.lang:type=Runtime|VmName|VmVendor|VmVersion?java.lang:type=OperatingSystem
# Default:
java.lang:type=Memory?java.lang:type=Threading?java.lang:type=Runtime|sun.java.command
As the comment from the configuration file example states, the attribute
filters match only the innermost (I'd say the "real") attributes -
meaning that instead of fetching 50+ irrelevant Runtime/SystemProperties
attributes in a higher level container data structure only the
intesresting attributes can be selected.
There's also a list mode available so that one can examine what
attributes (metrics) are available and what the current filtering
configuration is providing.
The code is currently in my Fedora people area (no public git repo
available, sorry), as said the tools.jar is the only requirement so
using -cp /etc/alternatives/java_sdk/lib/tools.jar:. or such with
java/javac should be enough:
https://myllynen.fedorapeople.org/pcp-jmx/
This is 1.6 / 1.7 / 1.8 compatible, changes to support 1.6 and 1.7 were
reasonable and didn't affect the code much on the whole.
Lastly I think I could mention that I've seen some of the earlier PCP
list discussions about Parfait and Jolokia based instrumentation. They
are certainly interesting alternatives and might be wanted by some and
could be even more suitable than this JMX PMDA in some cases but since
the standard JMX metrics are enough for many, I think if those Parfait
and Jolokia alternatives materialize in the PCP upstream one day then we
can simply have different alternatives (PMDAs) to cover different use
cases. However, I'm currently interested only in this JMX PMDA.
1) https://docs.oracle.com/javase/8/docs/technotes/guides/jmx/index.html
2) https://docs.oracle.com/javase/8/docs/jdk/api/attach/spec/index.html
3) https://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
4) To start a JVM with remote JMX enable use something like:
java -Dcom.sun.management.jmxremote.port=9012
-Dcom.sun.management.jmxremote.rmi.port=9012
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false ...
5)
https://docs.oracle.com/javase/8/docs/api/javax/management/ObjectName.html
Thanks,
--
Marko Myllynen
|