Hi guys,
----- Original Message -----
> On 2016-04-12 04:53, Nathan Scott wrote:
> > ----- Original Message -----
> >
> >> With the JSON PMDA I think we can also have an acceptable compromise
> >> here (do it Right but still leave some rope on the floor for those who
> >> are tempted to shoot themselves in the foot or can't / don't want to
> >> generate complete metadata e.g. for in-house/proprietary components).
> >
> > +1
>
> Oh, well, I think I was being overly optimistic here.. [...]
Thanks for attempting it, sorry I didn't foresee those issues either.
> >> Since we already have data flowing thru pmdajmx.pl I'd assume it'd be
> >> easier to switch to the somewhat similar JSON PMDA than the MMV PMDA.
Hmm, that assumption is looking shaky now.
> > Yep - for MMV we'd need a small amount of new code to allow proxying too -
> > not tricky or difficult in any way,
FWIW this proxying code was implemented last week, so is in place for us if
we choose to go that route now.
> [...] So looks like
> we don't have perfect alternatives available at the moment (see also
> below).
We still have Parfait+MMV forging ahead nicely, and increasingly it seems to
me like a good platform to build on for "external" JMX monitoring too.
> One theoretical alternative would be to rewrite the current
> pmdajmx.pl as pmdajmx.py which would allow dynamic metrics but that
> would surely prevent reusing existing code so far from ideal.
Yep, plenty of downsides there - a python rewrite will solve some problems but
not others (still requires that Java helper, which is not something we want to
be merging in core PCP).
> (But would
> be most likely less effort than extending the Perl PMDA API to support
> dynamic metrics.)
Oh yes, definitely. And far more code than solving this using a purely Java
solution, using Parfait as our base.
> I think at this stage with lots of new ideas, requirements, and
> understanding gained it might be a good time to take hands off the
> keyboard for a while and collect a list requirements and nice-to-have
> features (and things we don't want) and then re-evaluate how to get there.
+1
OK, here's my thoughts on where things are at (firstly, from an end-users POV).
1. New->intermediate users, possibly no PCP expertise/time, wishing to get
things
running quickly. Users comfortable with the use of a -javaagent in their
specific environment (via command line / properties file)
- *most people are expected to fall into this category, needs to work well*
- pretty much code & testing complete at this point - ready for review/merge
into main parfait code base, I think; (needs review)
- needs documenting too, esp. as to build and use for first time users
- could still use the improved jar-build, as per TallPauls detailed email
(but, not clear on urgency there, since spring utils not in use anymore,
and maven-shading-plugin working fine so far)
[ Red Hat specific: needs some more RPM packaging work ]
2. Expert users, significant PCP expertise, looking for optimal instrumentation
efficiency and going beyond basic JMX for performance analysis needs (but of
course, getting JMX values "for free" too with their use of Parfait metrics)
- Parfait API, interfaces to Metrics API, Spring and other frameworks are all
done and readily available
- API should be updated to use freshly-accepted JSR-363 units interfaces
- API namespace should be switched over to io.pcp.parfait
- needs documenting as to how to switch existing applications namespace use
[ Red Hat specific: needs some more RPM packaging work, leveraging above ]
3. New->intermediate users, possibly no PCP expertise/time, wishing to try
things
out but unable to use the java-agent option (for various reasons). These
folk
are OK with having a permanently running additional Java daemon, polling
their
JMX values from one/more applications that start/end within the lifecycle of
the long-running daemon.
- tried and discarded perl pmdajmx + java helper (lots of reasons), pmdajson
+
java helper (different set of reasons)
- note: the long-running daemon does *not* have to be a PMDA itself
- also: the daemon is java code & will require unit tests, so not in pcp/qa
- we've not yet tried using Parfait to solve this, via a new Parfait daemon:
[ Looks like an excellent option at this stage: this would involve creating
a
new (e.g. "jmx-proxy") Parfait sub-project, using MMV shared memory
mappings
via existing Parfait code, just like parfait-agent. Retask several parts
of
existing JMXConnector.java code to complete the puzzle - provide
application
scanning, filtering, and connecting, optional tools.jar use, and so on.
Importantly, this way Java code is in git tree with Java build system
(maven)
and setup for Java tests (not pcp/qa) ]
[ Red Hat specific: leverages RPM packaging work done in #1 & #2 already ]
4. New->intermediate users, possibly no PCP expertise/time, wishing to try
things
out but unable to use a -javaagent and not comfortable with using
long-running
java daemons. These folk require the monitoring java code "on-demand".
- no existing solutions considered this case so far (thanks to Scott @netflix
for raising it recently)
- I think we could use Parfait to solve this simply, once we've implemented
the
jmx-proxy daemon as described above
[ This would involve creating a new command line interface that uses
jmx-proxy
in an on-demand fashion - so starting from the command line, attached to
the
requested java process, and run until stopped or monitored process exits.
As
above, generating MMV shared memory mappings for existing pmdammv to use.
In
fact, this may involve no new code - its just a special case of jmx-proxy?
ie
no scanning for java processes - just attached to a specified process? ]
[ Red Hat specific: leverages RPM packaging work done in #1, #2 & #3 already
]
Finally, from a developers POV (i.e. not user POV now) we will also want:
a. A mode for parfait-agent to discover and report JMX beans it doesn't yet
know
how to export correctly (counter/instant/discrete,
bytes/KB/MB/msec/nsec/usec,
help text, and so on)
b. Convenient reporting / tooling to aid conversion from unknown JMX -> valid
pcp
metric metadata being exported
c. A mode for jmx-proxy to do the same - sharing much code, surely - as (a).
Oh, and because someone asked me - #1 and #2 are "internal" solutions, and #3
and
#4 are "external" - this is just a made-up terminology to try to explain things
in
terms of how instrumentation is accessed (especially JMX, but not limited to
it).
Thoughts? The above is basically the plan I'm working to currently. #1 is
nearly
complete, #2 builds on that, #3 and #4 build further on that groundwork. Are
there
any scenarios / cases we know of that will not be covered by #1 - #4 above?
cheers.
--
Nathan
|