----- Original Message -----
> ...
> Yeah, I know that the MMV dso is pretty simple. My problem is that I
> have
> over 1200 values in this stats structure. There's one global one, and
> another in TLS. Each thread updates the TLS struct, and every so often
> (e.g., when an RPC completes) it grabs a lock and adds in all the TLS
> values. Then the lock gets dropped, and the TLS counters get zeroed.
>
Interesting stuff. For reference, the Aconex application also uses MMV
(with client side library implemented in Java), and also has quite alot
of thread-local values (many othres too though). Its in the order of
several hundred metrics and several thousand values, for the core Java
application.
So, interesting ... and alot of the issues you have are similar. There
certainly seems to be scope to extend the MMV services to address some
of the common short-comings (more a lack of infrastructure I guess, a
simplistic initial implementation which doesn't really consider thread
instrumentation, and forces the caller to do a fair bit of legwork to
make using it inexpensive).
> Now, if I enable PCP support in the app, after adding in the stats, it
> updates the MMV values as well. At first, I just did a lookup for
> every
> value update, but that had *significant* performance impact. So, as a
No doubt.
> quick
> work-around, I made an array of <char*,char*,mmv_disk_value_t*>. At
> startup, I fill in the metric name and instance names (the char
> pointers) and the pointer to the MMV value.
*nod*, sounds better.
> Then I sort them. Then, at update, I
> just bsearch to find the right one. This helped a *lot*, but it still
> sucks
> big time. (Not to mention that the code is kinda weird looking.) So, I
> have ~1200 values, and they get updated anywhere from nothing to >1000
Ok.
> times
> a second. I do realize that having a global structs for the stats is
> not a
> very good idea, but it works pretty well if it's just 1200 adds. Once
> you throw in a bunch of strcmps, things slow down.
For sure, yeah, don't do that. :) In hindsight, possibly a mistake to
expose all the wrappers around the core mmv_inc_value / mmv_set_value
routines that allow string-based metric/instance lookup during update.
So you ended up with code that does initial lookups, caches the handle
(mmap file offset) for each value, and then does the updating and the
coherency up above the API - thats pretty much where we've ended up as
well (although the JVM has plenty of its own issues as well).
> It really makes sense that there'd be a slowdown...
Yep, absolutely.
> 1000 ops/sec, 1200 values/op, ~11 cmps/value == ~13 million cmps/sec.
> (Yes, I could skip over some of the value updates since adding zero
> is kinda useless.)
>
> At some point soon, I'd like to also expose some event-type metrics.
> MMV, as far as I can tell, can't handle those at all.
Thats right (predates all that stuff).
All food for thought on thinking about future versions of the app level
instrumentation services - thanks!
cheers.
--
Nathan
|