Hi guys,
----- Original Message -----
> [...]
> Oh, ok. I didn't realize that there was a distinction and I didn't know
> that libpcp already supported treating a collection of .0 - .N volumes
> as a single archive (didn't get that deep into the code). So part of
> what I thought we didn't support is already supported. Great!
:)
> >> PM_CONTEXT_ARCHIVE could be extended to support more than one archive
> >> volume.
> I wasn't actually proposing multiple explicit contexts within pmapi
> code, but rather extending a single PM_CONTEXT_ARCHIVE to be able to
> handle more than one archive.
The difficulty there is that pmNewContext(3) can take one path only via
its "name" parameter (hence Franks suggestions about quoted globbing and
using sub-directories - as these identify multiple archives via a single
string).
> > This is possible, but causes a potentially tragic amount I/O. What
> > would be desirable is to extend the libpcp code for PM_CONTEXT_ARCHIVE
> > handling to transparently jump from archive to archive, in much the
> > same way it already jumps from volume to volume, as the current "time"
> > changes.
> Seems doable.
The volume-skipping analogy is a good one I think. However, some new
kinds of problems arrive - such as having to deal with differences in
the metadata between archives, which cannot happen between volumes (as
they share a .meta file - e.g., metric pmDesc structures might change
in the new world order, name<->PMID mappings, hostnames - these things
do not happen in the multi-volume-archive case).
Other tricky aspects are the need/desire to have a single __pmContext
structure but multiple __pmArchCtl structures ... but this is getting
fairly low level, you'll have to ponder that aspect deeply soon enough
I guess.
You'll also want to think about the data structures you use to represent
all the archives, and the timespans they cover. Think about how pmchart
has a need to quickly position to anywhere within the time series and
start replaying near-immediately (any tool that supports -S/-O uses that
actually). That is done using the temporal index, today, of which there
is one per archive. There is no across-archives temporal index, yet, of
course.
So I guess pmNewContext will need to first identify all of the available
archives, build in-memory structures for each, and then build some extra
data structure that allows quick identification of archives that are near
to the given time position (see pmSetMode(3) for the PMAPI interface).
So, heh, have no doubt - its a really difficult series of problems that
you're tackling here, even just limiting things to archives-only.
> >> Streaming live data:
> >> [...]
I didn't follow the need to talk to pmlogger - what was the rationale
there? Why not treat all archives as potentially live? If we don't,
it precludes archive updates being written via distributed filesystems
(pmlogger is not necessarily local to the host doing archive replay).
With Kens pmlogger changes from a few months back, we should be in much
better shape there in terms of no longer observing partially-consistent
results in libpcp.
The auto-switch-to-live-mode you're talking about would be great (later)
- that is one of the cornerstones of the original unified context mode
RFC (so, that, and this ability to switch between archives that you're
looking into pretty much are "unified context" mode). The idea there is
to "stack" __pmContext structures (which can be live and/or archive) and
have a new __pmUnifyCtl structure (or some good name) - analogous to the
__pm{Arch,Host}Ctl structures anyway - that sits atop the stack and holds
current positions for sub-contexts.
It might be useful for you to think about stacking here too - so, having
sub-contexts for each individual archive, and one "special" __pmContext
structure at the top of the stack, providing the temporal indexing into
the others. The PMAPI clients do not need to know that the top of the
stack is "special", for the most part they access the context via handle
only (in this scheme, pmNewContext would return a handle identifying the
top of the stack, and from there we cascade down to individual archives
within libpcp).
Also, be aware that __pmContext, __pmArchCtl, __pmLogCtl and friends are
part of the ABI, so we need to be sensitive to any changes in these data
structures (extending at the end only, etc).
> > Lots of good ideas in there, but how about we leave this part until
> > later?
> Why wait to start developing ideas?
Don't wait, and don't limit your thinking - please continue to put all
your ideas out here. Otherwise we'll be restricting our (collective)
thinking to a fixed, limited implementation and may not see the bigger
picture / looming problems in the next stage(s).
Can't wait to see progress though - this is going to be a fantastic step
forward!
cheers.
--
Nathan
|