Hi, Ken -
> More History 101 below ... 8^)>
Thanks!
> > I was talking more about representation on disk, as opposed to a
> > particular reading/caching algorithm.
>
> Hmm ... then I don't understand what "merge" means in the original
> suggestion. Can you provide a bit more detail of your thoughts?
(With respect to instance-domain updates, that each record could be
differential to the one in effect at the previous timestamp. I used
"merge" when perhaps I should have used "diff" and "patch" instead.)
> [...]
> > So why a separate [.meta] file?
>
> [...]
> Now, the only operating system I know (and love) that offers a file system
> that supports multiple growing logical segments within the one file
> container is the Michigan Terminal System (MTS) that was an OS/360
> alternative in the days before Unix escaped from Bell Labs.
Careful -- you might start dating yourself!
> So with the file systems we have to work with, the only option would
> be to have variant records in the one file and intermix the metadata
> and pmResult data. But the metadata includes the PMNS and the
> metric descriptors and if there was a single data stream all
> applications would have to read and process EVERY record in the
> archive before they could answer the simplest questions about the
> PMNS or the metric descriptors. [...]
Understood. That scanning EVERY record business could be beaten if
the PMNS/metric/indom data were located in convenient spots. For
example, if they were generally interleaved with pmResult data in time
order but also replicated at predictable places such as the end of the
file, and contained links (file offsets) to earlier metadata records,
then a complete metadata-only search could take
O(number-of-metadata-updates) rather than O(number-of-pmResults).
> To expedite random access into the archives we have an index, the .index
> file, and this also grows over the life of pmlogger. If the metadata was
> combined with the pmResult data then one needs to consider what to do with
> the index data ... putting it in the one file is a waste of time unless you
> have MTS and the ability to create a third growing segment in the file (if
> you need to read the file to extract the index, you don't need the index!),
> [...]
Again, it depends where the index is placed. If it's in predictable
locations, one can eschew the full scan.
> I still think this is the _right_ solution for Unix/Linux style file
> systems.
It's been a fine solution. I'm wondering whether the benefits we'd
get from a streamable format would be worth the moderate complexity.
> Of course if there was a free, fast, multi-platform, reliable DBMS
> at the time we designed the PCP archive format [...] we'd probably
> ended up with binary blobs for all the stuff that really mattered
> and the DB infrastructure would not have provided much leverage).
Right.
> > If we had the meta+metric data together, we could have instantly a
> > pipe/streaming format - something we've wanted for special PMDA
> > purposes, but could come in handy in other ways.
>
> What are the "special PMDA purposes"? [...]
Perhaps "PMDA purposes" was misleading, sorry. Some possibilities
include having tools such as systemtap, or live importers, or
summary/stats engines, generate & consume archive-format streams.
Perhaps it could turn out to feed a more workable version of that
earlier "tail -f archive" effort Max/Greg were talking about.
- FChE
|