PCP does not currently provide any tool to temporally compress archives;
temporal compression involves taking the raw data and generating
an equivalent archive with longer average sampling intervals.
Note that "equivalent" is a rubbery term here, as this sort of compression
inevitably implies loss of information; what we mean is that if the raw
archive is compressed using a sample interval of say, 3 hrs, then when
you replay with a sample interval of 3 hrs or longer, the raw archive
and the compressed archive will produce the same results.
We've tended to use compress/gzip to achieve approximately 6:1 statistical
compression, but this only solves the disk space issue, not the time
issue of processing long archives versus the smaller summary archives.
I believe temporal compression would be a useful addition, and
we've contemplated it in the past (pmlogextract was going to do this
originally), but never had the resources to convert the designs into code.
The tricky issues include:
- pmlogger can collect different groups of metrics with different
sampling frequencies in the same archive
- pmlc may be used to change the set of metrics being collected
and/or the sampling frequency for metrics and/or the instances
being logged ... all on the fly while pmlogger is running
- gaps in the archive caused by pmcd being restarted (when
pmlogextract merges the archives, mark records are added and
temporal compression needs to be very careful in this region
- counter overflows, and precision promotion
- semantics for discrete and instantaneous metrics
If anyone competent C developer using PCP would like to learn more about
PCP archives and would like to develop a PCP tool that would be useful
for people like Mark, please get in touch with me and I can provide you
with additional information to help get you started.
---------- Forwarded message ----------
Date: Fri, 12 Jan 2001 14:26:11 -0800
From: MARK.BONN@xxxxxxxxxxxxxxx
To: ptg@xxxxxxxxxxxxxxxxx
Subject: Sprint Customer: PCP Archive Question
Questions:
1. The PCP manual eludes to the ability to compress pmlogger archives
into summary archives. However, the manual provides no cook-book how to
compress the logs. For example, If I have an archive with a metric
collection interval of 5 minutes, is there a way to create a new archive
with all the counters rolled up into hourly counters or daily counters.
I realize that I can use pmchart with a longer interval to produce the
same end result, however to create a long term capacity planning
database, I would like to summarize the data. Any thoughts, or do I
just compress the data for long term archiving?
Thanks,
Mark Bonn
Systems Management
Phone: 916.636.6906
Pager: Pin 383-0808
Fax: 916.636.6232
Email: Mark.Bonn@xxxxxxxxxxxxxxx
|