On Fri, 2009-10-09 at 10:40 +1100, Nathan Scott wrote:
> Hi Ken,
>
> Probably you have a better handle on this stuff than me, so
> here's a couple of observations/questions on archive rotation.
>
> - should pmlogextract prefer the final archive in its choice
> of timezone, rather than the first? [ since daylight savings
> changes usually/always/sometimes (dunno what other geographies
> ae like, but I assume they're like .au?) happen early in the
> morning, and the "standard" (as per man pages) time for doing
> archive rotation and merging in just after midnight. ]
This is 100% no-win territory. It does not matter when you run the
daily script, the run that spans a timezone change is sort of doomed ...
if you use the timezone from the first archive the times are off _after_
the timezone change, if you use the timezone from the last archive the
times are off _before_ the timezone change.
I think the status quo is probably the best one could hope for and the
warnings are legitimate.
> - any thoughts on .0 (data) files being mysteriously removed?
> pretty sure there's nothing touching our production logs
> outside of pmlogger_check/daily ... something in there seems
> to be nuking (/not creating?) .0 files (though .meta & .index
> files exist - so I suspect its somehow being nuked. I've seen
> it a few times, out of the blue, randomly - haven't been able
> to see any pattern to it though.
In the output below is this all contiguous from one run? If so,
Oct 9 merging the previous day's archives complain about timezone
changes ... but Oct 8/9 is not a timezone change day for Oz ... ???
Now the missing archive is from a previously unmerged set of archives
from Oct 2 ... I have no explanation for this ... I can see how it might
be possible to have a missing file from an archive that is exactly
$CULLAFTER days old, but this does not match your observation.
I'd suggest using -t 7 to keep the last week's worth of verbose logs to
help diagnose this.
> cheers.
>
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.15.15 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.15.15 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogger_merge: Error: "volume 0" file missing for archive
> > "20091002.00.10"
> > pmlogger_merge: Error: "volume 0" file missing for archive
> > "20091002.05.45"
> > pmlogger_merge: Error: "volume 0" file missing for archive
> > "20091002.06.00"
> > pmlogger_merge: Error: "volume 0" file missing for archive
> > "20091002.13.15"
> > pmlogger_merge: These error(s) are fatal, no output archive has been
> > created.
> > pmlogger_daily: Error: problems executing pmlogger_merge for host
> > "db2"
> > [/var/lib/pcp/config/pmlogger/control:36] ... logging for host "db2"
> > unchanged
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.06.50 timezone: EST-11 [will be ignored]
> > pmlogextract: Warning: timezone mismatch for input archives
> > archive: 20091008.00.10 timezone: EST-10 [will be used]
> > archive: 20091008.07.00 timezone: EST-11 [will be ignored]
>
|