Hi -
> [ sar, argus, syslog, log4j, pcp ... examples of the more regular
> style of log rotation from the production environments I know - none
> ever write the entire contents of the file since recording begins,
> each and every day ]
What some of those have in common is that they include no suite of
processing tools for analyzing the rich data again. Sysadmins are
stuck with manual identification of the slices and/or plain
per-file/line text search tools. Contrast to fancier analysis tools
like splunk/chainsaw/journald, that integrate. We should grasp toward
the latter.
> [...] Conversely, there's been no acknowledgement that there are
> zero advantages to wun-beeg-log should libpcp be extended to support
> transparent multi-archives, as is planned. [...]
The defaults today are only for the state of the software today.
> [...]
> I must not have explained my concerns well. Let me try a different
> tack using a real (really real, not some hypothetical) production
> example from a real system serving a small-medium-size web presence.
>
> Within one data centre we'd typically use the centralised logging
> setup as laid out in the book. A logger server with roughly 20-30
> hosts being logged [...]
The scaling stories are interesting and compelling. But our pmmgr
defaults log exactly one host, local:. (pmmgr, like the other pcp
daemons, probably aren't even enabled by default!) The moment you're
talking about 20-30 hosts being logged, you've left the realm of
the defaults entirely.
No one has asserted that all defaults are appropriate for much larger
scales. On the contrary, I have been emphasizing the tautology that
people who run non-default setups will need to use non-default settings.
> > [...] She doesn't have to remember how to splice it together; to
> > play with multiple -a options or comma-separated -a suboptions; or
> > clumsy gui dialog boxes; or manual pmlogextract steps. It's
> > already in one convenient chunk for instant gratification. It's
> > as close to "grand-unified" as we can get to today.
> That's not the case - one could set up a daily task that creates a
> summary-style archive, taking the most important metrics from the
> daily logs each day, and producing one merged archive from them to
> summarise the last week or month. [...]
Yes, and this is not a counterexample! We don't provide such a daily
task, so you're in effect asking the new-to-PCP person to invent one.
> [...] My main fear here is for the data loss this scheme will
> induce. [...]
I hope I have cleared up tangible data-loss related worries in detail.
If there is data loss risk today, one'd be caused by the lack of
fsync()'s, which affects the whole suite. I'll work on that
immediately. And I'll work on adding more pmmgr log-management
options with different performance/utility tradeoffs (but all safe
from data loss).
- FChE
|