Hi Frank,
Phew, this one is going on and on too. Onward...
----- Original Message -----
> ...
> No, pmmgr does not "change" archives in the sense of rewriting or
> modifying them. pmmgr's "pmlogmerge" option creates *new ones*. The
*nod* I understand what its doing, my choice of words in "change" was
poor, sorry. What I meant was at the end of that pmlogmerge every
single byte in the (new) wun beeg log is different to what was in the
file of the same name on the day before.
Same data at different offsets == different data to backup tools, cp,
whatever is reading it. This is very important - in fact, I can't
think of any example of a log-file-tool that behaves this way (chops
off the start and adds to the end of a data stream, then writes out the
entire data stream, every day for all of recorded history)? Always, in
the ones I can think of, they use rotation to a new file and only the
most recently written data is ever in-flight.
[ sar, argus, syslog, log4j, pcp ... examples of the more regular style
of log rotation from the production environments I know - none ever
write the entire contents of the file since recording begins, each and
every day ]
> This thread hardly acknowledged the fact that there is a trade-off
> being discussed here. The positive side is that merging makes it easy
Well, in my defence, that's not really true - my exact words were:
"There are advantages & disadvantages to this. The advantage is we
get a single archive (well, except for todays data) which means the
tools can directly operate on more than a single days worth of data.
And that is great!!!"
Count 'em - that's three exclamation marks! It really is great!!!
So, I understand full well this advantage and did acknowledge it,
honest! Conversely, there's been no acknowledgement that there
are zero advantages to wun-beeg-log should libpcp be extended to
support transparent multi-archives, as is planned. Zero. None.
Nada. Zip.
Anyway - I was convinced then, and am increasingly convinced, that
the disadvantages far, far outweigh that one advantage. Don't get
me wrong - initially I thought this was a great feature too! But
then the subtle issues started to dawn.
I must not have explained my concerns well. Let me try a different
tack using a real (really real, not some hypothetical) production
example from a real system serving a small-medium-size web presence.
Within one data centre we'd typically use the centralised logging
setup as laid out in the book. A logger server with roughly 20-30
hosts being logged close by it (same rack, or pair of racks), all
closely co-operating hosts so often need to be analysed together.
Each host would have a daily log size in the order of 100-200MB
depending on its function (well instrumented applications, so much
bigger logs there). Its critical performance data, losing it is bad,
bad news (KPI reports that customers see are driven by this, not to
mention regular problem analysis and fire-fighting) but at the same
time we cannot be putting more than a minimal load on all systems
under observation (including the logger host: resources are tight!).
So, at least twenty hosts * 100MB per day, often alot more. Merging
logs for 1 day of that size is not a problem, any host can do it (i.e.
no special memory/storage configuration needed). And the length of
time we retain data for has no impact on this, the default PCP setup.
Now, lets imagine the logger host begins using a wun beeeeg log model.
>100MB * 14 days == >1GB per host in steady state (so after the first
14 days, we need to manage merging this amount of data every day for
every host). The log merging for all 20 hosts happens around the same
time (it needs to, out of business hours), so while we're merging the
>20GB worth of data (i.e. read(2) >20GB into the page cache, write(2)
it out to a new file with todays data appended, but the earliest day
removed) - 20GB clean data, another 20GB dirty to the new file;
=> significant page cache pressure, significant read and write I/O;
=> significant impact on this system.
Now we have >20GB of dirty data to writeout. That data is all dirty
and it covers everything we have logged to date, if we lose it we are
toast. As an aside - yet another subtle problem - note here that our
merging efforts require *twice* as much ondisk space to hold all the
data for the temporary merge file (which becomes the new file in the
end) alongside the existing data - temporarily, but still need that
space.
Anyway, there's a big window in there where all of our data (and the
metadata about our data) is in-flux. We are at hugely increased risk
no matter how you slice and dice this (yes, can reduce the risk with
fsyncs, but its not entirely removed). TBH, its hard to conjure up
a worse I/O pattern than we have created for ourselves here. OTOH,
the other scheme is near-optimal - no merging at all if there's just
the one log, which is the typical case (ie no pmcd/pmlogger restarts
on that day), otherwise only small amounts of I/O of the latest data.
Now, extend - consider those folks who think "hey, a 2 weeks default is
not for me, I want to keep 2 months worth of data here". Extend again,
20 hosts? Bah, I need 120 hosts for this upcoming launch of a product.
So, the behaviour of the wun-beeg-log scheme gets worse and worse - in
every way I can think of. Storage, memory, potential data loss - we
even manage to burn more CPU time merging, every day, than we used to.
> for a PCP user to use the data. She doesn't have to remember how to
> splice it together; to play with multiple -a options or
> comma-separated -a suboptions; or clumsy gui dialog boxes; or manual
> pmlogextract steps. It's already in one convenient chunk for instant
> gratification. It's as close to "grand-unified" as we can get to today.
That's not the case - one could set up a daily task that creates a
summary-style archive, taking the most important metrics from the daily
logs each day, and producing one merged archive from them to summarise
the last week or month. Using existing tools - people do this sort of
thing. But, we have better plans for the future, lets keep attacking
those.
> I believe there is value in making convenient defaults new PCP users.
*nod*, certainly.
> It's the defaults of a new tool we're talking about, not changing the
> behaviors of the ones that established installations have gotten
> accustomed to (and are quite capable of overriding defaults). In that
> context, let's not be afraid to experiment with more sophisticated
> defaults in the future too, as long as they maximize new-user utility.
My main fear here is for the data loss this scheme will induce. But
there are many other fears too - both Ken and I listed several more.
So there's no fear of experimentation - its welcome and a wonderful
thing. But with that experimentation we need to reflect after each
experiment, and see what has worked well and what hasn't ... in this
case we need to change the defaults to provide an acceptable level
of data safety (amongst other things), and move on to addressing the
remaining challenges in novel and exciting ways.
Both Ken and I have now independently advised this is not a good default
setting, I think the case has been well made and is compelling - please
can you look into changing it for the next release? (I'm happy to have
the option still there, personally, though perhaps others are not? Good
for running experiments and for folks who understand the risks and trade
offs involved - but not switched on by default). Thanks!!
cheers.
--
Nathan
|