----- Original Message -----
> The new code only fsyncs at archive file close time. That in turns
> only affects files that were written-to (because fsync of a read-only
> file is a noop). AFAIK, none of our tools append to existing
> archives, so I'm not sure what kind of multi-write-pass processing you
> are referring to.
Yes, none of the tools do this - twas a general API comment, that we
should not assume these things have never been done (from some other
API user) or will never be done (by either others or ourselves).
> > - most modern filesystems perform delayed allocation. If we
> > make an assumption about when an appropriate time to flush is,
> > we force the filesystems to allocate space early, and this can
> > result in less optimal allocation patterns (causing seeks) [...]
>
> I don't see how this applies. We only fsync at file close, after
> which there will be no further allocation.
There's no guarantee of that though, and an API should not force it
to be that way.
Client tools are free to open & close multiple times if it suits them
to do so, and it's preferable to offer the option of sync'ing once
(or not at all - this is not a one-size-fits-all situation).
> > [...] I think the tools are the right place to make these changes,
> > and possibly even only via new command line options, for use by the
> > daily log rotation regimes.
>
> Knobs to override it are justifiable (like for that non-tmpfs
> intermediate file case), but I suggest defaulting to greater
> data-safety rather than less. This is the same default used for
> modern text editors, git, and of course databases of all kinds:
> they all fsync on close by default.
These are not performance analysis tools, which one might be using
to study the effects of fsync (or disk access patterns in general,
that could be disrupted by increased sync activity).
> As to "for daily log rotation", are you suggesting that pmlogger's own
> fresh/original output (which has the lowest write volume/rate, thus
> the lowest cost for fsync's)
("thus the lowest cost" - this is a bit of an oversimplification
FWIW, and is not necessarily an accurate assertion - see below).
> should not be considered at least as
> precious as the log-merging postprocessors'?
Not saying one is more precious than the other, no, but there are
tradeoffs to be made and places where we simple cannot practically
insert fsync calls ... pmlogger may be sampling multiple times per
second, and it may well be doing so during critical points of an
analysis. Whereas the daily rotation happens in the middle of the
night - ie very infrequently and the time of day favours use of a
more widely impacting event.
So - we definitely cannot afford to fsync after every single log
write. This would make high frequency log writing impossible and/
or inaccurate, and it thwarts those filesystem optimisations I was
discussing (delayed allocation for optimal placement, etc).
cheers.
--
Nathan
|