pcp
[Top] [All Lists]

Re: pmlogger -u questions

To: Ken McDonell <kenj@xxxxxxxxxxxxxxxx>
Subject: Re: pmlogger -u questions
From: Nathan Scott <nathans@xxxxxxxxxx>
Date: Mon, 14 Apr 2014 19:42:57 -0400 (EDT)
Cc: "Frank Ch. Eigler" <fche@xxxxxxxxxx>, pcp@xxxxxxxxxxx
Delivered-to: pcp@xxxxxxxxxxx
In-reply-to: <534C6531.6050502@xxxxxxxxxxxxxxxx>
References: <01e901cf56df$4ce97de0$e6bc79a0$@internode.on.net> <1665962954.4723287.1397437104781.JavaMail.zimbra@xxxxxxxxxx> <534B4330.1060008@xxxxxxxxxxxxxxxx> <y0meh104nvl.fsf@xxxxxxxx> <534C4FF4.5000304@xxxxxxxxxxxxxxxx> <20140414212551.GK14108@xxxxxxxxxx> <534C6531.6050502@xxxxxxxxxxxxxxxx>
Reply-to: Nathan Scott <nathans@xxxxxxxxxx>
Thread-index: cLJSwv2cAafSQrN8f3Ma9zF0sbVEeg==
Thread-topic: pmlogger -u questions

----- Original Message -----
> On 15/04/14 07:25, Frank Ch. Eigler wrote:
> > Hi -
> >
> > On Tue, Apr 15, 2014 at 07:15:32AM +1000, Ken McDonell wrote:
> >
> >> [...]  Can we agree that -u semantics and no stdio should be the
> >> default behaviour going forward? [...]
> >
> > That would make sense to me ...
> 
> Having actually made the change now, I like this even more ... in the
> simplest implementation it is a one line change in __pmLogNewFile() to add
> 
>      setvbuf(f, NULL, _IONBF, 0);
> 
> and then everyone who creates an archive (pmlogger, pmlogrewite,
> pmlogextract, pmlogreduce, ...) gets unconditional unbuffered I/O.
> 
> To get the logical record semantics I need a little more hackery as the
> trailer 4 bytes is currently done in a separate fwrite().  And the
> metadata writes are all fragmented, so I may need to do something
> smarter there.
> 

I concur with Frank here - definitely warrants some deep cost analysis -
it sounds risky WRT introducing performance regression.  What is the I/O
size we generally submit from pmlogger data writes, OOC?  This may be an
opportunity to improve too, by gathering larger consistent chunks before
submitting I/O... 

cheers.

--
Nathan

<Prev in Thread] Current Thread [Next in Thread>