Re: [pcp] Debugging sigpipe in pmda

[Ken McDonell <kenj@xxxxxxxxxxxxxxxx>]

On 17/08/16 07:45, Jeff Hanson wrote:
> ...
> The behavior seems only to have started when I increased the number of
> systems from which
> metrics could be fetched.  In my two tests it took 20 minutes and then
> 30+ minutes to
> replicate with two pmval <cluster metric> running at default sample rate.

This makes sense.  More nodes => more work => longer delay if the work is done 
in the PDU handler of the PMDA.

> What then would be the preferred architecture?

Not sure if I have the latest cluster PMDA source (it is outside the core PCP 
tree), but ... I'd start here

        if (FD_ISSET(tc->fd, &readableFds)) {
                /* TODO: This should be made non-blocking so that failure
                 * to get a response from a single client doesnt cause the
                 * pmda to block and get killed
                 */
                cluster_client_read(tc, i);
                ...

and much more seriously, the PMDA needs a second thread with a timer loop, 
pulling all the cluster metric refreshing logic into this thread and then 
guarding all the pmdaCache*() calls with a local mutex.