Re: [pcp] pmlogger_check stuck if host is down

[Rares Vernica <rvernica@xxxxxxxxx>]

Hi Nathan,

Thanks for your pointers. I looked more into this.

On Thu, Apr 28, 2016 at 8:03 PM, Nathan Scott <nathans@xxxxxxxxxx> wrote:

> ----- Original Message -----

> > [...]
> > If one of the remote hosts is down, pmlogger_check gets stuck on that host
> > and takes about 30 min to move on. I ran pmlogger_check with -VV and the
> > output looks like:
> >
> > [...]
> > > ps ax | grep pml
>
> (any pmprobe processes running OOC? Âthat grep would have excluded 'em, but
> I wonder if thats where the blockage is)

While pmllogger_check was being stuck on the host which is down, I run ps to check for pmprobe. I run ps every 3-4 seconds and here is the output:

# ps ax | grep bb-02
19974 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf -r -c -q -h bb-02 /tmp/pcp.SH9Zh8Psb/pmlogger
26891 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/numa
26903 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/numa
26906 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 pmprobe -h bb-02 -v origin.numa.routerload
# ps ax | grep bb-02
19974 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf -r -c -q -h bb-02 /tmp/pcp.SH9Zh8Psb/pmlogger
26921 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/numa-summary
26933 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/numa-summary
26936 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 pmprobe -h bb-02 -v origin.numa.migr.intr.total
# ps ax | grep bb-02
19974 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf -r -c -q -h bb-02 /tmp/pcp.SH9Zh8Psb/pmlogger
26951 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/xbow
26963 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/xbow
26966 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 pmprobe -h bb-02 -v xbow.nports
# ps ax | grep bb-02
19974 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf -r -c -q -h bb-02 /tmp/pcp.SH9Zh8Psb/pmlogger
26951 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/xbow
26963 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 /bin/sh /usr/libexec/pcp/bin/pmlogconf-setup -h bb-02 /var/lib/pcp/config/pmlogconf/sgi/xbow
26966 pts/0ÂÂÂ S+ÂÂÂÂ 0:00 pmprobe -h bb-02 -v xbow.nports

So, I can see that pmlogconf is making progress, but it is very slow. It takes more than 8 minutes to go through the groups of metrics for only one host. Is there a way to short-circuit it if the host is down and the remote pmcd did not respond after the first metric?

Thanks!

Rares