pcp
[Top] [All Lists]

pmcd restart race condition

To: pcp developers <pcp@xxxxxxxxxxx>
Subject: pmcd restart race condition
From: Martins Innus <minnus@xxxxxxxxxxx>
Date: Fri, 2 Oct 2015 13:04:58 -0400
Delivered-to: pcp@xxxxxxxxxxx
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0
Hi,
I'm seeing some sort of race condition that I haven't been able to track down. If you create a bunch of /var/lib/pcp/pmdas/<pmda>/.NeedInstall files and then restart pmcd, sometimes, but not always, all the pmda/pmcd logs are gone, and only the .prev versions remain:


[minnus@cpn-d13-27 pmcd]$ pwd
/var/log/pcp/pmcd
[minnus@cpn-d13-27 pmcd]$ ls -l
total 80
-rw-r--r-- 1 root root    85 Oct  2 12:54 gpfs.log.prev
-rw-r--r-- 1 root root    86 Oct  2 12:54 linux.log.prev
-rw-r--r-- 1 root root    87 Oct  2 12:54 logger.log.prev
-rw-r--r-- 1 root root   197 Oct  2 12:54 nfsclient.log.prev
-rw-r--r-- 1 root root   174 Oct  2 12:54 nvidia.log.prev
-rw-r--r-- 1 root root 39440 Oct  2 12:54 perfevent.log.prev
-rw-r--r-- 1 root root  2638 Oct  2 12:54 pmcd.log.prev
-rw-r--r-- 1 root root    85 Oct  2 12:54 proc.log.prev
-rw-r--r-- 1 root root    85 Oct  2 12:54 root.log.prev
-rw-r--r-- 1 root root    86 Oct  2 12:54 slurm.log.prev
-rw-r--r-- 1 root root    84 Oct  2 12:54 xfs.log.prev

A subsequent pmcd restart recreates the logs and all is well.

This is not always repeatable, but if anyone has any ideas ?? I'm not sure how long this has been going on.

As far as I can tell, when you have more than one .NeedInstall file, the pmcd restart process gets nested/recursive/happens multiple times or something. This is on Centos 7 and current git pcp.

Thanks

Martins

<Prev in Thread] Current Thread [Next in Thread>