On Wed, May 30, 2007 at 10:20:26AM +1000, Nathan Scott wrote:
>
> We run a somewhat similar setup (multiple hosts monitored, RHEL4),
> and I've never hit this issue. Can you post your pmie control file
> and 'ps -ef | grep pmie' (preferably when multiple pmie instances
> incorrectly running)?
I stopped and restarted all pmie's yesterday, and now today we have
11 duplicate instances. Here's my control-file:
---------------------------------------------------------
$ cat control |grep -v ^#|grep -v ^$
$version=1.0
bpf.mydomain.com n PCP_LOG_DIR/pmie/bpf.mydomain.com/pmie.log
-c config.mydomain
mail1.mydomain.com n PCP_LOG_DIR/pmie/mail1.mydomain.com/pmie.log
-c config.mydomain
ntp1isp.mydomain.com n PCP_LOG_DIR/pmie/ntp1isp.mydomain.com/pmie.log
-c config.mydomain
transit.mydomain.com n PCP_LOG_DIR/pmie/transit.mydomain.com/pmie.log
-c config.transit
provdbs1.mydomain.com n PCP_LOG_DIR/pmie/provdbs1.mydomain.com/pmie.log
-c config.mydomain
provdbm1.mydomain.com n PCP_LOG_DIR/pmie/provdbm1.mydomain.com/pmie.log
-c config.mydomain
stl1.mydomain.com n PCP_LOG_DIR/pmie/stl1.mydomain.com/pmie.log
-c config.mydomain
maildbm1.mydomain.com n PCP_LOG_DIR/pmie/maildbm1.mydomain.com/pmie.log
-c config.mydomain
porting.mydomain.com n PCP_LOG_DIR/pmie/porting.mydomain.com/pmie.log
-c config.mydomain
billing1.mydomain.com n PCP_LOG_DIR/pmie/billing1.mydomain.com/pmie.log
-c config.mydomain
portal1.mydomain.com n PCP_LOG_DIR/pmie/portal1.mydomain.com/pmie.log
-c config.mydomain
dhcp2voip.mydomain.com n
PCP_LOG_DIR/pmie/dhcp2voip.mydomain.com/pmie.log -c config.mydomain
dhcp2tv.mydomain.com n PCP_LOG_DIR/pmie/dhcp2tv.mydomain.com/pmie.log
-c config.mydomain
dhcp2isp.mydomain.com n PCP_LOG_DIR/pmie/dhcp2isp.mydomain.com/pmie.log
-c config.mydomain
dhcp1voip.mydomain.com n
PCP_LOG_DIR/pmie/dhcp1voip.mydomain.com/pmie.log -c config.mydomain
dhcp1tv.mydomain.com n PCP_LOG_DIR/pmie/dhcp1tv.mydomain.com/pmie.log
-c config.mydomain
dhcp1isp.mydomain.com n PCP_LOG_DIR/pmie/dhcp1isp.mydomain.com/pmie.log
-c config.mydomain
m1dhcp1.mydomain.com n PCP_LOG_DIR/pmie/m1dhcp1.mydomain.com/pmie.log
-c config.mydomain
m1dhcp2.mydomain.com n PCP_LOG_DIR/pmie/m1dhcp2.mydomain.com/pmie.log
-c config.mydomain
m2dhcp1.mydomain.com n PCP_LOG_DIR/pmie/m2dhcp1.mydomain.com/pmie.log
-c config.mydomain
m2dhcp2.mydomain.com n PCP_LOG_DIR/pmie/m2dhcp2.mydomain.com/pmie.log
-c config.mydomain
http1.mydomain.com n PCP_LOG_DIR/pmie/http1.mydomain.com/pmie.log
-c config.mydomain
ns1.mydomain.com n PCP_LOG_DIR/pmie/ns1.mydomain.com/pmie.log
-c config.mydomain
ns2.mydomain.com n PCP_LOG_DIR/pmie/ns2.mydomain.com/pmie.log
-c config.mydomain
ldapm1.mydomain.com n PCP_LOG_DIR/pmie/ldapm1.mydomain.com/pmie.log
-c config.mydomain
ldapm2.mydomain.com n PCP_LOG_DIR/pmie/ldapm2.mydomain.com/pmie.log
-c config.mydomain
tvservices.mydomain.com n
PCP_LOG_DIR/pmie/tvservices.mydomain.com/pmie.log -c config.mydomain
prov1.mydomain.com n PCP_LOG_DIR/pmie/prov1.mydomain.com/pmie.log
-c config.mydomain
ztc1.mydomain.com n PCP_LOG_DIR/pmie/ztc1.mydomain.com/pmie.log
-c config.mydomain
ztc2.mydomain.com n PCP_LOG_DIR/pmie/ztc2.mydomain.com/pmie.log
-c config.mydomain
emergency.mydomain.com n
PCP_LOG_DIR/pmie/emergency.mydomain.com/pmie.log -c config.mydomain
hermes.mydomain.com n PCP_LOG_DIR/pmie/hermes.mydomain.com/pmie.log
-c config.mydomain
log01.mydomain.com n PCP_LOG_DIR/pmie/log01.mydomain.com/pmie.log
-c config.mydomain
atmail1.mydomain.com n PCP_LOG_DIR/pmie/atmail1.mydomain.com/pmie.log
-c config.mydomain
atmail2.mydomain.com n PCP_LOG_DIR/pmie/atmail2.mydomain.com/pmie.log
-c config.mydomain
smtp1.mydomain.com n PCP_LOG_DIR/pmie/smtp1.mydomain.com/pmie.log
-c config.mydomain
smtp2.mydomain.com n PCP_LOG_DIR/pmie/smtp2.mydomain.com/pmie.log
-c config.mydomain
maildb2.mydomain.com n PCP_LOG_DIR/pmie/maildb2.mydomain.com/pmie.log
-c config.mydomain
mw1.mydomain.com n PCP_LOG_DIR/pmie/mw1.mydomain.com/pmie.log
-c config.mydomain
asavdb1.mydomain.com n PCP_LOG_DIR/pmie/asavdb1.mydomain.com/pmie.log
-c config.mydomain
asav1.mydomain.com n PCP_LOG_DIR/pmie/asav1.mydomain.com/pmie.log
-c config.mydomain
asav2.mydomain.com n PCP_LOG_DIR/pmie/asav2.mydomain.com/pmie.log
-c config.mydomain
asav3.mydomain.com n PCP_LOG_DIR/pmie/asav3.mydomain.com/pmie.log
-c config.mydomain
asav4.mydomain.com n PCP_LOG_DIR/pmie/asav4.mydomain.com/pmie.log
-c config.mydomain
asav5.mydomain.com n PCP_LOG_DIR/pmie/asav5.mydomain.com/pmie.log
-c config.mydomain
asav6.mydomain.com n PCP_LOG_DIR/pmie/asav6.mydomain.com/pmie.log
-c config.mydomain
mobileprov.mydomain.com n
PCP_LOG_DIR/pmie/mobileprov.mydomain.com/pmie.log -c config.mydomain
wiki.mydomain.com n PCP_LOG_DIR/pmie/wiki.mydomain.com/pmie.log
-c config.mydomain
---------------------------------------------------------
And a 'ps -ef | grep pmie':
---------------------------------------------------------
$ ps -ef|grep pmie
root 29365 1 0 May29 ? 00:00:00 pmie -b -h bpf.mydomain.com -l
/var/log/pcp/pmie/bpf.mydomain.com/pmie.log -c config.mydomain
root 29421 1 0 May29 ? 00:00:01 pmie -b -h mail1.mydomain.com
-l /var/log/pcp/pmie/mail1.mydomain.com/pmie.log -c config.mydomain
root 29485 1 0 May29 ? 00:00:22 pmie -b -h ntp1isp.mydomain.com
-l /var/log/pcp/pmie/ntp1isp.mydomain.com/pmie.log -c config.mydomain
root 29561 1 0 May29 ? 00:00:00 pmie -b -h transit.mydomain.com
-l /var/log/pcp/pmie/transit.mydomain.com/pmie.log -c config.transit
root 29649 1 0 May29 ? 00:00:00 pmie -b -h
provdbs1.mydomain.com -l /var/log/pcp/pmie/provdbs1.mydomain.com/pmie.log -c
config.mydomain
root 29749 1 0 May29 ? 00:00:00 pmie -b -h
provdbm1.mydomain.com -l /var/log/pcp/pmie/provdbm1.mydomain.com/pmie.log -c
config.mydomain
root 29865 1 0 May29 ? 00:00:00 pmie -b -h stl1.mydomain.com -l
/var/log/pcp/pmie/stl1.mydomain.com/pmie.log -c config.mydomain
root 30000 1 0 May29 ? 00:00:00 pmie -b -h
maildbm1.mydomain.com -l /var/log/pcp/pmie/maildbm1.mydomain.com/pmie.log -c
config.mydomain
root 30136 1 0 May29 ? 00:00:00 pmie -b -h porting.mydomain.com
-l /var/log/pcp/pmie/porting.mydomain.com/pmie.log -c config.mydomain
root 30284 1 0 May29 ? 00:00:00 pmie -b -h
billing1.mydomain.com -l /var/log/pcp/pmie/billing1.mydomain.com/pmie.log -c
config.mydomain
root 30444 1 0 May29 ? 00:00:00 pmie -b -h portal1.mydomain.com
-l /var/log/pcp/pmie/portal1.mydomain.com/pmie.log -c config.mydomain
root 30625 1 0 May29 ? 00:00:00 pmie -b -h
dhcp2voip.mydomain.com -l /var/log/pcp/pmie/dhcp2voip.mydomain.com/pmie.log -c
config.mydomain
root 30816 1 0 May29 ? 00:00:00 pmie -b -h dhcp2tv.mydomain.com
-l /var/log/pcp/pmie/dhcp2tv.mydomain.com/pmie.log -c config.mydomain
root 31012 1 0 May29 ? 00:00:23 pmie -b -h
dhcp2isp.mydomain.com -l /var/log/pcp/pmie/dhcp2isp.mydomain.com/pmie.log -c
config.mydomain
root 31220 1 0 May29 ? 00:00:00 pmie -b -h
dhcp1voip.mydomain.com -l /var/log/pcp/pmie/dhcp1voip.mydomain.com/pmie.log -c
config.mydomain
root 31440 1 0 May29 ? 00:00:00 pmie -b -h dhcp1tv.mydomain.com
-l /var/log/pcp/pmie/dhcp1tv.mydomain.com/pmie.log -c config.mydomain
root 31672 1 0 May29 ? 00:00:23 pmie -b -h
dhcp1isp.mydomain.com -l /var/log/pcp/pmie/dhcp1isp.mydomain.com/pmie.log -c
config.mydomain
root 31922 1 0 May29 ? 00:00:00 pmie -b -h m1dhcp1.mydomain.com
-l /var/log/pcp/pmie/m1dhcp1.mydomain.com/pmie.log -c config.mydomain
root 32178 1 0 May29 ? 00:00:00 pmie -b -h m1dhcp2.mydomain.com
-l /var/log/pcp/pmie/m1dhcp2.mydomain.com/pmie.log -c config.mydomain
root 32446 1 0 May29 ? 00:00:00 pmie -b -h m2dhcp1.mydomain.com
-l /var/log/pcp/pmie/m2dhcp1.mydomain.com/pmie.log -c config.mydomain
root 32727 1 0 May29 ? 00:00:00 pmie -b -h m2dhcp2.mydomain.com
-l /var/log/pcp/pmie/m2dhcp2.mydomain.com/pmie.log -c config.mydomain
root 574 1 0 May29 ? 00:00:00 pmie -b -h http1.mydomain.com
-l /var/log/pcp/pmie/http1.mydomain.com/pmie.log -c config.mydomain
root 886 1 0 May29 ? 00:00:00 pmie -b -h ns1.mydomain.com -l
/var/log/pcp/pmie/ns1.mydomain.com/pmie.log -c config.mydomain
root 1208 1 0 May29 ? 00:00:00 pmie -b -h ns2.mydomain.com -l
/var/log/pcp/pmie/ns2.mydomain.com/pmie.log -c config.mydomain
root 1539 1 0 May29 ? 00:00:00 pmie -b -h ldapm1.mydomain.com
-l /var/log/pcp/pmie/ldapm1.mydomain.com/pmie.log -c config.mydomain
root 1879 1 0 May29 ? 00:00:00 pmie -b -h ldapm2.mydomain.com
-l /var/log/pcp/pmie/ldapm2.mydomain.com/pmie.log -c config.mydomain
root 2233 1 0 May29 ? 00:00:00 pmie -b -h
tvservices.mydomain.com -l /var/log/pcp/pmie/tvservices.mydomain.com/pmie.log
-c config.mydomain
root 2604 1 0 May29 ? 00:00:00 pmie -b -h prov1.mydomain.com
-l /var/log/pcp/pmie/prov1.mydomain.com/pmie.log -c config.mydomain
root 3007 1 0 May29 ? 00:00:00 pmie -b -h ztc1.mydomain.com -l
/var/log/pcp/pmie/ztc1.mydomain.com/pmie.log -c config.mydomain
root 3395 1 0 May29 ? 00:00:00 pmie -b -h ztc2.mydomain.com -l
/var/log/pcp/pmie/ztc2.mydomain.com/pmie.log -c config.mydomain
root 3795 1 0 May29 ? 00:00:00 pmie -b -h
emergency.mydomain.com -l /var/log/pcp/pmie/emergency.mydomain.com/pmie.log -c
config.mydomain
root 4207 1 0 May29 ? 00:00:00 pmie -b -h hermes.mydomain.com
-l /var/log/pcp/pmie/hermes.mydomain.com/pmie.log -c config.mydomain
root 4634 1 0 May29 ? 00:00:00 pmie -b -h log01.mydomain.com
-l /var/log/pcp/pmie/log01.mydomain.com/pmie.log -c config.mydomain
root 5077 1 0 May29 ? 00:00:01 pmie -b -h atmail1.mydomain.com
-l /var/log/pcp/pmie/atmail1.mydomain.com/pmie.log -c config.mydomain
root 5525 1 0 May29 ? 00:00:02 pmie -b -h atmail2.mydomain.com
-l /var/log/pcp/pmie/atmail2.mydomain.com/pmie.log -c config.mydomain
root 5985 1 0 May29 ? 00:00:02 pmie -b -h smtp1.mydomain.com
-l /var/log/pcp/pmie/smtp1.mydomain.com/pmie.log -c config.mydomain
root 6457 1 0 May29 ? 00:00:02 pmie -b -h smtp2.mydomain.com
-l /var/log/pcp/pmie/smtp2.mydomain.com/pmie.log -c config.mydomain
root 6945 1 0 May29 ? 00:00:03 pmie -b -h maildb2.mydomain.com
-l /var/log/pcp/pmie/maildb2.mydomain.com/pmie.log -c config.mydomain
root 7444 1 0 May29 ? 00:00:00 pmie -b -h mw1.mydomain.com -l
/var/log/pcp/pmie/mw1.mydomain.com/pmie.log -c config.mydomain
root 7953 1 0 May29 ? 00:00:00 pmie -b -h asavdb1.mydomain.com
-l /var/log/pcp/pmie/asavdb1.mydomain.com/pmie.log -c config.mydomain
root 8474 1 0 May29 ? 00:00:00 pmie -b -h asav1.mydomain.com
-l /var/log/pcp/pmie/asav1.mydomain.com/pmie.log -c config.mydomain
root 9007 1 0 May29 ? 00:00:27 pmie -b -h asav2.mydomain.com
-l /var/log/pcp/pmie/asav2.mydomain.com/pmie.log -c config.mydomain
root 9557 1 0 May29 ? 00:00:00 pmie -b -h asav3.mydomain.com
-l /var/log/pcp/pmie/asav3.mydomain.com/pmie.log -c config.mydomain
root 10114 1 0 May29 ? 00:00:28 pmie -b -h asav4.mydomain.com
-l /var/log/pcp/pmie/asav4.mydomain.com/pmie.log -c config.mydomain
root 10682 1 0 May29 ? 00:00:00 pmie -b -h asav5.mydomain.com
-l /var/log/pcp/pmie/asav5.mydomain.com/pmie.log -c config.mydomain
root 11263 1 0 May29 ? 00:00:00 pmie -b -h asav6.mydomain.com
-l /var/log/pcp/pmie/asav6.mydomain.com/pmie.log -c config.mydomain
root 11859 1 0 May29 ? 00:00:00 pmie -b -h
mobileprov.mydomain.com -l /var/log/pcp/pmie/mobileprov.mydomain.com/pmie.log
-c config.mydomain
root 12492 1 0 May29 ? 00:00:00 pmie -b -h wiki.mydomain.com -l
/var/log/pcp/pmie/wiki.mydomain.com/pmie.log -c config.mydomain
root 17487 1 0 May29 ? 00:00:00 pmie -b -h
dhcp2voip.mydomain.com -l /var/log/pcp/pmie/dhcp2voip.mydomain.com/pmie.log -c
config.mydomain
root 18798 1 0 May29 ? 00:00:00 pmie -b -h
dhcp1voip.mydomain.com -l /var/log/pcp/pmie/dhcp1voip.mydomain.com/pmie.log -c
config.mydomain
root 19444 1 0 May29 ? 00:00:00 pmie -b -h dhcp1tv.mydomain.com
-l /var/log/pcp/pmie/dhcp1tv.mydomain.com/pmie.log -c config.mydomain
root 23375 1 0 May29 ? 00:00:00 pmie -b -h ns2.mydomain.com -l
/var/log/pcp/pmie/ns2.mydomain.com/pmie.log -c config.mydomain
root 480 1 0 May29 ? 00:00:00 pmie -b -h
mobileprov.mydomain.com -l /var/log/pcp/pmie/mobileprov.mydomain.com/pmie.log
-c config.mydomain
root 567 1 0 May29 ? 00:00:00 pmie -b -h m1dhcp2.mydomain.com
-l /var/log/pcp/pmie/m1dhcp2.mydomain.com/pmie.log -c config.mydomain
root 3722 1 0 May29 ? 00:00:00 pmie -b -h
tvservices.mydomain.com -l /var/log/pcp/pmie/tvservices.mydomain.com/pmie.log
-c config.mydomain
root 4424 1 0 May29 ? 00:00:00 pmie -b -h prov1.mydomain.com
-l /var/log/pcp/pmie/prov1.mydomain.com/pmie.log -c config.mydomain
root 5145 1 0 May29 ? 00:00:00 pmie -b -h ztc1.mydomain.com -l
/var/log/pcp/pmie/ztc1.mydomain.com/pmie.log -c config.mydomain
root 6838 1 0 May29 ? 00:00:00 pmie -b -h hermes.mydomain.com
-l /var/log/pcp/pmie/hermes.mydomain.com/pmie.log -c config.mydomain
root 19073 1 0 May29 ? 00:00:00 pmie -b -h ztc2.mydomain.com -l
/var/log/pcp/pmie/ztc2.mydomain.com/pmie.log -c config.mydomain
---------------------------------------------------------
The duplicates are on dhcp1tv, dhcp1voip, dhcp2voip, hermes, m1dhcp2,
mobileprov, ns2, prov1, tvservices, ztc1 and ztc2. When I run a
/etc/init.d/pmie stop, these 11 will not stop and I have to kill them
manually. After I start pmie from the initscript again, there's only
one instance for each host, so I'm pretty confident it's the pmie_check
that's mistakingly spawning these.
-jf
|