|
Let me start off by saying I know nothing about PCP. I installed PCP on about 80 compute nodes of an HPC cluster. Most of these are working fine but I noticed 4 virtual machines completely lock up and die shortly after starting pmcd after booting or starting it during boot. The physical systems, which are configured exactly the same, do not lock up. By lock up I mean the machine is completely dead. Not 100% CPU busy, not out of memory, etc. but completely unresponsive. When this happens the system no longer even responds to ping so the kernel itself (or its networking) is dead. However, I do not see a kernel panic and can't get a crash dump so I can't see what is happening. The fact that this userland daemon is somehow killing the kernel but not triggering a kernel panic is very odd and worrying. Here's how I am using PCP. I am using XDMod which has a plugin, SUPReMM, which requires PCP. I installed and configured PCP via this SaltStack config: I then enabled and started pmcd, pmlogger, and pmie. At this point the VMs will hang in as little as under a minute. Since there are four machines effected and three daemons in the mix, I set the following for what daemon to start on boot (left side being the names of the VMs) to narrow down the issue:
After a number of reboots only dn2 will hang. I also was able to hang a machine by starting pmcd via systemctl after the system booted. Oddly, I have not been able to hit this issue every time I start the daemon. Any clue what is going on or how we can proceed with this issue? — |