Hi Martins,
It is good to hear someone's become interested in this topic again!
----- Original Message -----
> Hello,
> I have started looking into reviving the hotproc pmda. Before I
> get too far into it, I wanted to make sure no one else was looking at it
> and if my approach looks reasonable.
AFAIK, noone else has been looking into it recently. I have wondered for
some time whether its functionality could be alternately achieved, using
cgroups (albeit on Linux only) however ... and there is increasing levels
of interest in PCP and cgroups in some quarters (esp. due to containers
making extensive use of cgroups -- LXC & Docker being hot topics atm).
> [...]
> I haven't yet really thought about all the extra stuff the hotproc
> pmda gave you in terms of cpu cycles consumed by un-tracked processes,
> etc. At this point my use case is just to present non-root, high cpu
> processes but I figure trying to get as much of the original hotproc
> functionality would be desirable. Hopefully these can be added in as I
> progress.
>
> I am not tied at all to the approach as implemented. I just wanted
> to get something up and running as quickly as possible to get a feel for
> the level of effort required. Feel free to suggest better ways of doing
> this. I have many more questions once I can get some direction on the
> way to proceed.
My recollection is that pmdahotproc presents all of the proc metrics for a
set of "interesting" processes, below the hotproc.* namespace. It uses a
set of rules to identify the processes of interest, which are evaluated on
a user-defined polling interval.
On Linux, the cgroup concept could provide a more flexible mechanism. The
kernel support means the "hot process" evaluation side of things becomes
optional (IOW, one could deem processes worthy of logging or whatever-one-
would-otherwise-use-hotproc-metrics-for simply by ensuring the processes
to be monitored start in a specific cgroup, using existing cgroup tools.
Which, for some potential use-cases, is a big win cos walking the complete
list of processes can be prohibitively expensive, for very large process
counts. A handy mechanism for obtaining only the processes in a specific
cgroup is provided by the kernel already, and the Linux pmdaproc already
knows how to use it.
There's also the cgclassify(1) daemon, which could be used to provide some
of the other pmdahotproc bits, by automatically moving processes into a
closely-monitored cgroup. Or, perhaps more realistically (if we want to
use live performance data for making cgroup classifications, which we'll
surely want to do at some point), the parts of pmdahotproc that perform
the rule evaluation could be carved off into a pmdaproc/cgclassify helper
utility, and instead of having pmdahotproc providing both the metrics and
the "process hotness" evaluation functions, we could use the existing proc
PMDA (-r option, and/or the proc.control.perclient.cgroups metric) for the
metrics, and a separate hunk of code for the cgroup evaluation. Possibly
pmie could be used for this classification side of things, then we'd not
need a new tool for that either - worth experimenting with I think.
There are other good advantages to using cgroups for this task too. The
kernel tracks memory utilisation, CPU utilisation, hardware performance
counters, and now it seems even some I/O stats, at the cgroup level. So,
hopefully we can get a richer set of stats, and more cheaply, than the
traditional pmdahotproc provided.
And then there's the whole PMDA source code & namespace management issues
that you'll have come across in looking into pmdahotproc so far ... it'd
be nice to dodge all of that and simply have the one pmdaproc, if we can.
cheers.
--
Nathan
|