Hi -
On Tue, Nov 18, 2014 at 11:26:01PM -0500, Nathan Scott wrote:
> [...]
> > #3. From looking more at strace and the code, it looks as though the pmda
> > might be doing too much work per unit fetch.
> > [...]
> > #4. Instance domain unawareness. A variant of #3, when a pcp client is
> > [...]
> > #5. Multiple metrics unawareness (so to speak). A variant of #3,
>
> There's subtleties to the protocol you may not be following here
> and thats misleading you in interpreting the observed results, I
> think. The instance PDUs are a separate entity to fetch PDUs,
> but the PMDA needs to take similar refresh actions to respond to
> each.
The wire protocol is not at issue here. The pmcd / pmda does get told
eventually of the instances / metrics of specific interest to each
client, so by the time the custom pmda fetch callbacks are called,
they has all the info they need.
It's just that the way the linux-proc pmda responds to a fetch(), it
considers it necessary to refresh to world for each incoming call
(src/pmdas/linux_proc/pmda.c:2388). That's only a little burdensome
on small machines, but will outright fail to scale to hundreds or
thousands of cgroups.
If instead the pmda fetch callback produced pinpoint answers to the
incoming pmid/inst pair by looking up a single value in /proc or
/sys/fs/cgroup, there would be no wasted work. Many of those caches
(misnamed - they're just ordinary lookup tables here rather than
locality-based accelerators) might not be needed at all.
> > linux/cgroups-root*.tgz. It's clever & useful, but cannot be as
> > thorough.)
>
> Oh, it can be far more thorough than we could ever hope to be with
> testing by-hand [...]
I was not comparing it to testing-by-hand, but rather to testing
against a real live OS in its unforseeable gloriously messy diversity.
> [...] I imagine you simply have many more active cgroups than
> I did on my dev box when I created cgroups-root-001.tgz [...]
That could be part of it. Another part is that a small sequence of
operations against a fixed data set is not going to exercise the code
the same way as long-term operation against a varying data set.
Perhaps we could use a fuzzing-inspired test case which blasts all
PMDAs with a random bunch of traffic for an extended period, just
looking for the preservation of pcp service/uptime.
- FChE
|