Hi, Lukas -
> [...] This is a situation where (as Frank has mentioned) state of
> the pmda and papi counters is required to have a meaningful test,
> so, aiui, using -L as currently in the testcase will not work. If
> the answer is a combination of dbpmda/pmcd and a helper PAPI test
> program, then so be it, and I'll be happy to work on an approach
> that way!
A quicker approach to that testing just s/-L...// in the test case;
that way your qa script gets to test your code as an end-user would.
> As somebody new to PCP and to the inner workings of its testsuite, I
> intend to make testcases a priority when introducing new
> functionality, and will make every effort to do so. If I may humbly
> inquire, would it not make sense to test how the pmda reacts when
> "papi proper" (ie, not the test papi) returns, say, an PAPI_ECNFLCT
> error? [...]
Certainly. IMHO, testing against the real PAPI library is worth its
weight in complexity. Testing against a synthetic-PAPI would be a
necessary evil at best, if we are unable to evoke desired corner cases
on demand from the real thing. (For one thing, testing primarily
against a synthetic-PAPI could easily generate a false sense of
confidence. For another thing, a nontrivial synthetic-PAPI would
itself be a piece of software in need of testing...)
> I think an approach like this could make sense, and looking over the
> further emails, adding *additional* tests that do this, while
> keeping the older ones in place seems agreeable.
Yes. I did not mean to come across as suggesting that the current
test cases should be dropped - just that those that use -L in a futile
manner switch to -hlocal:.
> In this case I think the hard(er?) part is reliably finding a case
> that could cause an ECNFLCT, aiui, that changes based on the
> hardware. My initial thought is to find a way to iterate over the
> available metrics, until we reach a limit, and then adding another
> to set us 'over'.
Yup. Now that the code enables multiplexing by default, it might take
a few iterations to get to that saturation point, but one will be
reached at -some- point.
> I'm also aware I need to pay attention to installing and removing
> the pmda as part of the test.
That part's taken care of as qa/903.
> In regards to the "sanity checking that they are growing at
> reasonable rates". Considering that this will be run on different
> hardware, potentially with vastly different counter performance and
> capabilities, do you have any thoughts on how we could estimate what
> a 'reasonable rate' would be? My first thought is to perhaps
> compare rates/ratios of different, specific, metrics. However, I'm
> not sure how much those vary per architecture. Thoughts?
Yeah, it's tricky. I was indeed thinking of selecting a few generally
available metrics like TOT_CYC that counts regularly and fast, and one
of the L?_?CM (cache miss) counters that should increment on an
ongoing basis even on an idlish system, but relatively slowly. The
test code might need to do some searching to pick one, and estimate
its growth rate. (Then start enabling/disabling, flipping around, etc.)
- FChE
|