----- Original Message -----
> [...]
> Even if fixed now, it needs a testcase imo
*nod*
> and this is the goal of qa/799. Only reason it was commited was so I
> could continue to build pcp with Makepkgs.
Ah, this is almost certainly because we drive parts of the build (esp.
the source tarball generation) from git now instead of the way we used
to (which was via explicitly listing all source files via makefiles).
Anyway, it should be enough to "git add" your new test, for Makepkgs to
be able to find it - and not go all the way through with git commit until
you're good and ready. Not a problem that the test is committed though,
of course.
> This is a situation where (as
> Frank has mentioned) state of the pmda and papi counters is required to
> have a meaningful test, so, aiui, using -L as currently in the testcase
> will not work. If the answer is a combination of dbpmda/pmcd and a
> helper PAPI test program, then so be it, and I'll be happy to work on an
> approach that way!
Sounds good to me.
> As somebody new to PCP and to the inner workings of its testsuite, I
> intend to make testcases a priority when introducing new functionality,
> and will make every effort to do so.
You will fly high and far in PCP-land with that kind of attitude. :)
> not make sense to test how the pmda reacts when "papi proper" (ie, not the
> test papi) returns, say, an PAPI_ECNFLCT error? This could even be in
> _addition_ to the test against the qa/src/*.c papi. I imagine this would
> only increase the level of test coverage the testsuite offers?
*nod*
> I think an approach like this could make sense, and looking over the
> further emails, adding *additional* tests that do this, while keeping
> the older ones in place seems agreeable. In this case I think the
> hard(er?) part is reliably finding a case that could cause an ECNFLCT,
> aiui, that changes based on the hardware. My initial thought is to find
> a way to iterate over the available metrics, until we reach a limit, and
> then adding another to set us 'over'.
If that does the trick, awesome. It might also pay dividends to wander
through the PAPI code a bit, to see if there are situations where it'll
return this ECNFLCT code in a fashion that is deterministic independent
of underlying hardware (and/or chat to Will too - maybe he knows ways to
trigger that off the top of his head).
> In regards to the "sanity checking that they are growing at reasonable
> rates". Considering that this will be run on different hardware,
> potentially with vastly different counter performance and capabilities,
> do you have any thoughts on how we could estimate what a 'reasonable
> rate' would be? My first thought is to perhaps compare rates/ratios of
> different, specific, metrics. However, I'm not sure how much those vary
> per architecture. Thoughts?
My only thought there is to keep in mind we don't have to test PAPI itself
here - we can and should stop at the point that we have confidence that
the pmdapapi metrics provide an accurate reflection of the values returned
by PAPI for individual counters. And they don't have to match exactly, of
course, sampling time differences will introduce some value fluctuations -
the qa/common.check code provides a _within_tolerance() function that may
prove handy.
cheers.
--
Nathan
|