Hi Ken,
> > for different PCP versions. However, I've found several
> > of the tests to have *deep* knowledge of internals of PCP,
> > and they can break in non-intuitive ways when the planets
> > are not aligned.
>
> Because PCP follows a fundamentally a client-server architecture it
> is absolutely critical that the QA components are _not_ exposed to
> version
> dependencies but rather accommodate version differences so that
> cross-version inter-operation can be tested, e.g. new clients with
> old pmcd and vice versa.
Right. We're definitely on the same page there. This always was the
case (even before, when pcp and pcpqa were one and the same tree),
and I'm 100% agreeing this must remain the case.
But there's another problem - while the remote interactions must allow
for different versions to be tested together, I don't think it makes
sense to run pcpqa binaries built for one libpcp to run with newer or
older libpcp variants. That's what bit me, and what I'd like to prevent
from happening.
For example, line 67 in this test source file:
http://oss.sgi.com/cgi-bin/gitweb.cgi?p=pcp/pcpqa.git;a=blob;f=src-oss/descreqX2.c;h=ed8fb2b8c758c75a550f149fe788408b1f6f3b98;hb=HEAD
makes use of (IMO) a valid (for a test) piece of knowledge about the
offset of a field within an internal libpcp data structure. That is
not exposed via the PMAPI, but should be tested. Changes within the
(internal) context structure prevent a libpcp-3.6.5 (for example) to
work with a pcp-testsuite-2.0.0 (once that exists).
So, I really want the QA binaries from a build to match with libpcp
(and libpcp_pmda.so, and the rest, of course), and I would like the
native package manager for each platform to ensure that. I think it
is too much to expect the test authors to be able to get the version
checks right manually for the descreqX2.c case above (in fact it may
not be possible, for that case).
> If merging the trees encourages (even implicitly) the creation of a
> version of the QA packages that _only_ work with the same version of
> the
> PCP packages then that would be a bad thing(tm) ... if QA continues
> to
> be multi-version aware, then that would be a good thing(tm).
Yep, agreed and agreed.
> > - Ken did some initial work on the pcpqa tree a couple of
> > months back & it can now be made produces a .deb package.
>
> I stopped work on this when I could not decide what the QA package
> should contain:
> a) compiled binaries and scripts, or
> b) source and scripts.
>
> a) means you need one package for each QA platform of interest.
Yes, a/ is my plan. And its greatly simplified when the qa package
comes into existance at the same time as the pcp package its built
for. With b/ I'd be very concerned about relying on "make" to get
descreqX2.c compiled correctly every time.
>
> Since some of the tests require a toolchain (not may I agree), b)
> seems
> simpler, means only one package for all platforms, and using tar to
> package this would be simpler as we'd avoid all of the uglier parts
> of the packaging "gooiness".
Not really agreeing there - tar is not going to work for managing new
versions of the tests. Example above again (descreqX2.c) for "why".
> > ...
> > I vastly favour the second option for simplicity - I'd like
> > to end up with one pcp tree again at the end, and have the
> > same model for both pcp and pcp-gui.
>
> Other than my issue above, I don't really mind. I think it is highly
> desirable to preserve the git history.
OK.
> The one other "big" issue to be discussed is _what_ does unattended
> QA
> really mean? I think I'd vote for a scheme where we had tests in the
> QA
> suite that were _not_ run as part of the unattended run, so that we
> could avoid the more difficult of the QA setup issues (like a working
> X
> server, a close Cisco router, ...) ... and this would make much of
> Tomas' issues with the setup checks just go away.
Yes, I think that's fine.
For me now I'm most concerned with getting the right pcp qa binaries on
the right platforms, with the right versions of libpcp[_pmda] (locally),
where the matrix of test platforms is already very large and growing.
cheers.
--
Nathan
|