Re: [RFC] Using Vagrant for QA tests

[Ryan Doyle <rdoyle@xxxxxxxxxx>]

Hi Frank

> Neat.  One problem with an overly hardcoded approach like
> <https://github.com/Aconex/pcp/blob/vagrant-qa/Vagrantfile> is that it
> can encourage a monoculture of pcp test environments (a particular
> ubuntu version etc. in this case). 

Luckily Vagrant can support multiple VMs. For example we could support Ubuntu, FreeBSD, OmniOS etc...

------

  config.vm.define "precise64" do |b| 
    b.vm.box = "precise64"
    b.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box"
  end

  config.vm.define "freebsd9" do |b|

    b.vm.box = "freebsd9"

    b.vm.box_url = "http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_freebsd-9.2_chef-provisionerless.box"

  end

  config.vm.define "omnios" do |b|

    b.vm.box = "omnios"

    b.vm.box_url = "http://omnios.omniti.com/media/omnios-latest.box"

  end

-----

These can then all be addressed via `vagrant ssh freebsd9|precise64|omnios` to get inside them

Vagrant boxes (the base images) can either be found at sites like http://www.vagrantbox.es/ or they can be rolled yourself and put somewhere on the web.

> If we can use Vagrant or such to be able to instantiate a wide range
> of diverse environments (like kenj's farm on steroids), that'd address
> this nicely.

Hopefully the example above gives you a better idea of what can be supported. Basically, if it can run on VirtualBox, it can be managed and automated through Vagrant.

> As far as building prerequisites go, this could be another instance of
> the above problem: a monoculture of build configurations.  PCP has
> autoconf* machinery to detect presence or absence of numerous optional
> libraries, and that stuff needs to be exercised within the cartesian
> product / configuration space.

I can't say I'm a fan of detecting what you can run and skipping the rest. Ideally I should be able to run the complete test suite. It's easy to break tests and not know about it until someone else has to run them for you (because they are the only ones with the special setup).

> That is a great property.  I believe lberk is working on improving
> qa/check and pals to automate prerequisite setup in general.

That's good news, I'll be keeping an eye on this :)

----
Ryan Doyle

---

From: "Frank Ch. Eigler" <fche@xxxxxxxxxx>
To: "Ryan Doyle" <rdoyle@xxxxxxxxxx>
Cc: pcp@xxxxxxxxxxx
Sent: Friday, 12 December, 2014 2:19:36 AM
Subject: Re: [RFC] Using Vagrant for QA tests

Hi -

rdoyle wrote:

> [...]
> I've created a branch "vagrant-qa" at the Aconex guthub repository (https://
> github.com/Aconex/pcp.git) if you'd like to have a look.
> [...]

Neat.  One problem with an overly hardcoded approach like
<https://github.com/Aconex/pcp/blob/vagrant-qa/Vagrantfile> is that it
can encourage a monoculture of pcp test environments (a particular
ubuntu version etc. in this case).  That can lead to false positive
test results, when new|old code avoids being run on actual new|old OS
kernels, architectures, ..., and so has a beautiful pass rate on the
VM but fails in real life.  "All Of This Has Happened Before And Will
Happen Again."

If we can use Vagrant or such to be able to instantiate a wide range
of diverse environments (like kenj's farm on steroids), that'd address
this nicely.

> - Anyone can run the QA and they don't need to know how to setup the
> prerequisites. [...]

As far as building prerequisites go, this could be another instance of
the above problem: a monoculture of build configurations.  PCP has
autoconf* machinery to detect presence or absence of numerous optional
libraries, and that stuff needs to be exercised within the cartesian
product / configuration space.

> - Set up of the environment required to run the
> QA test is captured as code

That is a great property.  I believe lberk is working on improving
qa/check and pals to automate prerequisite setup in general.

- FChE