On Fri, Dec 13, 2013 at 03:10:39AM -0800, Christoph Hellwig wrote:
> On Fri, Dec 13, 2013 at 12:09:34PM +1100, Dave Chinner wrote:
> > > You can have different test devices, or simply not bother with aging
> > > it for every run. You're missing the coverage of all the test dir
> > > using tests, which are a lot with the above version anyway.
> >
> > IOWs, you're saying that you don't consider MKFS_OPTIONS as a first
> > class citizen. I've been using it for 7 or 8 years for exactly this
> > purpose - iterating testing of a change quickly across multiple
> > configurations without perturbing the long term aging of the test
> > device.
>
> But you're limiting yourself to the tests only using the scratch
> device for that testing, leaving out all the ones using the TEST
> directory.
>
> > I'm not opposed to making the change, just pointing out that
> > reducing the usage of the scratch device has a fairly significant
> > impact on test coverage for anyone who uses MKFS_OPTIONS in their
> > workflow...
>
> It does have an impact for that particular workload, but I think that
> workload is broken as you only test your specific config for those
> tests using the scratch device, and do not get the coverage for the
> tests using the test device.
>
> git-grep -l TEST_DIR tests/generic/ | grep -v out | wc -l
> 65
> git-grep -l TEST_DIR tests/xfs/ | grep -v out | wc -l
> 23
>
>
> hch@brick:~/work/xfstests$ git-grep -l _require_scratch tests/generic/ | wc -l
> 58
> hch@brick:~/work/xfstests$ git-grep -l _require_scratch tests/xfs/ | wc -l
> 128
>
> So you're missing close to 2/3s of the tests already.
I think you got that the wrong way around: that's 2/3rds of the
tests (186) use the scratch device rather than the test device.
There's also roughly 100 tests (of ~160) in the quick group that use
the scratch device.
Hence doing smoke tests by simply changing the MKFS_OPTIONS gets a
significant amount of coverage *quickly*, and that's usually more
than sufficient to flush out bugs during development.
Cheers,
Dave.
--
Dave Chinner
david@xxxxxxxxxxxxx
|