[rspec-users] Spec run heuristics (Re: Cucover: coverage-aware 'lazy' cucumber runs)

Matt Wynne matt at mattwynne.net
Mon Apr 13 16:09:07 EDT 2009

On 13 Apr 2009, at 12:46, aslak hellesoy wrote:

> A couple of years ago I was on a project that had fallen into the  
> trap of too many integration tests (exactly the horror scenario that  
> J.B. Rainsberger describes: http://www.jbrains.ca/permalink/239).  
> The whole suite had hundreds of slow watir tests and took several  
> hours to run. If there was a failure, it was usually just in a  
> coupel of them.
> We ended up improving this a lot with a homegrown distributed RSpec  
> runner (based on Drb) and employing techniques from pairwise testing  
> (http://www.pairwise.org/).
> At the time I also explored a third technique that we never put in  
> use: Heuristics.
> If we could establish relationships between arbitrary files and  
> failing tests over time, then we would be able to calulate the  
> probablilty that a certain commit would break certain tests. We  
> could then choose to run the tests that had a high probablilty of  
> breakage, and exclude the others. A neural network could potentially  
> be used to implement this. The input neurons would be files in the  
> codebase (on if it's changed, off if not) and the output neurons  
> would be the tests to run.
> So if someone develops a better AutoTest with a plugin architecture,  
> and that doesn't have to run as a long/lived process, then I'd be  
> very interested in writing the neural network part - possibly backed  
> by FANN (http://leenissen.dk/fann/)
> It's so crazy it has to be tried!
> Aslak

Aslak as usual you have stepped it up another seventy levels!

I'm really keen to write a generic test runner that works on both  
RSpec and Cucumber examples alike (and could extend for other testing  
libraries). Allowing for pluggable strategies for test selection as  
Stephen has described is a terrific idea.

Aslak do you have any ideas in the pipeline for building a fancier  
test runner for Cucumber? I know there was some brain-storming on the  
list a while back about a 'thick client'.

Matt Wynne

More information about the rspec-users mailing list