[rspec-users] Spec run heuristics (Re: Cucover: coverage-aware 'lazy' cucumber runs)

Matt Wynne matt at mattwynne.net
Mon Apr 13 16:02:00 EDT 2009

On 12 Apr 2009, at 23:51, Ben Mabey wrote:

> Stephen Eley wrote:
>> On Sat, Apr 11, 2009 at 2:02 PM, Ashley Moran
>> <ashley.moran at patchspace.co.uk> wrote:
>>> I was just idly thinking, could a code-coverage based system could  
>>> be
>>> combined with some sort of failure (fragility) history to balance  
>>> the time
>>> cost of heavy feature runs with the benefits of having something run
>>> end-to-end?  We've had reverse-modification-time spec ordering for  
>>> ages
>>> which is a useful start.
>> I've had it in my head for a while now that someday (yes, that
>> mythical 'someday') I want to write a better autotest.  Maybe this is
>> heresy, but I am a huge fan of the _idea_ behind autotest and totally
>> annoyed by its implementation.  It's extensible only in strange ways
>> (hence wrappers like autospec), and its fundamental strategy is too
>> static.  I once lost most of a day trying to fix merb_cucumber so the
>> features would run when they should, and was ready to hurl cats  
>> when I
>> realized autotest's idea of context chaining was to make you list  
>> them
>> all in the classname in alphabetical order.  Look at the files in the
>> Cucumber gem's 'lib/autotest' directory and you'll see what I mean.
>> A proper design would let you plug in your own file-change discovery
>> strategy, plug in multiple runners (RSpec, Cucumber, yadda yadda)  
>> with
>> true modularity, specify lists of observers on directories or files,
>> and allow different output views.  An _ideal_ design would also let
>> you set priority rules like you're describing here, so you get  
>> instant
>> feedback only on the stuff you're working with, and do end-to-end  
>> runs
>> in the background.

+1 to this Stephen, I am with you 100%.

A direct email about the pipe-dream is on it's way.

> That would be very cool, you have lots of good ideas there.  Being  
> able to plug in your own file-change strategy would be killer.   
> Another cool idea I ran across the other idea is being able to  
> specify in your examples that which ones are "focussed".  Meaning,  
> autotest will only run the focussed ones and not bother running the  
> entire suite. Once you have solved the problem at hand you remove  
> the focussed tag and the whole suite is then ran.  This idea, *which  
> is already implemented*, comes from Micronaut[1].  The idea is very  
> similar to Cucumber's and RSpec's[2] tagging feature (yet to come  
> for rspec).  The cool thing about micronaunt is that they have tied  
> it into autotest.  Ideally, we could be able to tell autotest, or  
> whatever program, to only run tests that are tagged a certain way--  
> and then you could override that with the "focused" tag.  So, we can  
> add that to our list of cool things to have. :)

I actually don't think it should be necessary to tell the tool where  
to focus. As long as it understands the relationship between your  
tests and your source code, and you're making changes test-first, the  
tool should be able to know which parts of your code are unstable and  
likely to need re-testing after a change.

Matt Wynne

More information about the rspec-users mailing list