[rspec-users] Spec run heuristics (Re: Cucover: coverage-aware 'lazy' cucumber runs)

[Ben Mabey]

Stephen Eley wrote:
> On Sat, Apr 11, 2009 at 2:02 PM, Ashley Moran
> <ashley.moran at patchspace.co.uk> wrote:
>   
>> I was just idly thinking, could a code-coverage based system could be
>> combined with some sort of failure (fragility) history to balance the time
>> cost of heavy feature runs with the benefits of having something run
>> end-to-end?  We've had reverse-modification-time spec ordering for ages
>> which is a useful start.
>>     
>
> I've had it in my head for a while now that someday (yes, that
> mythical 'someday') I want to write a better autotest.  Maybe this is
> heresy, but I am a huge fan of the _idea_ behind autotest and totally
> annoyed by its implementation.  It's extensible only in strange ways
> (hence wrappers like autospec), and its fundamental strategy is too
> static.  I once lost most of a day trying to fix merb_cucumber so the
> features would run when they should, and was ready to hurl cats when I
> realized autotest's idea of context chaining was to make you list them
> all in the classname in alphabetical order.  Look at the files in the
> Cucumber gem's 'lib/autotest' directory and you'll see what I mean.
>
> A proper design would let you plug in your own file-change discovery
> strategy, plug in multiple runners (RSpec, Cucumber, yadda yadda) with
> true modularity, specify lists of observers on directories or files,
> and allow different output views.  An _ideal_ design would also let
> you set priority rules like you're describing here, so you get instant
> feedback only on the stuff you're working with, and do end-to-end runs
> in the background.
>   
That would be very cool, you have lots of good ideas there.  Being able 
to plug in your own file-change strategy would be killer.  Another cool 
idea I ran across the other idea is being able to specify in your 
examples that which ones are "focussed".  Meaning, autotest will only 
run the focussed ones and not bother running the entire suite. Once you 
have solved the problem at hand you remove the focussed tag and the 
whole suite is then ran.  This idea, *which is already implemented*, 
comes from Micronaut[1].  The idea is very similar to Cucumber's and 
RSpec's[2] tagging feature (yet to come for rspec).  The cool thing 
about micronaunt is that they have tied it into autotest.  Ideally, we 
could be able to tell autotest, or whatever program, to only run tests 
that are tagged a certain way-- and then you could override that with 
the "focused" tag.  So, we can add that to our list of cool things to 
have. :)

-Ben

1. 
http://blog.thinkrelevance.com/2009/3/26/introducing-micronaut-a-lightweight-bdd-framework
2. 
https://rspec.lighthouseapp.com/projects/5645/tickets/682-conditional-exclusion-of-example-groups