[rspec-users] Spec run heuristics (Re: Cucover: coverage-aware 'lazy' cucumber runs)

Matt Wynne matt at mattwynne.net
Wed Apr 15 04:14:03 EDT 2009


Yeah this is all good stuff.

We're basically talking about 'personal CI' here I think[1]. Nothing  
wrong with that, but we must remember some of this can be deferred  
until check-in.

[1]http://silkandspinach.net/2009/01/18/a-different-use-for-cruisecontrol/

On 14 Apr 2009, at 21:13, Andrew Premdas wrote:

> One simple thing I asked about the other day was running multiple  
> instances of autotest to do different things. Currently I'd like to  
> run one for my specs and one for my features, but you could easily  
> extend this idea. Creating several profiles that run at the same  
> time, with the long running ones having a low priority would give a  
> range of feedback that eventually would be fairly complete (on a big  
> project it might fully catch up overnight, or at the weekend) whilst  
> providing sufficient feedback to be able to iterate quickly with  
> reasonable confidence
>
> 2009/4/14 Stephen Eley <sfeley at gmail.com>
> On Mon, Apr 13, 2009 at 7:46 AM, aslak hellesoy
> <aslak.hellesoy at gmail.com> wrote:
> >
> > So if someone develops a better AutoTest with a plugin  
> architecture, and
> > that doesn't have to run as a long/lived process, then I'd be very
> > interested in writing the neural network part - possibly backed by  
> FANN
> > (http://leenissen.dk/fann/)
>
> In the immortal words of Socrates, "That rocks."
>
> The nice thing about separating concerns like this -- the reason why
> design patterns appeal so much to me -- is that pieces can rather
> easily be built up incrementally.  As Matt said, thinking 'neural
> networks' with this is way beyond the level of anything I'd had in my
> head.  But it's a damn cool idea.
>
> I can think of a few ways to handle this particular chunk, with levels
> of complexity that would scale up:
>
> 1.) DUMB REACTION: Just re-run the tests that failed in this run
> cycle.  Then, periodically, run tests that succeeded in the background
> as a regression check.  (This isn't much beyond what Autotest does
> now.)
>
> 2.) OBSERVERS: Allow handlers to register themselves against certain
> files, so that when a file is changed, that handlers gets run.
> Multiple handlers can observe any given file, and a handler can
> declare multiple rules, including directories or pattern matches.
> (Again, Autotest has something _sort of_ like this, but not nearly as
> flexible.)
>
> 3.) PERSISTENCE: Track the history of tests and the times they were
> created, edited, last run, and last failed.  Also track file
> modification times.  When a file changes, run the tests first that are
> either new or have failed since the last time the file was changed.
> Then run the tests that went from failing to passing in that time.
> (This could certainly be improved -- I haven't sat down to figure out
> the actual rule set -- but you get the gist.  Know when things changed
> and set priorities accordingly.)
>
> 4.) INTELLIGENCE: Aslak's neural network.  Let the system figure out
> which tests matter to which files, and run what it thinks it ought to
> run.  Maybe use code coverage analysis.  It can 'learn' and improve
> when the full suite is run and it discovers new failures.
>
> In all four of these cases, I still think it's imperative to run the
> full suite.  None of these methods are foolproof, and code is tricky
> and makes weird things happen in weird crevices.  That's _why_ testing
> must be done.  But running the suite doesn't have to be a 'blocking'
> activity, like it is with Autotest now.  It can happen in bits and
> pieces, when nothing else is going on, and it can be configured to
> only grab your attention when something failed unexpectedly.
>
> (That's one of the prime reasons for the 'multiple output views'
> feature, by the way.  When I said output views I wasn't just thinking
> of the console or a window.  I'm also thinking Dashboard widgets, or
> gauges in the toolbar, or RSS feeds, or dynamic wallpaper, or whatever
> else anyone can think of.  Stuff that stays out of your way until it
> either needs you or you choose to look at it.)
>
> Still making sense?  This is starting to sound pretty big and pretty
> complex -- but I don't think it strictly needs to be.  #1 and #2 above
> are pretty easy.  The others don't have to be built before releasing
> something.  And, of course, you wouldn't have to pick just one
> selection module.  You could run or disable all of these depending on
> your project's needs, or come up with your own system involving Tarot
> cards and Linux running on a dead badger.(*)
>
> I just want to build a core that detects changes in stuff, tells other
> stuff about it, and passes on what that stuff says about it to a third
> set of stuff.  The rest is implementation-specific details.  >8->
>
> (* http://www.strangehorizons.com/2004/20040405/badger.shtml )
>
>
> --
> Have Fun,
>   Steve Eley (sfeley at gmail.com)
>   ESCAPE POD - The Science Fiction Podcast Magazine
>   http://www.escapepod.org
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users
>
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users

Matt Wynne
http://beta.songkick.com
http://blog.mattwynne.net





More information about the rspec-users mailing list