[rspec-users] Spec/Test Speed

Pat Maddox pergesu at gmail.com
Sun Oct 7 11:42:50 EDT 2007


On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:
>
> On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote:
>
> > On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:
> >>
> >> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote:
> >>
> >>> Scott,
> >>>
> >>> I don't really have a lot to contribute on how to make it faster,
> >>> other than to outline what we've been doing on our projects.
> >>>
> >>> On one of our current projects we have the following 2570 examples
> >>> that run in ~70 seconds on our pairing stations (mac minis, 1.83
> >>
> >> I assume your "pairing stations" are two separate mac-minis, in which
> >> you practice pair programming?  Or is this a cluster of two mac-
> >> minis?
> >>
> >> But this sounds great - 70 seconds for 2500 specs.  How many of those
> >> are model specs (that hit the database)?
> >>
> >>> c2d).  In general across our various machines is at or a little more
> >>> than a minute for specs for controllers, models, helpers, lib,
> >>> views,
> >>> and plugins.   Our Story suite takes longer, but it's still under
> >>> development so I don't really count it at this point.   We have Ruby
> >>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5.
> >>> (current as of a month ago) from macports.
> >>>
> >>> We make good use of mocking and stubbing through our controller
> >>> tests, and little use of fixtures.  We primarily use the
> >>> spec_attribute_helper (or factory method) as Luke Redpath and Dan
> >>> Manges have outlined in their respective blog articles.     I've
> >>> been
> >>> looking at deep test, or possible spec_distributed as a way to speed
> >>> things up more.   Our main issue is our precommit task (rake cruise
> >>
> >> The factory method (or attribute_helper) still hits the database.  I
> >> don't see it as any sort of performance gain. In fact, I've even
> >> developed a plugin around the Factory idea, and it was only when I
> >> started using it in all of my tests that the speed really started to
> >> affect me (I was using mocking/stubbing, with much frustration prior
> >> to that point).  But to me it's pretty clear the plugin (or the
> >> factory) is not the problem - the hit is the database.  DHH saw this
> >> hit, and since they were using fixtures,he found that creating the
> >> fixtures, and then wrapping each test in a transaction was a huge
> >> performance gain.  I wonder if the same would be true with setups/
> >> before(:each)...
> >>
> >> The obvious thing to do to solve the performance problem is to remove
> >> the hit to the database.  The question is: At what level of
> >> abstraction should this be done?  The one camp (which would include
> >> fellows like Jay Fields), would mock/stub everything they don't
> >> write.  For me, I see testing as more than testing - it's the
> >> documentation to my code which never lies to me (this documentation
> >> is so good, that I can give it to my boss, who is not a programmer).
> >
> > I think you'd be far better served by user stories in this case.  The
> > user story is the tool we use to communicate with non-geeks, be it
> > customers or our boss.  Low-level specs fall squarely in the developer
> > realm, as far as I'm concerned.  They're easy to read because we don't
> > like to torture ourselves.  I think that they're fantastic for
> > communicating intent with other developers, but that's where it ends.
> >
> > So, when you move documentation as a goal from specs over to stories,
> > that frees specs to take advantage of techniques like mocks to help
> > with your design.  Documentation is a pleasant side effect of BDD, not
> > a goal.
>
> Documentation is certainly a goal of BDD (Just not necessarily for
> the business user - as you pointed out).  I don't write my specs to
> be understandable by the business user, although it just often
> happens that they are (at least in this rails context, since even
> most of the fields of the database are more-or-less user facing).
> The point that I was trying to hammer home is that mocking/stubbing
> adds a lot of extra noise to a rails test, where it doesn't in the
> other projects that I've worked on.

Well, one of the main parts of programming is making tradeoffs.
You're probably not going to run 2500 tests that hit the db in under
30 seconds.  So you either don't hit the db or you don't run all the
tests all the time.  You can certainly change autotest to not
automatically run the full suite after you go red->green.  Only have
it run the specs for the files you changed.

Break your specs into fast and slow groups.  Run the fast ones
continuously and the slow ones left often.  Work on moving one or two
slow specs into the fast group every day.  You'll probably discover
some techniques that you can share with all of us :)

As far as mocks adding a lot of extra noise, I find in general that
when mocks are painful for me it's because my code is too tightly
coupled.  Just like my body tells me something is wrong through pain,
so does my code.


> But whether or not you call the specs "stories" or "integration
> specs" (it is, after all, all a continuum of what is considered an
> "integration" test), I still have no idea how to run them any faster.

Well, I don't want to beat this to death, but I think it's dangerous
to look at specs and stories as simply different degrees of the same
concept.  The fundamental difference is that you should not change a
story without consulting a customer, and you should not have to
consult a customer to change a spec.  Stories represent assumptions
and business value.  Assumptions are learned through conversations
with the customer, and only the customer can define business value.
Specs are merely a tool to help us generate that business value, and
velocity goes way down if we have to worry that changing some specs
will impact the customer in some way.  Worse, if we don't worry about
it, we may end up changing some assumptions around and thus deliver
the wrong product.

In the ever popular analogy of car mechanics, I would want my mechanic
to ask me before replacing an axle on my car, but I would not examine
the wrench he chooses to perform the maintenance.

Pat


More information about the rspec-users mailing list