[rspec-users] Spec/Test Speed

Scott Taylor mailing_lists at railsnewbie.com
Sun Oct 7 13:08:38 EDT 2007

On Oct 7, 2007, at 11:42 AM, Pat Maddox wrote:

> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:
>> On Oct 7, 2007, at 1:47 AM, Pat Maddox wrote:
>>> On 10/6/07, Scott Taylor <mailing_lists at railsnewbie.com> wrote:
>>>> On Oct 7, 2007, at 12:31 AM, Chad Humphries wrote:
>>>>> Scott,
>>>>> I don't really have a lot to contribute on how to make it faster,
>>>>> other than to outline what we've been doing on our projects.
>>>>> On one of our current projects we have the following 2570 examples
>>>>> that run in ~70 seconds on our pairing stations (mac minis, 1.83
>>>> I assume your "pairing stations" are two separate mac-minis, in  
>>>> which
>>>> you practice pair programming?  Or is this a cluster of two mac-
>>>> minis?
>>>> But this sounds great - 70 seconds for 2500 specs.  How many of  
>>>> those
>>>> are model specs (that hit the database)?
>>>>> c2d).  In general across our various machines is at or a little  
>>>>> more
>>>>> than a minute for specs for controllers, models, helpers, lib,
>>>>> views,
>>>>> and plugins.   Our Story suite takes longer, but it's still under
>>>>> development so I don't really count it at this point.   We have  
>>>>> Ruby
>>>>> 1.8.6 installed from MacPorts on all machines, as well as MySQL 5.
>>>>> (current as of a month ago) from macports.
>>>>> We make good use of mocking and stubbing through our controller
>>>>> tests, and little use of fixtures.  We primarily use the
>>>>> spec_attribute_helper (or factory method) as Luke Redpath and Dan
>>>>> Manges have outlined in their respective blog articles.     I've
>>>>> been
>>>>> looking at deep test, or possible spec_distributed as a way to  
>>>>> speed
>>>>> things up more.   Our main issue is our precommit task (rake  
>>>>> cruise
>>>> The factory method (or attribute_helper) still hits the  
>>>> database.  I
>>>> don't see it as any sort of performance gain. In fact, I've even
>>>> developed a plugin around the Factory idea, and it was only when I
>>>> started using it in all of my tests that the speed really  
>>>> started to
>>>> affect me (I was using mocking/stubbing, with much frustration  
>>>> prior
>>>> to that point).  But to me it's pretty clear the plugin (or the
>>>> factory) is not the problem - the hit is the database.  DHH saw  
>>>> this
>>>> hit, and since they were using fixtures,he found that creating the
>>>> fixtures, and then wrapping each test in a transaction was a huge
>>>> performance gain.  I wonder if the same would be true with setups/
>>>> before(:each)...
>>>> The obvious thing to do to solve the performance problem is to  
>>>> remove
>>>> the hit to the database.  The question is: At what level of
>>>> abstraction should this be done?  The one camp (which would include
>>>> fellows like Jay Fields), would mock/stub everything they don't
>>>> write.  For me, I see testing as more than testing - it's the
>>>> documentation to my code which never lies to me (this documentation
>>>> is so good, that I can give it to my boss, who is not a  
>>>> programmer).
>>> I think you'd be far better served by user stories in this case.   
>>> The
>>> user story is the tool we use to communicate with non-geeks, be it
>>> customers or our boss.  Low-level specs fall squarely in the  
>>> developer
>>> realm, as far as I'm concerned.  They're easy to read because we  
>>> don't
>>> like to torture ourselves.  I think that they're fantastic for
>>> communicating intent with other developers, but that's where it  
>>> ends.
>>> So, when you move documentation as a goal from specs over to  
>>> stories,
>>> that frees specs to take advantage of techniques like mocks to help
>>> with your design.  Documentation is a pleasant side effect of  
>>> BDD, not
>>> a goal.
>> Documentation is certainly a goal of BDD (Just not necessarily for
>> the business user - as you pointed out).  I don't write my specs to
>> be understandable by the business user, although it just often
>> happens that they are (at least in this rails context, since even
>> most of the fields of the database are more-or-less user facing).
>> The point that I was trying to hammer home is that mocking/stubbing
>> adds a lot of extra noise to a rails test, where it doesn't in the
>> other projects that I've worked on.
> Well, one of the main parts of programming is making tradeoffs.
> You're probably not going to run 2500 tests that hit the db in under
> 30 seconds.  So you either don't hit the db or you don't run all the
> tests all the time.  You can certainly change autotest to not
> automatically run the full suite after you go red->green.  Only have
> it run the specs for the files you changed.
> Break your specs into fast and slow groups.  Run the fast ones
> continuously and the slow ones left often.  Work on moving one or two
> slow specs into the fast group every day.  You'll probably discover
> some techniques that you can share with all of us :)

My current thought is this:

1. Use sqlite in memory database when developing (but use the CI  
server to run the specs against mysql)
2. Start breaking up the spec files into more managable groups.   
Right now 100 of our 500 specs are in one file (the user_spec.rb).   
It's a social networking app, so it makes sense that most of the  
specs would be centered around the user.  But the description blocks  
should show some sort of logical grouping, which can then be split up  
into different spec files (user_with_items_spec.rb,  
user_with_comments_spec.rb, and so on).
3. Use autotest (or a custom tool) to rerun this one file that I'm  
changing.  It's pretty obvious that the mapping to autotest is flawed  
(Dave Astels has talked about the flaws of the one-to-one mapping of  
spec file to lib file (or app) several times).  Maybe some sort of  
tool which could inflect that any change to the  
user_with_items_spec.rb should rerun with a change either to the User  
or to the Item model, as well as the spec file.

> As far as mocks adding a lot of extra noise, I find in general that
> when mocks are painful for me it's because my code is too tightly
> coupled.  Just like my body tells me something is wrong through pain,
> so does my code.

Uhum...I don't think It's my code that's too tightly coupled (*clears  
throat* rails).  Seriously, though, I tend to agree with you - but  
here I don't have an option of rewriting rails test-first to be more  

>> But whether or not you call the specs "stories" or "integration
>> specs" (it is, after all, all a continuum of what is considered an
>> "integration" test), I still have no idea how to run them any faster.
> Well, I don't want to beat this to death, but I think it's dangerous
> to look at specs and stories as simply different degrees of the same
> concept.  The fundamental difference is that you should not change a
> story without consulting a customer, and you should not have to
> consult a customer to change a spec.  Stories represent assumptions
> and business value.  Assumptions are learned through conversations
> with the customer, and only the customer can define business value.
> Specs are merely a tool to help us generate that business value, and
> velocity goes way down if we have to worry that changing some specs
> will impact the customer in some way.  Worse, if we don't worry about
> it, we may end up changing some assumptions around and thus deliver
> the wrong product.

I agree with what you've said. But with rails (and a fat model), the  
specs for the model almost sound like a user story (albeit, not a  
full stack one).  Here is a tiny part of the specdoc which I sent to  
my boss:

The User
- should only find the one item which is not ignored, if there are
two items in the queue, and one is ignored

- should only find one item in the review queue, if there are two
items in the queue, but only one is ignored

- should see all items in the review queue, if none have yet been
ignored (and two are in the queue)

- should not be allowed to ignore items from review queue if he has
already reviewed 20 items
- should raise an error if a nil item is given to be skipped
- should raise an error if no item is given to be skipped
- should not skip the item if the user is not allowed to skip the item

Obviously you can see I didn't write these things for him - (the ones  
at the end prove it.  A "nil item" doesn't mean anything to my boss.   
But the other ones clearly show business value.  Once again, (some of  
these) specs just happen to be things which my boss can read.  I  
wrote those specs without mocking and stubbing, but they could as  
just as well been mocked/stubbed to death.


More information about the rspec-users mailing list