[rspec-users] How detailed should error scenarios be?
ben at benmabey.com
Fri Jul 24 03:10:40 EDT 2009
Nathan Benes wrote:
> I'm fairly new to cucumber and rspec but so far am falling in love with
> both. I've read up on several different articles concerning these
> testing tools including the 'beta' BDD rspec/cucumber book.
> I saw this thread here: http://www.ruby-forum.com/topic/183428
> Which concerns how small or detailed specs and scenarios should be - and
> the basic gist was to keep scenarios at a higher level and leave the
> detailed form
> fields etc to rspec specs.
> This was all wonderfully insightful but it brings me to another
> question. How detailed (or should they be included in cucumber tests at
> all?) should the error path be? "Happy paths" are great to test, but
> it's also necessary to test error paths so that users aren't
> encountering rails stack traces, empty feedback, etc.
> Should there be one scenario per "empty" field? Should there be one
> scenario per validation check per field? Should it be condensed to a
> single scenario that attempts to encompass all error paths?
> I have one specific, seemingly overly complicated scenario that attempts
> to go the one scenario per validation check per field route:
> Scenario Outline: Add account with invalid fields
> Given I am logged in as BudgetTest
> And I am on the New Account page
> When I fill in the form for account "<account_name>" with valid data
> And I submit the data "<value1>" for the "<field_type1>" field,
> And I submit the data "<value2>" for the "<field_type2>" field,
> And I press "Add Account"
> Then I should see the rendered template for "the new account page"
> And I should see an error indicating "<field_name>" was "<error>"
> I've removed the Scenarios: blocks because they would wordwrap and look
> terrible/undreadable. Following this is two sets of scenarios:
> Scenarios: missing required fields
> Scenarios: submitting bad data
> Some of the fields compare data with each other to determine validity
> which is why there's two data entries in the scenario outline. If the
> second is left blank then the defaults that were set in "When I fill in
> the form..." are used for it. Each "Scenarios" block contains a table
> with allll of the fields defined by <> in the outline. As you can see,
> it seems to me to be overly complicated, overly verbose, and perhaps
> doing more than it should be.
> I think maybe this test goes overboard...but what level of detail is
> good for error-path testing?
I think testing all of the validations from Cucumber is going overboard
*unless* the customer wants to see them there. I have done something
like that but it was when I was writing my own validation logic (in this
case I wasn't even using AR or another standard library). It was a nice
way of showing the customer what were and were not valid values.
However, if you are using ActiveRecord to do standard validations then I
see little value in checking each validation separately. It will take a
long time to run and it seems like it would really not add much value in
terms of regression catching beyond what the fast model-level code
examples would provide. So, like I said, the only value I would see in
that would be for a customer to see that it is working as they want it to.
In general, I like to keep happy paths in Cucumber features along with
some special error cases. Driving though the entire stack is a
double-edged sword. It provides great regression testing and is an
excellent way to frame scenarios for customers. However, the number of
execution paths you can execute from such a high level is inherently
limited... the number of objects and all of there potential states in
concert with one another results in a combinatorial explosion of number
of test cases required to test everything. That is why isolated specs
on a lower level are still very valuable and for me provide a great deal
of confidence when used along side some happy-path full-stack tests.
(And even then you can never test everything...)
Thats my current take on things at least... HTH,
More information about the rspec-users