[rspec-users] How detailed should error scenarios be?

[Ben Mabey]

Nathan Benes wrote:
> I'm fairly new to cucumber and rspec but so far am falling in love with
> both.  I've read up on several different articles concerning these
> testing tools including the 'beta' BDD rspec/cucumber book.
>
> I saw this thread here:  http://www.ruby-forum.com/topic/183428
>
> Which concerns how small or detailed specs and scenarios should be - and
> the basic gist was to keep scenarios at a higher level and leave the
> detailed form
> fields etc to rspec specs.
>
> This was all wonderfully insightful but it brings me to another
> question.  How detailed (or should they be included in cucumber tests at
> all?) should the error path be?  "Happy paths" are great to test, but
> it's also necessary to test error paths so that users aren't
> encountering rails stack traces, empty feedback, etc.
>
> Should there be one scenario per "empty" field?  Should there be one
> scenario per validation check per field?  Should it be condensed to a
> single scenario that attempts to encompass all error paths?
>
> I have one specific, seemingly overly complicated scenario that attempts
> to go the one scenario per validation check per field route:
>
>   Scenario Outline: Add account with invalid fields
>     Given I am logged in as BudgetTest
>     And I am on the New Account page
>     When I fill in the form for account "<account_name>" with valid data
>     And I submit the data "<value1>" for the "<field_type1>" field,
> "<field_name1>"
>     And I submit the data "<value2>" for the "<field_type2>" field,
> "<field_name2>"
>     And I press "Add Account"
>     Then I should see the rendered template for "the new account page"
>     And I should see an error indicating "<field_name>" was "<error>"
>
>
> I've removed the Scenarios:  blocks because they would wordwrap and look
> terrible/undreadable.  Following this is two sets of scenarios:
>
> Scenarios: missing required fields
>
> Scenarios: submitting bad data
>
> Some of the fields compare data with each other to determine validity
> which is why there's two data entries in the scenario outline.  If the
> second is left blank then the defaults that were set in "When I fill in
> the form..." are used for it.  Each "Scenarios" block contains a table
> with allll of the fields defined by <> in the outline.  As you can see,
> it seems to me to be overly complicated, overly verbose, and perhaps
> doing more than it should be.
>
> I think maybe this test goes overboard...but what level of detail is
> good for error-path testing?
>   

Hi Nathan,
I think testing all of the validations from Cucumber is going overboard 
*unless* the customer wants to see them there.  I have done something 
like that but it was when I was writing my own validation logic (in this 
case I wasn't even using AR or another standard library). It was a nice 
way of showing the customer what were and were not valid values.  
However, if you are using ActiveRecord to do standard validations then I 
see little value in checking each validation separately.  It will take a 
long time to run and it seems like it would really not add much value in 
terms of regression catching beyond what the fast model-level code 
examples would provide. So, like I said, the only value I would see in 
that would be for a customer to see that it is working as they want it to.

In general, I like to keep happy paths in Cucumber features along with 
some special error cases.  Driving though the entire stack is a 
double-edged sword.  It provides great regression testing and is an 
excellent way to frame scenarios for customers.  However, the number of 
execution paths you can execute from such a high level is inherently 
limited... the number of objects and all of there potential states in 
concert with one another results in a combinatorial explosion of number 
of test cases required to test everything.  That is why isolated specs 
on a lower level are still very valuable and for me provide a great deal 
of confidence when used along side some happy-path full-stack tests.  
(And even then you can never test everything...)

Thats my current take on things at least... HTH,

Ben