[rspec-users] How detailed should error scenarios be?

[Ben Mabey]

Ben Mabey wrote:
> Nathan Benes wrote:
>> I'm fairly new to cucumber and rspec but so far am falling in love with
>> both.  I've read up on several different articles concerning these
>> testing tools including the 'beta' BDD rspec/cucumber book.
>>
>> I saw this thread here:  http://www.ruby-forum.com/topic/183428
>>
>> Which concerns how small or detailed specs and scenarios should be - and
>> the basic gist was to keep scenarios at a higher level and leave the
>> detailed form
>> fields etc to rspec specs.
>>
>> This was all wonderfully insightful but it brings me to another
>> question.  How detailed (or should they be included in cucumber tests at
>> all?) should the error path be?  "Happy paths" are great to test, but
>> it's also necessary to test error paths so that users aren't
>> encountering rails stack traces, empty feedback, etc.
>>
>> Should there be one scenario per "empty" field?  Should there be one
>> scenario per validation check per field?  Should it be condensed to a
>> single scenario that attempts to encompass all error paths?
>>
>> I have one specific, seemingly overly complicated scenario that attempts
>> to go the one scenario per validation check per field route:
>>
>>   Scenario Outline: Add account with invalid fields
>>     Given I am logged in as BudgetTest
>>     And I am on the New Account page
>>     When I fill in the form for account "<account_name>" with valid data
>>     And I submit the data "<value1>" for the "<field_type1>" field,
>> "<field_name1>"
>>     And I submit the data "<value2>" for the "<field_type2>" field,
>> "<field_name2>"
>>     And I press "Add Account"
>>     Then I should see the rendered template for "the new account page"
>>     And I should see an error indicating "<field_name>" was "<error>"
>>
>>
>> I've removed the Scenarios:  blocks because they would wordwrap and look
>> terrible/undreadable.  Following this is two sets of scenarios:
>>
>> Scenarios: missing required fields
>>
>> Scenarios: submitting bad data
>>
>> Some of the fields compare data with each other to determine validity
>> which is why there's two data entries in the scenario outline.  If the
>> second is left blank then the defaults that were set in "When I fill in
>> the form..." are used for it.  Each "Scenarios" block contains a table
>> with allll of the fields defined by <> in the outline.  As you can see,
>> it seems to me to be overly complicated, overly verbose, and perhaps
>> doing more than it should be.
>>
>> I think maybe this test goes overboard...but what level of detail is
>> good for error-path testing?
>>   
>
> Hi Nathan,
> I think testing all of the validations from Cucumber is going 
> overboard *unless* the customer wants to see them there.  I have done 
> something like that but it was when I was writing my own validation 
> logic (in this case I wasn't even using AR or another standard 
> library). It was a nice way of showing the customer what were and were 
> not valid values.  However, if you are using ActiveRecord to do 
> standard validations then I see little value in checking each 
> validation separately.  It will take a long time to run and it seems 
> like it would really not add much value in terms of regression 
> catching beyond what the fast model-level code examples would provide. 
> So, like I said, the only value I would see in that would be for a 
> customer to see that it is working as they want it to.
>
> In general, I like to keep happy paths in Cucumber features along with 
> some special error cases.  Driving though the entire stack is a 
> double-edged sword.  It provides great regression testing and is an 
> excellent way to frame scenarios for customers.  However, the number 
> of execution paths you can execute from such a high level is 
> inherently limited... the number of objects and all of there potential 
> states in concert with one another results in a combinatorial 
> explosion of number of test cases required to test everything.  That 
> is why isolated specs on a lower level are still very valuable and for 
> me provide a great deal of confidence when used along side some 
> happy-path full-stack tests.  (And even then you can never test 
> everything...)
>
> Thats my current take on things at least... HTH,
>
> Ben
>


BTW, please use the cucumber mailing list for cucumber related questions:

http://groups.google.com/group/cukes

-Ben