[rspec-users] How detailed should error scenarios be?
ben at benmabey.com
Fri Jul 24 17:48:31 EDT 2009
Ben Mabey wrote:
> Nathan Benes wrote:
>> I'm fairly new to cucumber and rspec but so far am falling in love with
>> both. I've read up on several different articles concerning these
>> testing tools including the 'beta' BDD rspec/cucumber book.
>> I saw this thread here: http://www.ruby-forum.com/topic/183428
>> Which concerns how small or detailed specs and scenarios should be - and
>> the basic gist was to keep scenarios at a higher level and leave the
>> detailed form
>> fields etc to rspec specs.
>> This was all wonderfully insightful but it brings me to another
>> question. How detailed (or should they be included in cucumber tests at
>> all?) should the error path be? "Happy paths" are great to test, but
>> it's also necessary to test error paths so that users aren't
>> encountering rails stack traces, empty feedback, etc.
>> Should there be one scenario per "empty" field? Should there be one
>> scenario per validation check per field? Should it be condensed to a
>> single scenario that attempts to encompass all error paths?
>> I have one specific, seemingly overly complicated scenario that attempts
>> to go the one scenario per validation check per field route:
>> Scenario Outline: Add account with invalid fields
>> Given I am logged in as BudgetTest
>> And I am on the New Account page
>> When I fill in the form for account "<account_name>" with valid data
>> And I submit the data "<value1>" for the "<field_type1>" field,
>> And I submit the data "<value2>" for the "<field_type2>" field,
>> And I press "Add Account"
>> Then I should see the rendered template for "the new account page"
>> And I should see an error indicating "<field_name>" was "<error>"
>> I've removed the Scenarios: blocks because they would wordwrap and look
>> terrible/undreadable. Following this is two sets of scenarios:
>> Scenarios: missing required fields
>> Scenarios: submitting bad data
>> Some of the fields compare data with each other to determine validity
>> which is why there's two data entries in the scenario outline. If the
>> second is left blank then the defaults that were set in "When I fill in
>> the form..." are used for it. Each "Scenarios" block contains a table
>> with allll of the fields defined by <> in the outline. As you can see,
>> it seems to me to be overly complicated, overly verbose, and perhaps
>> doing more than it should be.
>> I think maybe this test goes overboard...but what level of detail is
>> good for error-path testing?
> Hi Nathan,
> I think testing all of the validations from Cucumber is going
> overboard *unless* the customer wants to see them there. I have done
> something like that but it was when I was writing my own validation
> logic (in this case I wasn't even using AR or another standard
> library). It was a nice way of showing the customer what were and were
> not valid values. However, if you are using ActiveRecord to do
> standard validations then I see little value in checking each
> validation separately. It will take a long time to run and it seems
> like it would really not add much value in terms of regression
> catching beyond what the fast model-level code examples would provide.
> So, like I said, the only value I would see in that would be for a
> customer to see that it is working as they want it to.
> In general, I like to keep happy paths in Cucumber features along with
> some special error cases. Driving though the entire stack is a
> double-edged sword. It provides great regression testing and is an
> excellent way to frame scenarios for customers. However, the number
> of execution paths you can execute from such a high level is
> inherently limited... the number of objects and all of there potential
> states in concert with one another results in a combinatorial
> explosion of number of test cases required to test everything. That
> is why isolated specs on a lower level are still very valuable and for
> me provide a great deal of confidence when used along side some
> happy-path full-stack tests. (And even then you can never test
> Thats my current take on things at least... HTH,
BTW, please use the cucumber mailing list for cucumber related questions:
More information about the rspec-users