[rspec-users] scenarios on production data

[Nick Hoffman]

On 2008-09-05, at 14:14, Jonathan Linowes wrote:
> it just seems to me that  while i'm running my tests on toy data  
> sets, it could be reassuring to see the same run on real production  
> data. Or, maybe not.
> I agree its not real scientific, but neither is the weather... :)   
> (aka, shit happens)

I know what you mean about it feeling reassuring to know that your  
tests/specs passed when run on production data. However, if your  
specs, scenarios, tests, etc cover all behaviours, situations, edge  
cases, etc, then you needn't worry. All is well.

You don't truly control what's in your production data. Thus, you'll  
be testing random things, and will obtain random results. While said  
results should pass (provided that your specs, scenarios, tests, etc  
are complete and all pass), the results won't give you any sort of  
quantifiable coverage other than "this set of production data passed".  
Is that actually useful?
-Nick