[rspec-users] Quickcheck testing framework

Joseph Wilk josephwilk at me.com
Tue May 18 16:10:27 EDT 2010

On 17 May 2010, at 11:38, David Chelimsky <dchelimsky at gmail.com> wrote:

> On May 16, 2010, at 11:10 PM, Scott Taylor wrote:
>> On May 16, 2010, at 8:13 PM, David Chelimsky wrote:
>>> On May 16, 2010, at 12:54 PM, Scott Taylor wrote:
>>>> Hey all,
>>>> I'm wondering if anyone has any experience with an automated test- 
>>>> case generation tool like Quickcheck (for erlang/haskell). I'd be  
>>>> interested in hearing any impressions, war stories, or dev  
>>>> workflows regarding a tool like this.  Talking off list to David  
>>>> C, he suggested that it might be a complimentary tool to a TDD/ 
>>>> BDD framework like rspec.

This is something I've been playing around with in Cucumber for quite  
a while. My main thought is I want to make more use of dead CPU time.  
When I'm sleeping I want my tests and system being exercised. What I'm  
thinking is a tool in Cucumber a little like heckle. Mutating the  
matches/inputs (which in cucumber represent regexp matches) and  
examing the output.

A cucumber test can be seen as a black box with inputs we can prod at  
and observe the output.

What I want out of this is a report which shows me failures and what  
inputs where used.

In order to prevent a sprawl of failures it would be useful to derive  
from the failures rules which describe a group of failing tests.

I.e with int between 1 and 100 test failed.

I think of this

This is a different usecase to Rspec but thought some of my thoughts  
might be useful.

>>> My thinking here is that it could be useful to drive out an  
>>> initial implementation using TDD, and at the point we think we've  
>>> got the solution we want, add something quickcheck-like to try to  
>>> poke holes in it. I'd probably then add new examples if any cases  
>>> I hadn't considered were revealed through this process.
>> Have you watched John Hughes' presentation on the matter?
>> http://video.google.com/videoplay?docid=4655369445141008672#
> I haven't yet. I'll give it a look-see later today.
>> It's sort of interesting that he won't do any TDD - he'll let the  
>> reduction process generate the "minimum" test case, and go from  
>> there (that's not explicitly stated in that video, although I'm  
>> pretty sure I've heard him say it before).
>> If I had a tool like this, I'm guessing I'd probably have a  
>> workflow like the following:
>> 1. use the random test case generator, and fix any issues that were  
>> obvious.
>> 2. If something wasn't obvious, I'd go and write a test case for in  
>> a more traditional testing tool (rspec).  I often use the debugger  
>> in conjunction with the spec runner, running the one test case with  
>> a debugger statement at the start of the test case.
>> 3. Any regressions would (obviously) happen in the traditional tool.
>> The big win with a tool like this is not testing boundary cases,  
>> it's in having the tool "write" the test cases for you.  OTOH, I  
>> wonder if the simplicity of the implementation would be sacrificed  
>> when taking this approach.
> My guess is that it would.
>> Another drawback - I have no idea how such a tool would integrate  
>> with a build server.
> What integration point would there need to be? It's just Ruby.
>>>> It appears as though there is a similar project out there for  
>>>> ruby named rushcheck (http://rushcheck.rubyforge.org/).
>>> It's up on github too: http://github.com/hayeah/rushcheck. Same  
>>> guy has this too: http://github.com/hayeah/rantly - random data  
>>> generator - looks like you could do stuff like:
>>> Rantly.new.each(100) do
>>> thing.method_that_accepts_a_string(string).should have_some_quality
>>> end
>> There's a blog post about the library here, if anyone is interested:
>> http://www.metacircus.com/hacking/2009/04/10/look-ma-no-monads.html
>> I've been thinking about integrating a port of the ruby library  
>> faker into scriptcheck, the javascript testing tool I've been  
>> working on:
>> http://github.com/Marak/Faker.js
>> http://github.com/smtlaissezfaire/scriptcheck
>>> This would cause 100 random strings to be generated and passed to  
>>> thing.method_that_accepts_a_string. Assuming the matcher verifies  
>>> some set of rules about the outcomes,  you've basically got quick  
>>> check.
>> Yeah, pretty much.  One issue, though, is that you don't want to  
>> hard code the number of random generations.
> Why not? Wouldn't it make sense to have smaller numbers in some  
> cases and larger ones in others?
>> You'll also want a convenient way to run just one given test case  
>> easily (which rspec already has). You'll probably also want to  
>> separate these random generation tests from the rest of your tests.
> Exactly! This is what I had in mind when I said "at the point we  
> think we've got the solution we want, add something quickcheck-like  
> to try to poke holes in it." The steps would be:
> 1. Drive out minimal implementation with specs
> 2. Write some quickcheck-ish tests in a separate location
> 3. Run them
> 4. If there are any failures, use them to evaluate and enhance the  
> specs that I'd already written
> This would really amplify the distinction between specs and tests.  
> Plus, the tests would be indirectly testing the specs as much as  
> they are testing the implementation. Of course, this is all  
> theoretical. If we could just use quickcheck and still get all the  
> documentation and implementation-driving benefits of TDD, I'd  
> probably move in that direction myself :)
>> Hitting a database 1000 times for one test is going to be costly.
> If we used the process I just outlined, we could run the specs using  
> autotest (ironic), and only run the tests on demand and on the CI  
> server.
>> Now that I'm thinking about it, it might make a ton of sense in  
>> languages like erlang or haskell where everything is functional  
>> because those languages lend themselves to parallelization since  
>> there are no shared resources.
>> Regards,
>> Scott
> _______________________________________________
> rspec-users mailing list
> rspec-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rspec-users

More information about the rspec-users mailing list