[rspec-users] Quickcheck testing framework
josephwilk at me.com
Tue May 18 16:10:27 EDT 2010
On 17 May 2010, at 11:38, David Chelimsky <dchelimsky at gmail.com> wrote:
> On May 16, 2010, at 11:10 PM, Scott Taylor wrote:
>> On May 16, 2010, at 8:13 PM, David Chelimsky wrote:
>>> On May 16, 2010, at 12:54 PM, Scott Taylor wrote:
>>>> Hey all,
>>>> I'm wondering if anyone has any experience with an automated test-
>>>> case generation tool like Quickcheck (for erlang/haskell). I'd be
>>>> interested in hearing any impressions, war stories, or dev
>>>> workflows regarding a tool like this. Talking off list to David
>>>> C, he suggested that it might be a complimentary tool to a TDD/
>>>> BDD framework like rspec.
This is something I've been playing around with in Cucumber for quite
a while. My main thought is I want to make more use of dead CPU time.
When I'm sleeping I want my tests and system being exercised. What I'm
thinking is a tool in Cucumber a little like heckle. Mutating the
matches/inputs (which in cucumber represent regexp matches) and
examing the output.
A cucumber test can be seen as a black box with inputs we can prod at
and observe the output.
What I want out of this is a report which shows me failures and what
inputs where used.
In order to prevent a sprawl of failures it would be useful to derive
from the failures rules which describe a group of failing tests.
I.e with int between 1 and 100 test failed.
I think of this
This is a different usecase to Rspec but thought some of my thoughts
might be useful.
>>> My thinking here is that it could be useful to drive out an
>>> initial implementation using TDD, and at the point we think we've
>>> got the solution we want, add something quickcheck-like to try to
>>> poke holes in it. I'd probably then add new examples if any cases
>>> I hadn't considered were revealed through this process.
>> Have you watched John Hughes' presentation on the matter?
> I haven't yet. I'll give it a look-see later today.
>> It's sort of interesting that he won't do any TDD - he'll let the
>> reduction process generate the "minimum" test case, and go from
>> there (that's not explicitly stated in that video, although I'm
>> pretty sure I've heard him say it before).
>> If I had a tool like this, I'm guessing I'd probably have a
>> workflow like the following:
>> 1. use the random test case generator, and fix any issues that were
>> 2. If something wasn't obvious, I'd go and write a test case for in
>> a more traditional testing tool (rspec). I often use the debugger
>> in conjunction with the spec runner, running the one test case with
>> a debugger statement at the start of the test case.
>> 3. Any regressions would (obviously) happen in the traditional tool.
>> The big win with a tool like this is not testing boundary cases,
>> it's in having the tool "write" the test cases for you. OTOH, I
>> wonder if the simplicity of the implementation would be sacrificed
>> when taking this approach.
> My guess is that it would.
>> Another drawback - I have no idea how such a tool would integrate
>> with a build server.
> What integration point would there need to be? It's just Ruby.
>>>> It appears as though there is a similar project out there for
>>>> ruby named rushcheck (http://rushcheck.rubyforge.org/).
>>> It's up on github too: http://github.com/hayeah/rushcheck. Same
>>> guy has this too: http://github.com/hayeah/rantly - random data
>>> generator - looks like you could do stuff like:
>>> Rantly.new.each(100) do
>>> thing.method_that_accepts_a_string(string).should have_some_quality
>> There's a blog post about the library here, if anyone is interested:
>> I've been thinking about integrating a port of the ruby library
>> working on:
>>> This would cause 100 random strings to be generated and passed to
>>> thing.method_that_accepts_a_string. Assuming the matcher verifies
>>> some set of rules about the outcomes, you've basically got quick
>> Yeah, pretty much. One issue, though, is that you don't want to
>> hard code the number of random generations.
> Why not? Wouldn't it make sense to have smaller numbers in some
> cases and larger ones in others?
>> You'll also want a convenient way to run just one given test case
>> easily (which rspec already has). You'll probably also want to
>> separate these random generation tests from the rest of your tests.
> Exactly! This is what I had in mind when I said "at the point we
> think we've got the solution we want, add something quickcheck-like
> to try to poke holes in it." The steps would be:
> 1. Drive out minimal implementation with specs
> 2. Write some quickcheck-ish tests in a separate location
> 3. Run them
> 4. If there are any failures, use them to evaluate and enhance the
> specs that I'd already written
> This would really amplify the distinction between specs and tests.
> Plus, the tests would be indirectly testing the specs as much as
> they are testing the implementation. Of course, this is all
> theoretical. If we could just use quickcheck and still get all the
> documentation and implementation-driving benefits of TDD, I'd
> probably move in that direction myself :)
>> Hitting a database 1000 times for one test is going to be costly.
> If we used the process I just outlined, we could run the specs using
> autotest (ironic), and only run the tests on demand and on the CI
>> Now that I'm thinking about it, it might make a ton of sense in
>> languages like erlang or haskell where everything is functional
>> because those languages lend themselves to parallelization since
>> there are no shared resources.
> rspec-users mailing list
> rspec-users at rubyforge.org
More information about the rspec-users