[rspec-users] Quickcheck testing framework
dchelimsky at gmail.com
Mon May 17 06:38:41 EDT 2010
On May 16, 2010, at 11:10 PM, Scott Taylor wrote:
> On May 16, 2010, at 8:13 PM, David Chelimsky wrote:
>> On May 16, 2010, at 12:54 PM, Scott Taylor wrote:
>>> Hey all,
>>> I'm wondering if anyone has any experience with an automated test-case generation tool like Quickcheck (for erlang/haskell). I'd be interested in hearing any impressions, war stories, or dev workflows regarding a tool like this. Talking off list to David C, he suggested that it might be a complimentary tool to a TDD/BDD framework like rspec.
>> My thinking here is that it could be useful to drive out an initial implementation using TDD, and at the point we think we've got the solution we want, add something quickcheck-like to try to poke holes in it. I'd probably then add new examples if any cases I hadn't considered were revealed through this process.
> Have you watched John Hughes' presentation on the matter?
I haven't yet. I'll give it a look-see later today.
> It's sort of interesting that he won't do any TDD - he'll let the reduction process generate the "minimum" test case, and go from there (that's not explicitly stated in that video, although I'm pretty sure I've heard him say it before).
> If I had a tool like this, I'm guessing I'd probably have a workflow like the following:
> 1. use the random test case generator, and fix any issues that were obvious.
> 2. If something wasn't obvious, I'd go and write a test case for in a more traditional testing tool (rspec). I often use the debugger in conjunction with the spec runner, running the one test case with a debugger statement at the start of the test case.
> 3. Any regressions would (obviously) happen in the traditional tool.
> The big win with a tool like this is not testing boundary cases, it's in having the tool "write" the test cases for you. OTOH, I wonder if the simplicity of the implementation would be sacrificed when taking this approach.
My guess is that it would.
> Another drawback - I have no idea how such a tool would integrate with a build server.
What integration point would there need to be? It's just Ruby.
>>> It appears as though there is a similar project out there for ruby named rushcheck (http://rushcheck.rubyforge.org/).
>> It's up on github too: http://github.com/hayeah/rushcheck. Same guy has this too: http://github.com/hayeah/rantly - random data generator - looks like you could do stuff like:
>> Rantly.new.each(100) do
>> thing.method_that_accepts_a_string(string).should have_some_quality
> There's a blog post about the library here, if anyone is interested:
>> This would cause 100 random strings to be generated and passed to thing.method_that_accepts_a_string. Assuming the matcher verifies some set of rules about the outcomes, you've basically got quick check.
> Yeah, pretty much. One issue, though, is that you don't want to hard code the number of random generations.
Why not? Wouldn't it make sense to have smaller numbers in some cases and larger ones in others?
> You'll also want a convenient way to run just one given test case easily (which rspec already has). You'll probably also want to separate these random generation tests from the rest of your tests.
Exactly! This is what I had in mind when I said "at the point we think we've got the solution we want, add something quickcheck-like to try to poke holes in it." The steps would be:
1. Drive out minimal implementation with specs
2. Write some quickcheck-ish tests in a separate location
3. Run them
4. If there are any failures, use them to evaluate and enhance the specs that I'd already written
This would really amplify the distinction between specs and tests. Plus, the tests would be indirectly testing the specs as much as they are testing the implementation. Of course, this is all theoretical. If we could just use quickcheck and still get all the documentation and implementation-driving benefits of TDD, I'd probably move in that direction myself :)
> Hitting a database 1000 times for one test is going to be costly.
If we used the process I just outlined, we could run the specs using autotest (ironic), and only run the tests on demand and on the CI server.
> Now that I'm thinking about it, it might make a ton of sense in languages like erlang or haskell where everything is functional because those languages lend themselves to parallelization since there are no shared resources.
More information about the rspec-users