Files | Admin

Notes:

Release Name: 0.8.5

Notes:
= Tryouts - v0.8 BETA

Tryouts is a high-level testing library (DSL) for your Ruby codes and command-line applications.

A tryout is made up of one of more drills. The return value of the drill block is compared to the expectations defined by one or more dreams. The goal is to achieve your dreams. 

=== Terminology

* Tryout: a set of drills (think: basketball tryouts)
* Drill: a single test. 
* Drill Sergeant: The class responsible for executing a drill.
* Dream: the expected outcome of a drill. There's always one or more dream per drill.

== Installation

Via Rubygems, one of:

    $ gem install tryouts
    $ gem install delano-tryouts

or via download:
* tryouts-latest.tar.gz[http://github.com/delano/tryouts/tarball/latest]
* tryouts-latest.zip[http://github.com/delano/tryouts/zipball/latest]


== Writing Tests

The examples below are a complete overview of Tryouts syntax. 

=== Testing Ruby Codes (:api)

    library :gibbler, "../path/to/gibbler/lib"

    tryouts "Common Usage", :api do

      # Define variables available only within
      # in the current tryout definition.
      set :local_variable, 3
      
      setup do
        # do stuff before all drills
      end
      
      clean do 
        # do stuff after all drills
      end
      
      # This drill block should return 3.
      drill "Maths R Us", local_variable do
        12 / 4
      end

      # You can specify a method to execute
      # on the return value of the drill block.
      drill "We want a symbol", :class, Symbol do
        Class.methods.first
      end

      # Dreams can also be specified explicitly which is
      # important b/c it's possible to specify multiple.
      dream :class, Array
      dream ['b', 'c', 'd']
      drill "Should return a list of 3" do
        letters = 'a'..'z'
        letters.to_a[1..local_variable]
      end

      # Drills can pass based on an exception too.
      dream :exception, NameError
      drill "Something failed" do
        raise NameError
      end

      # We can even put simple drills on a single line.
      drill 'Small, fast, and furious', 'Muggsy Bogues', :match, /Mug+sy Bogu?es/

    end


=== Testing Command-Line Applications (:cli)

You can execute drills on command-line applications. 
    
    command :rudy, "path/2/rudy"
    
    tryouts "Rudy CLI", :cli do
      
      # We have specified the name of the command above so 
      # we only need to specify the arguments in the drill.
      # $ rudy myaddress -e
      drill "can display my IP addresses", :myaddress, :e
      
      # The drill returns an Array of lines from STDOUT so
      # we can use Array#grep to check the command output.
      # If any lines match, this test will pass.
      dream :grep, /No machines running/
      drill "can call with no arguments"
      
      # Drills can also be defined with a block which gets
      # executed via Rye::Box#batch. Methods in the block
      # are executed as commands. The return value of the
      # block is considered the test output. 
      dream :match, /\d+\.\d+\.\d+\.\d+/
      drill "can execute a block of commands" do
        ret = rudy :q, :myaddress, :e
        ret.first
      end
      
    end

=== Running Performance Benchmarks (:benchmark)

You can also use Tryouts to run benchmarks. The tryouts method takes a second parameter which specifies which drill sergeant to use. Below we specify <tt>:benchmark</tt> so each drill is timed and executed multiple times.

   tryouts "Benchmark examples", :benchmark do
     
     # We create an Array and store it in a class
     # variable so it's available to other drills.
     drill "Create test array" do
       @@array = (1..100000).map { rand }
     end
     
     dream :mean, 3.0        # The mean should be <= 3.0 seconds
     dream :sdev, 0.1        # and the standard deviation <= 0.1
     drill("Array sort!") { @@array.dup.sort! }
     
     # You can also include a dream inline
     drill("Array sort", :mean, 3.0) { @@array.dup.sort }
     
     # The 3rd argument is the number of times to 
     # execute the drill block. The mean and sdev
     # are calculate based on all iterations. The
     # default is 5 and the maximum is 30.
     dream :sdev, 0.1, 15    
     drill("Array sort") { @@array.dup.sort }
      
   end

The drill blocks in a benchmark Tryout return Tryouts::Stats objects. See: Tryouts::Drill::Sergeant::Benchmark

== Running Tests

Tryouts comes with an executable called <tt>sergeant</tt>. This is the drill sergeant that tells you what percentage of your dreams come true.

    $ sergeant
    
    Gibbler Gazette                                                                

     Basic syntax with SHA1 
      "Object"                                                              PASS
      "Class"                                                               PASS
      ...
      "Knows when an Hash has changed"                                      PASS

    All 9 dreams came true 


The sergeant looks in the current working directory for a <tt>tryouts</tt> directory and will automatically load all files ending in <tt>_tryouts.rb</tt>. 

    $ ls -l tryouts/
    01_mixins_tryouts.rb
    10_syntax_tryouts.rb
    20_cli_tryouts.rb
    30_benchmark_tryouts.rb
    50_class_context_tryouts.rb
    
You can also specify specific files to load:

    $ sergeant any/file.rb

Verbose mode gives you some extra information, including the return values from the drills. The more v's, the more information:

    $ sergeant -v
    
In quiet mode, the sergeant returns only a PASS or FAIL message (in the case of a FAIL, it will also return a non-zero exit code): 

    $ sergeant -q
    PASS

== Screenshots

=== :api Tryouts

http://delano.github.com/tryouts/screens/tryouts-0.8-api.png

<i>from gibbler[http://github.com/delano/gibbler/blob/master/tryouts/10_basic_tryouts.rb]</i>


=== :cli Tryouts (verbose)

http://delano.github.com/tryouts/screens/tryouts-0.8-cli.png

<i>from tryouts[http://github.com/delano/tryouts/blob/master/tryouts/20_cli_tryouts.rb]</i>


=== :benchmark Tryouts (verbose)

http://delano.github.com/tryouts/screens/tryouts-0.8-benchmark.png

<i>from gibbler[http://github.com/delano/gibbler/blob/master/tryouts/80_performance_tryouts.rb]</i>

== News

=== 2009-07-20: New DSL keyword "set"

The "set" keyword has been introduced in 0.8.4 for defining variables available only to the Tryout definition they were declared in. See Example 1. 

=== 2009-07-01: Drills for command-line applications

Support for testing command-line applications has been re-introduced in 0.8. See example code further down in this README. 



== BETA Notice

Tryouts is very new (est. 2009-05-19) and has not been vetted by the scrutiny of time. In particular you can expect:

* The test definition syntax may change in future releases. 
* Unexpected errors. 
* Bugs! I love fixing bugs so if you find one let me know. 


== On Threads

Tryouts does some funky stuff to make it simple to write tests. This "funky stuff" means that this library is <em>not thread-safe at definition-time</em>. However, once all tryouts files are parsed (or in OO-syntax, once all objects are created), this class should be <em>thread-safe at drill-time</em>. 


== More Info

* Check out the sourcecodes[http://github.com/delano/tryouts]
* Read the rdocs[http://tryouts.rubyforge.org/]
* About Solutious[http://solutious.com/about/]
* Inspiration[http://www.youtube.com/watch?v=iB9nhyosDVs]


== Thanks

* Everyone at Utrecht.rb and Amsterdam.rb for putting up with my Ruby questions :]
* Sam Aaron (http://sam.aaron.name) for the early feedback. 
* Kalin Harvey for the encouragement.


== Credits

* Delano (@solutious.com)


== Related Projects

* Context[http://github.com/jeremymcanally/context/tree/master]
* Testy[http://github.com/ahoward/testy/tree/master]
* Expectations[http://expectations.rubyforge.org/]
* Zebra[http://github.com/giraffesoft/zebra/tree/master]

== License

See: LICENSE.txt


Changes: TRYOUTS, CHANGES * TODO: consider having set create var= methods too. * TODO: Support: "sergeant 30:52" * TODO: FIXED: output for quiet mode * TODO: Run clean after reports have been generated. See Gibbler 256 tryouts. * TODO: Load tryouts/setup.rb and tryouts/clean.rb * TODO: Support Hash syntax in set, set :name => value * TODO: Support tryouts/*_drills.rb and tryouts/*_tryout.rb * TODO: Clear stash between drills #### 0.8.5 (2009-10-07) ############################### NOTE: Benchmark tryouts must now use proc style dreams * CHANGE: Benchmark tryouts now display complex benchmark info (utime, stime, cutime, cstime, total, real) * CHANGE: Benchmark tryouts now run a brief warmup * CHANGE: Improved startup time (via autoload) #### 0.8.4 (2009-07-20) ############################### * CHANGE: "setup" and "clean" blocks now run in the same DrillContext instance as drills (so instance variables are now shared) * CHANGE: Non-fatal errors are now displayed only in verbose mode. * ADDED: "set" keyword inside tryout definitions. * ADDED: New dependency: attic #### 0.8.3 (2009-07-19) ############################### * FIXED: Updated gemspec #### 0.8.2 (2009-07-19) ############################### * CHANGE: Moved Drill and Reality to separate files. * CHANGE: dream blocks are now lazy executed just before the drill is executed. #### 0.8.1 (2009-07-14) ############################### * CHANGE: Fixed parentheses warning in tryouts.rb for Ruby 1.8 * CHANGE: rye is now loaded when running CLI tryouts. #### 0.8.0 (2009-07-01) ############################### NOTE: :cli drills are available again * CHANGE: Dream definition block no longer support the rcode, output, format, and emsg methods. Now: the return value of the block is used at the dream.output dream do "output value" end dream :format do "output value" end * CHANGE: Improved clarity of [nodream] error message * ADDED: Re-introduced :cli drill support. #### 0.7.4 (2009-06-29) ############################### * FIXED: The stash method is now available to :benchmark drills * FIXED: Specifying a single argument to drill with a block now correctly parses the arg as the expected dream output. * CHANGE: Differentiated non-fatal error messages for successful drills * CHANGE: Context sensitive message for LoadErrors in Tryouts#library #### 0.7.3 (2009-06-27) ############################### * ADDED: Display Tryouts group name for runtime errors. * ADDED: Improved LoadError error message * ADDED: Print error messages for drills that PASS. * ADDED: Better error handling when supplied tryouts file is not found #### 0.7.2 (2009-06-26) ############################### NOTE: You will need to make a syntax change to your tryouts. OLD: dream OUTPUT, :format NEW: dream :format, OUTPUT * CHANGE: Order of dream arguments is reversed! * CHANGE: Reduced CLI width to 79 #### 0.7.1 (2009-06-26) ############################### * FIXED: Updated manifest in gemspec * CHANGE: :cli testing is disabled indefinitely * ADDED: Found Muggsy Bogues #### 0.7.0 (2009-06-25) ############################### * FIXED: Stash wasn't being displayed in the drill report * CHANGE: CLI is now formatted to be 80 characters wide. * ADDED: Tryouts::Drill::Sergeant::Benchmark * ADDED: Tryouts::Stats * ADDED: new dream formats: :mean and :sdev which compare with <=. dream 10, :mean drill "do something" do; ...; end # stats.mean <= 10 * ADDED: new dream format: :proc dream lambda { |x| x.real < 0.1 }, :proc drill("array sort!") { @@array.dup.sort! } #### 0.6.3 (2009-06-25) ############################### NOTE: command testing (:cli) is still disabled. * FIXED: Now correctly append to existing Tryouts object * CHANGE: Renamed :regex format to :match * ADDED: Hella better runtime error handling * ADDED: Better support for quiet (-q) * ADDED: Verbose output now displays each dream vs reality * ADDED: Skipped drills now appear in the report * ADDED: new dream format: :ne (!=) #### 0.6.2 (2009-06-24) ############################### NOTE: command testing (:cli) is still disabled. * CHANGE: dream arguments are now ordered: format, output * ADDED: One-liner drill syntax * ADDED: new dream formats: :regex, :gt, :gte, :lt, :lte, :size * ADDED: Calls to xdrill now also clear the dream catcher * ADDED: Support for multiple dreams per drill #### 0.6.1 (2009-06-24) ############################### NOTE: command testing (:cli) is still disabled. * FIXED: JRuby is now supported * FIXED: Now correctly requiring sysinfo library * CHANGE: Don't require rye until a CLI sergeant is created. #### 0.6.0 (2009-06-24) ############################### NOTE: External dreams files are no longer supported. NOTE: command testing (:cli) is disabled. * CHANGE: Removed all mention of external dreams files. * CHANGE: dream method is not available inside drill block. * CHANGE: rcode, emsg, backtrace renamed to ecode, error, trace * ADDED: dream is now available above drill blocks. * ADDED: Better verbose and debugging output * ADDED: Cleaner test output * ADDED: drill success is now solely based on output. #### 0.5.1 (2009-06-13) ############################### * ADDED: dream method is now available inside drill block * ADDED: Drills that return true are assumed to pass #### 0.5.0 (2009-06-07) ############################### * FIXED: Fix for running drills without dreams and for specifying verbose without a command name * CHANGE: The executable has been renamed to 'sergeant' * CHANGE: Gave reality a default rcode (0) * CHANGE: Now using module_eval (was: class_eval) for setup and clean * ADDED: Displays percentage of failed dreams * ADDED: Drill stash! * ADDED: xdream and better handling for drills and dreams with no name #### 0.4.1 (2009-06-07) ############################### * CHANGE: The CLI output is no longer terrifyingly ugly #### 0.4.0 (2009-06-05) ############################### NOTE: Initial public release