[Rake-devel] An official pull request for auto-parallelizing with -j.

James M. Lawrence quixoticsycophant at gmail.com
Sun Apr 12 23:22:09 EDT 2009

Subtitle: My sales pitch to end the Drake project.

Drake and its engine, CompTree, have been slimmed down since their
beginnings.  No more IPC forking for CompTree, no more utility libs,
no more duplication of CompTree inside Rake -- just the meat.

CompTree has been running well and its API is minimal and stable.  No
bugs of which I am aware.  I am confident about using it as a

Drake had just one bug, the result of prerequisites being
context-sensitive (this behavior did not even cross my mind).  The
rubinius folks use Drake, last I heard.

Some remarks about the pull request:
* Just one git commit -- all noise has been removed.
* No effective changes to the default single-threaded mode.
* Rather small, code-wise (especially when refactoring subtracted)
* Passes all unit tests in single-threaded mode.
* Passes all unit tests in parallel mode.
* Adds parallel-mode-specific tests.
* Full documentation: adds a note in rakefile.rdoc with a link to the
  new parallel.rdoc.  Update command_line_usage.rdoc.
* The changes to rake.rb are minor and straightforward:
  - Add -j option.
  - Add a cache for accumulating tasks with a dry run.
  - Add hooks to rake/parallel.rb
  - Add a lock to prevent invoke inside invoke (can't have cake and
    eat it too--either hand off control to the parallelizer or don't).
* Add rake/parallel.rb: 79 lines of code, 100% coverage.  The
  interface to comp_tree lies here and only here.
* lazy 'require': rake/parallel.rb and comp_tree are included only
  when threads > 1.
* Requires comp_tree gem.  I am willing to embed a copy of CompTree
  inside Rake if that is preferred instead.

Some remarks about CompTree:
* 257 lines of code, 100% coverage.
* Has a grand total of 5 methods in the API; Rake uses 3 of them.
* Unlikely to change in any significant way; similar to a math library.
* The core algorithm not changed since its inception, and its
  implementation has only undergone cosmetic changes.
* Tests have pounded on a wide variety of graph sizes and shapes with
  small to large numbers of threads.  No deadlocks, race conditions,
  or other thread problems have been discovered.
* No longer sets Thread.abort_on_exception (though the user is free to
  set it).

Some remarks about auto-parallelizing:
* It gives Rake a little more prestige.  More wealth and fame for Jim.
* Ever greater numbers of CPU cores are becoming available.  Rake has
  to jump on the train eventually; might as well be now.


Kind Regards,
James M. Lawrence

More information about the Rake-devel mailing list