[Rake-devel] An official pull request for auto-parallelizing with -j.
James M. Lawrence
quixoticsycophant at gmail.com
Sun Apr 12 23:22:09 EDT 2009
Subtitle: My sales pitch to end the Drake project.
Drake and its engine, CompTree, have been slimmed down since their
beginnings. No more IPC forking for CompTree, no more utility libs,
no more duplication of CompTree inside Rake -- just the meat.
CompTree has been running well and its API is minimal and stable. No
bugs of which I am aware. I am confident about using it as a
Drake had just one bug, the result of prerequisites being
context-sensitive (this behavior did not even cross my mind). The
rubinius folks use Drake, last I heard.
Some remarks about the pull request:
* Just one git commit -- all noise has been removed.
* No effective changes to the default single-threaded mode.
* Rather small, code-wise (especially when refactoring subtracted)
* Passes all unit tests in single-threaded mode.
* Passes all unit tests in parallel mode.
* Adds parallel-mode-specific tests.
* Full documentation: adds a note in rakefile.rdoc with a link to the
new parallel.rdoc. Update command_line_usage.rdoc.
* The changes to rake.rb are minor and straightforward:
- Add -j option.
- Add a cache for accumulating tasks with a dry run.
- Add hooks to rake/parallel.rb
- Add a lock to prevent invoke inside invoke (can't have cake and
eat it too--either hand off control to the parallelizer or don't).
* Add rake/parallel.rb: 79 lines of code, 100% coverage. The
interface to comp_tree lies here and only here.
* lazy 'require': rake/parallel.rb and comp_tree are included only
when threads > 1.
* Requires comp_tree gem. I am willing to embed a copy of CompTree
inside Rake if that is preferred instead.
Some remarks about CompTree:
* 257 lines of code, 100% coverage.
* Has a grand total of 5 methods in the API; Rake uses 3 of them.
* Unlikely to change in any significant way; similar to a math library.
* The core algorithm not changed since its inception, and its
implementation has only undergone cosmetic changes.
* Tests have pounded on a wide variety of graph sizes and shapes with
small to large numbers of threads. No deadlocks, race conditions,
or other thread problems have been discovered.
* No longer sets Thread.abort_on_exception (though the user is free to
Some remarks about auto-parallelizing:
* It gives Rake a little more prestige. More wealth and fame for Jim.
* Ever greater numbers of CPU cores are becoming available. Rake has
to jump on the train eventually; might as well be now.
James M. Lawrence
More information about the Rake-devel