[Rake-devel] An official pull request for auto-parallelizing with -j.
hongli at phusion.nl
Mon Apr 13 04:29:15 EDT 2009
On Mon, Apr 13, 2009 at 5:22 AM, James M. Lawrence
<quixoticsycophant at gmail.com> wrote:
> Subtitle: My sales pitch to end the Drake project.
> Drake and its engine, CompTree, have been slimmed down since their
> beginnings. No more IPC forking for CompTree, no more utility libs,
> no more duplication of CompTree inside Rake -- just the meat.
> CompTree has been running well and its API is minimal and stable. No
> bugs of which I am aware. I am confident about using it as a
> Drake had just one bug, the result of prerequisites being
> context-sensitive (this behavior did not even cross my mind). The
> rubinius folks use Drake, last I heard.
> Some remarks about the pull request:
> * Just one git commit -- all noise has been removed.
> * No effective changes to the default single-threaded mode.
> * Rather small, code-wise (especially when refactoring subtracted)
> * Passes all unit tests in single-threaded mode.
> * Passes all unit tests in parallel mode.
> * Adds parallel-mode-specific tests.
> * Full documentation: adds a note in rakefile.rdoc with a link to the
> new parallel.rdoc. Update command_line_usage.rdoc.
> * The changes to rake.rb are minor and straightforward:
> - Add -j option.
> - Add a cache for accumulating tasks with a dry run.
> - Add hooks to rake/parallel.rb
> - Add a lock to prevent invoke inside invoke (can't have cake and
> eat it too--either hand off control to the parallelizer or don't).
> * Add rake/parallel.rb: 79 lines of code, 100% coverage. The
> interface to comp_tree lies here and only here.
> * lazy 'require': rake/parallel.rb and comp_tree are included only
> when threads > 1.
> * Requires comp_tree gem. I am willing to embed a copy of CompTree
> inside Rake if that is preferred instead.
> Some remarks about CompTree:
> * 257 lines of code, 100% coverage.
> * Has a grand total of 5 methods in the API; Rake uses 3 of them.
> * Unlikely to change in any significant way; similar to a math library.
> * The core algorithm not changed since its inception, and its
> implementation has only undergone cosmetic changes.
> * Tests have pounded on a wide variety of graph sizes and shapes with
> small to large numbers of threads. No deadlocks, race conditions,
> or other thread problems have been discovered.
> * No longer sets Thread.abort_on_exception (though the user is free to
> set it).
> Some remarks about auto-parallelizing:
> * It gives Rake a little more prestige. More wealth and fame for Jim.
> * Ever greater numbers of CPU cores are becoming available. Rake has
> to jump on the train eventually; might as well be now.
Awesome! I've made Phusion Passenger's Rakefile thread-safe (I was
using chdir before) in order to test your fork. Strangely the results
are not as spectacular as I've hoped:
OS X with Intel Core 2 Duo, Ruby 1.8:
-j 1: 1 minute 4 seconds
-j 2: 56 seconds
-j 4: 57 seconds
I thought that it might have something to do with Ruby 1.8's green
threading, although I'm pretty sure that 1.8 is able to preempt
system(). So I tried Ruby 1.9:
-j 1: 1 minute 2 seconds
-j 4: 56 seconds
I'm not sure whether this is a problem in your Rake fork or in my
Rakefile. It would be nice if there's a way to output the parallelized
dependency tree so that I can see whether I did something wrong.
On the bright side, compilation succeeded so this probably means that
your code is mostly bug-free.
Phusion | The Computer Science Company
E-mail: info at phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)
More information about the Rake-devel