[Rake-devel] An official pull request for auto-parallelizing with -j.

Hongli Lai hongli at phusion.nl
Mon Apr 13 04:29:15 EDT 2009

On Mon, Apr 13, 2009 at 5:22 AM, James M. Lawrence
<quixoticsycophant at gmail.com> wrote:
> Subtitle: My sales pitch to end the Drake project.
> Drake and its engine, CompTree, have been slimmed down since their
> beginnings.  No more IPC forking for CompTree, no more utility libs,
> no more duplication of CompTree inside Rake -- just the meat.
> CompTree has been running well and its API is minimal and stable.  No
> bugs of which I am aware.  I am confident about using it as a
> dependency.
> Drake had just one bug, the result of prerequisites being
> context-sensitive (this behavior did not even cross my mind).  The
> rubinius folks use Drake, last I heard.
> Some remarks about the pull request:
> * Just one git commit -- all noise has been removed.
> * No effective changes to the default single-threaded mode.
> * Rather small, code-wise (especially when refactoring subtracted)
> * Passes all unit tests in single-threaded mode.
> * Passes all unit tests in parallel mode.
> * Adds parallel-mode-specific tests.
> * Full documentation: adds a note in rakefile.rdoc with a link to the
>  new parallel.rdoc.  Update command_line_usage.rdoc.
> * The changes to rake.rb are minor and straightforward:
>  - Add -j option.
>  - Add a cache for accumulating tasks with a dry run.
>  - Add hooks to rake/parallel.rb
>  - Add a lock to prevent invoke inside invoke (can't have cake and
>    eat it too--either hand off control to the parallelizer or don't).
> * Add rake/parallel.rb: 79 lines of code, 100% coverage.  The
>  interface to comp_tree lies here and only here.
> * lazy 'require': rake/parallel.rb and comp_tree are included only
>  when threads > 1.
> * Requires comp_tree gem.  I am willing to embed a copy of CompTree
>  inside Rake if that is preferred instead.
> Some remarks about CompTree:
> * 257 lines of code, 100% coverage.
> * Has a grand total of 5 methods in the API; Rake uses 3 of them.
> * Unlikely to change in any significant way; similar to a math library.
> * The core algorithm not changed since its inception, and its
>  implementation has only undergone cosmetic changes.
> * Tests have pounded on a wide variety of graph sizes and shapes with
>  small to large numbers of threads.  No deadlocks, race conditions,
>  or other thread problems have been discovered.
> * No longer sets Thread.abort_on_exception (though the user is free to
>  set it).
> Some remarks about auto-parallelizing:
> * It gives Rake a little more prestige.  More wealth and fame for Jim.
> * Ever greater numbers of CPU cores are becoming available.  Rake has
>  to jump on the train eventually; might as well be now.
> http://github.com/quix/rake/tree/mainline-merge-candidate

Awesome! I've made Phusion Passenger's Rakefile thread-safe (I was
using chdir before) in order to test your fork. Strangely the results
are not as spectacular as I've hoped:

OS X with Intel Core 2 Duo, Ruby 1.8:
-j 1: 1 minute 4 seconds
-j 2: 56 seconds
-j 4: 57 seconds

I thought that it might have something to do with Ruby 1.8's green
threading, although I'm pretty sure that 1.8 is able to preempt
system(). So I tried Ruby 1.9:
-j 1: 1 minute 2 seconds
-j 4: 56 seconds

I'm not sure whether this is a problem in your Rake fork or in my
Rakefile. It would be nice if there's a way to output the parallelized
dependency tree so that I can see whether I did something wrong.

On the bright side, compilation succeeded so this probably means that
your code is mostly bug-free.

Hongli Lai
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: info at phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

More information about the Rake-devel mailing list