[Rake-devel] wish: better support for multi-threaded builds

Mark Watson watsonmw at gmail.com
Thu Aug 14 15:14:39 EDT 2008

We use a thread pool version of the multitask for building some c/cpp
libraries.  Works fairly well for this simple case, but does loose
some cycles when one of threads blocks waiting on another thread to

I would still like to see a 'make -j{num_threads}' like functionality
though.  Sure it would require more care in writing the rakefile to
insure that all the dependencies are setup correctly.  But by default
rake could use the existing task invoke, and only when the minus j
option is specified would a task queue be used to execute the tasks in
parellel.  This would insure that existing rakefiles worked so long as
they weren't executed with rake -j.


2008/8/13 Ittay Dror <ittay.dror at gmail.com>:
> Hi,
> There are several issues with using MultiTask to parallelize the build:
> 1. If some top level tasks run in parallel, and each of them recursively
> runs other tasks, and one of the bottom tasks fail, it is impossible to stop
> the other tasks, short of a very ugly abort of all threads.
> 2. Tasks that run in parallel can't tell when another task's execution has
> failed. They may read a wrong timestamp from the failed task.
> 3. Threads are created per prerequisite task, rather than a fixed number
> (based initially on the number of cores/cpus), which causes thrashing
> 4. Even if a thread pool will be utilized, dependency information is still
> hard to take into account. Imagine a task has 2 prerequisites, where one
> depends on the other. Adding the tasks into a thread pool, they may be
> invoked in two different threads, but one waits on the other so the thread
> is not utilized. Maybe add a  "distance" method which calculates how far one
> task if from the other in dependency (adding nil if not dependent), so when
> adding tasks to a queue, they are added to the queue where the current tasks
> have the minimal distance (nil being infinite).
> Ittay
> --
> --
> Ittay Dror <ittay.dror at gmail.com>
> _______________________________________________
> Rake-devel mailing list
> Rake-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rake-devel

More information about the Rake-devel mailing list