[Rake-devel] wish: better support for multi-threaded builds

Ittay Dror ittay.dror at gmail.com
Thu Aug 14 00:36:04 EDT 2008


There are several issues with using MultiTask to parallelize the build:
1. If some top level tasks run in parallel, and each of them recursively 
runs other tasks, and one of the bottom tasks fail, it is impossible to 
stop the other tasks, short of a very ugly abort of all threads.
2. Tasks that run in parallel can't tell when another task's execution 
has failed. They may read a wrong timestamp from the failed task.
3. Threads are created per prerequisite task, rather than a fixed number 
(based initially on the number of cores/cpus), which causes thrashing
4. Even if a thread pool will be utilized, dependency information is 
still hard to take into account. Imagine a task has 2 prerequisites, 
where one depends on the other. Adding the tasks into a thread pool, 
they may be invoked in two different threads, but one waits on the other 
so the thread is not utilized. Maybe add a  "distance" method which 
calculates how far one task if from the other in dependency (adding nil 
if not dependent), so when adding tasks to a queue, they are added to 
the queue where the current tasks have the minimal distance (nil being 


Ittay Dror <ittay.dror at gmail.com>

More information about the Rake-devel mailing list