[Rake-devel] Comments about parallel building

Heath Kehoe hkehoe at budcat.com
Wed May 19 14:00:53 EDT 2010


For our project one of the top goals was minimizing build times; so 
being able to invoke compilers/linkers in parallel to take advantage of 
multiple-core computers was a necessity.

(Since finishing my implementation, I have learned about drake. I intend 
to take a look at that as a possible alternative approach.)

The user specifies "j=N" on the command line, similar to make's -jN. 
When N>1, tasks are created as MultiTasks. Since FileTasks aren't 
MultiTasks, when there's a library or application, a MultiTask is 
created that has all the object files as prerequisistes; then the 
library/app includes that MultiTask in its prerequisites. This allows 
the object files to compile in parallel.

Since MultiTasks run all their prereqs at the same time, I made a custom 
version of the 'sh' method which synchronizes using a semaphore; that 
semaphore is initialized with the j=N value that the user specified. 
That way, only N commands will be invoked at once.

Recently, I tried updating to ruby 1.9.1. My MultiTask approach fell 
down because it created too many threads. 1.9 uses 'real' threads which 
consume more resources than 1.8's 'fake' (but very lightweight) threads. 
When I modified MultiTask to spawn threads in batches to limit their 
number, it worked but it was many times slower than before, probably due 
to the additional overhead of creating/joining lots of OS threads.

So we're sticking with ruby 1.8 for now. As I said, perhaps drake's 
approach will be better for 1.9, I'll try that out sometime.

-heath

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________


More information about the Rake-devel mailing list