[Rake-devel] Parallel tasks with Rake

Jim Weirich jim.weirich at gmail.com
Thu Sep 11 00:44:46 EDT 2008

On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote:

Hi James,

Saw your announcement on ruby-talk and want to say good job on getting  
drake out.  Now that 0.8.2 is released, I've taken some time to look  
at some of what you've done.  It looks impressive.

BTW, I'm copying this email to the Rake Development list.  I think it  
would be great to get this conversation on the list for everyone to  
participate.  You can sign up at http://rubyforge.org/mailman/listinfo/rake-devel

I've pulled the changes into a branch of rake and have begun playing  
with it.  I've not delved into the deeps of CompTree yet, but the  
changes to rake look pretty straight forward.

And of course, I have some questions:

(1) Are you using Ruby threads or processes for the parallelism?

(2) We should think about the sematics of the the command "rake -j2 a  
b"  Are "a" and "b" executed in parallel or sequentially.  It looks  
like the code base goes with sequentially, and I think this is the  
right choice.  But it may be worth a discussion.

(3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT".   
Generated from what?  Will I be able to regenerate them if they need  
changing?  Would it be better to just use CompTree as a gem?

(4) As far as I can tell, when running with num_threads > 1, you  
invoke all the tasks and gather the task arguments.  Then you pass the  
task dependency graph off to the CompTree code to execute the code in  
parallel.  So all the code execution actually happens after ALL the  
invokes are done on the code, rather than intermingled in standard  
rake.  Is my understanding correct.  (if so, very interesting ... I'm  
thinking that if it wasn't for the need for the task arguments, you  
could skip the invoke step and pass the dependency graph immediately  
to your CompTree package, yes?)

(5) I see there is a synchronization lock in the invoke method.  Since  
this part of the code is executed by a single task (the main task),  
I'm not sure I see the need for a lock.  Am I missing something?

(6) Have you tried running any of this under Ruby 1.9?

That's all for now.  Again, thanks for the work you put into this.   
I'll probably have more questions later.

-- Jim Weirich
-- jim.weirich at gmail.com

More information about the Rake-devel mailing list