[Rake-devel] Parallel tasks with Rake

[Ittay Dror]

Jim Weirich wrote:
> On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote:
>
> Hi James,
>
> Saw your announcement on ruby-talk and want to say good job on getting 
> drake out.  Now that 0.8.2 is released, I've taken some time to look 
> at some of what you've done.  It looks impressive.
>
> BTW, I'm copying this email to the Rake Development list.  I think it 
> would be great to get this conversation on the list for everyone to 
> participate.  You can sign up at 
> http://rubyforge.org/mailman/listinfo/rake-devel
>
> I've pulled the changes into a branch of rake and have begun playing 
> with it.  I've not delved into the deeps of CompTree yet, but the 
> changes to rake look pretty straight forward.
>
> And of course, I have some questions:
>
> (1) Are you using Ruby threads or processes for the parallelism?
>
> (2) We should think about the sematics of the the command "rake -j2 a 
> b"  Are "a" and "b" executed in parallel or sequentially.  It looks 
> like the code base goes with sequentially, and I think this is the 
> right choice.  But it may be worth a discussion.
For me it is more intuitive to expect a and b to execute in parallel 
(unless one depends on the other)
>
> (3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT".  
> Generated from what?  Will I be able to regenerate them if they need 
> changing?  Would it be better to just use CompTree as a gem?
>
> (4) As far as I can tell, when running with num_threads > 1, you 
> invoke all the tasks and gather the task arguments.  Then you pass the 
> task dependency graph off to the CompTree code to execute the code in 
> parallel.  So all the code execution actually happens after ALL the 
> invokes are done on the code, rather than intermingled in standard 
> rake.  Is my understanding correct.  (if so, very interesting ... I'm 
> thinking that if it wasn't for the need for the task arguments, you 
> could skip the invoke step and pass the dependency graph immediately 
> to your CompTree package, yes?)
>
> (5) I see there is a synchronization lock in the invoke method.  Since 
> this part of the code is executed by a single task (the main task), 
> I'm not sure I see the need for a lock.  Am I missing something?
Why the need to do this inside 'invoke'? Isn't it enough to use this 
code on the top level tasks only? (invoking them along the way)

Ittay
>
> (6) Have you tried running any of this under Ruby 1.9?
>
> That's all for now.  Again, thanks for the work you put into this.  
> I'll probably have more questions later.
>

-- 
--
Ittay Dror <ittay.dror at gmail.com>