[Rake-devel] Parallel tasks with Rake
ittay.dror at gmail.com
Thu Sep 11 00:55:19 EDT 2008
Jim Weirich wrote:
> On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote:
> Hi James,
> Saw your announcement on ruby-talk and want to say good job on getting
> drake out. Now that 0.8.2 is released, I've taken some time to look
> at some of what you've done. It looks impressive.
> BTW, I'm copying this email to the Rake Development list. I think it
> would be great to get this conversation on the list for everyone to
> participate. You can sign up at
> I've pulled the changes into a branch of rake and have begun playing
> with it. I've not delved into the deeps of CompTree yet, but the
> changes to rake look pretty straight forward.
> And of course, I have some questions:
> (1) Are you using Ruby threads or processes for the parallelism?
> (2) We should think about the sematics of the the command "rake -j2 a
> b" Are "a" and "b" executed in parallel or sequentially. It looks
> like the code base goes with sequentially, and I think this is the
> right choice. But it may be worth a discussion.
For me it is more intuitive to expect a and b to execute in parallel
(unless one depends on the other)
> (3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT".
> Generated from what? Will I be able to regenerate them if they need
> changing? Would it be better to just use CompTree as a gem?
> (4) As far as I can tell, when running with num_threads > 1, you
> invoke all the tasks and gather the task arguments. Then you pass the
> task dependency graph off to the CompTree code to execute the code in
> parallel. So all the code execution actually happens after ALL the
> invokes are done on the code, rather than intermingled in standard
> rake. Is my understanding correct. (if so, very interesting ... I'm
> thinking that if it wasn't for the need for the task arguments, you
> could skip the invoke step and pass the dependency graph immediately
> to your CompTree package, yes?)
> (5) I see there is a synchronization lock in the invoke method. Since
> this part of the code is executed by a single task (the main task),
> I'm not sure I see the need for a lock. Am I missing something?
Why the need to do this inside 'invoke'? Isn't it enough to use this
code on the top level tasks only? (invoking them along the way)
> (6) Have you tried running any of this under Ruby 1.9?
> That's all for now. Again, thanks for the work you put into this.
> I'll probably have more questions later.
Ittay Dror <ittay.dror at gmail.com>
More information about the Rake-devel