[Rake-devel] Parallel tasks with Rake
jim.weirich at gmail.com
Thu Sep 11 00:44:46 EDT 2008
On Sep 4, 2008, at 2:33 AM, James M. Lawrence wrote:
Saw your announcement on ruby-talk and want to say good job on getting
drake out. Now that 0.8.2 is released, I've taken some time to look
at some of what you've done. It looks impressive.
BTW, I'm copying this email to the Rake Development list. I think it
would be great to get this conversation on the list for everyone to
participate. You can sign up at http://rubyforge.org/mailman/listinfo/rake-devel
I've pulled the changes into a branch of rake and have begun playing
with it. I've not delved into the deeps of CompTree yet, but the
changes to rake look pretty straight forward.
And of course, I have some questions:
(1) Are you using Ruby threads or processes for the parallelism?
(2) We should think about the sematics of the the command "rake -j2 a
b" Are "a" and "b" executed in parallel or sequentially. It looks
like the code base goes with sequentially, and I think this is the
right choice. But it may be worth a discussion.
(3) I see a lot of the files are marked "GENERATED -- DO NOT EDIT".
Generated from what? Will I be able to regenerate them if they need
changing? Would it be better to just use CompTree as a gem?
(4) As far as I can tell, when running with num_threads > 1, you
invoke all the tasks and gather the task arguments. Then you pass the
task dependency graph off to the CompTree code to execute the code in
parallel. So all the code execution actually happens after ALL the
invokes are done on the code, rather than intermingled in standard
rake. Is my understanding correct. (if so, very interesting ... I'm
thinking that if it wasn't for the need for the task arguments, you
could skip the invoke step and pass the dependency graph immediately
to your CompTree package, yes?)
(5) I see there is a synchronization lock in the invoke method. Since
this part of the code is executed by a single task (the main task),
I'm not sure I see the need for a lock. Am I missing something?
(6) Have you tried running any of this under Ruby 1.9?
That's all for now. Again, thanks for the work you put into this.
I'll probably have more questions later.
-- Jim Weirich
-- jim.weirich at gmail.com
More information about the Rake-devel