[Rake-devel] Multiple Rakefile build

Patrick Bennett patrick.bennett at inin.com
Sat Dec 3 02:11:34 EST 2005

Jim Weirich wrote:

>Leon Yeh said:
>>Hi All, I have just started using Rake, I usually use ant, but got
>>caught the Ruby fever :-)
>>I have been using RoR for web development and in need rake to do data
>>migration. I wanted to build a hirarchical build system similar to
>>Makefile. [...]
>>What I want to do is that the top-level Rakefile go thru the folders and
>>  invoke each of Rakefile.
>I tend to avoid recursive make/rake invocations.  I prefer to build a
>unified dependency graph for the entire project.  You can read about this
>approach here:
I personally believe this is a horrible approach (at least for the size 
of builds I deal with).
The article you point out mainly speaks to the (very true) harm in 
recursively spawning new instances of make.  All of this is because 
'make' at least provides no other way to do this than to recursively 
spawn new instances of make.
At our company, our nightly builds now top 70 gigs, so building a 
unified dependency graph for everything at once before building a single 
line of code would be, well, kind of crazy (Ruby wouldn't handle it very 
well anyway).
Boost.Build works this way to an extent, and when you want to recompile 
a specific component, you don't want to wait for it to load the 
dependency information for every piece of code in the system before it 
looks at the 5 files in your test component.

What makes sense in my mind at least, is to mix the models (or at least 
support it ;>).
One instance of rake, keep a queue of rakefiles to run, running each in 
order, and keep certain key dependency information across rakefiles 
while clearing the tasks between rakefiles.
The C++ dependency generator data is maintained [the same headers tend 
to get referenced over and over again] across rakefiles.  The target 
dependency information is persisted into shared manifests so
Loading every source/object/target/link reference dependency for the 
entire system (Boost.Build works the same way) is only good for 
relatively 'small' systems.
With our system at least, almost every single executable has at least 4 
build variations (windows debug/release, and linux debug/release).  
That's at a bare minimum.  We have about 700+ components - some with 
hundreds of source files each.  Any given C++ source file might 
reference well over 1100 header files.  Each variation of the build 
types needs a different graph (different paths potentially, so different 
headers - different compilation options, different targets - so they 
have to be different tasks).  Loading it all into a single dependency 
graph would work fine - if you worked on a small system.

>Of course, you don't have to put the entire dependency graph in a single
>Rakefile.  You can distribute it across your subdirectories and just
>require the fragments as you need.  Then everything runs in a single rake
>process and rake handles all necessary builds.
This makes some sense, but if you see the only option is to either load 
everything all at once and then run, or to load one rakefile and run, 
with no alternative in between, then I disagree.

>One problem with really large projects is the possibility of name
>collisions amoung the different sub-directories.  
Yup, Boost.Build handled this by fully resolving all the defined 
tasks/files/etc into, effectively, unique namespaces based on the fully 
resolved paths.

>The CVS head of Rake now
>supports namespaces so each subdirectory can create tasks in its own
>namespace without conflict.  (The code is ready for a beta-release ... I
>just haven't had time to document the new features yet).
I definitely think it's a good idea that you intend to support building 
the dependency graph based on rakefiles from multiple directories.  It 
could definitely prove useful in some cases.
However, if you intend on this being the one, true way, then I think 
you're being a tad short-sighted.
The paper you sighted (which is 8 years old) talks about a project 
containing 3,000 files as a hypothetical large system [at least that's 
the impression I had].  That's so small it's kind of laughable.
A single branch of our codebase contains *at least* 100,000+ files. :\


*Patrick Bennett* | Software Engineer
phone & fax +1.317.715.8302 | patrick.bennett at inin.com
*Interactive Intelligence Inc.*
Deliberately Innovative
www.inin.com <http://www.inin.com/>

More information about the Rake-devel mailing list