[Rake-devel] Multiple Rakefile build
patrick.bennett at inin.com
Sat Dec 3 02:11:34 EST 2005
Jim Weirich wrote:
>Leon Yeh said:
>>Hi All, I have just started using Rake, I usually use ant, but got
>>caught the Ruby fever :-)
>>I have been using RoR for web development and in need rake to do data
>>migration. I wanted to build a hirarchical build system similar to
>>What I want to do is that the top-level Rakefile go thru the folders and
>> invoke each of Rakefile.
>I tend to avoid recursive make/rake invocations. I prefer to build a
>unified dependency graph for the entire project. You can read about this
I personally believe this is a horrible approach (at least for the size
of builds I deal with).
The article you point out mainly speaks to the (very true) harm in
recursively spawning new instances of make. All of this is because
'make' at least provides no other way to do this than to recursively
spawn new instances of make.
At our company, our nightly builds now top 70 gigs, so building a
unified dependency graph for everything at once before building a single
line of code would be, well, kind of crazy (Ruby wouldn't handle it very
Boost.Build works this way to an extent, and when you want to recompile
a specific component, you don't want to wait for it to load the
dependency information for every piece of code in the system before it
looks at the 5 files in your test component.
What makes sense in my mind at least, is to mix the models (or at least
support it ;>).
One instance of rake, keep a queue of rakefiles to run, running each in
order, and keep certain key dependency information across rakefiles
while clearing the tasks between rakefiles.
The C++ dependency generator data is maintained [the same headers tend
to get referenced over and over again] across rakefiles. The target
dependency information is persisted into shared manifests so
Loading every source/object/target/link reference dependency for the
entire system (Boost.Build works the same way) is only good for
relatively 'small' systems.
With our system at least, almost every single executable has at least 4
build variations (windows debug/release, and linux debug/release).
That's at a bare minimum. We have about 700+ components - some with
hundreds of source files each. Any given C++ source file might
reference well over 1100 header files. Each variation of the build
types needs a different graph (different paths potentially, so different
headers - different compilation options, different targets - so they
have to be different tasks). Loading it all into a single dependency
graph would work fine - if you worked on a small system.
>Of course, you don't have to put the entire dependency graph in a single
>Rakefile. You can distribute it across your subdirectories and just
>require the fragments as you need. Then everything runs in a single rake
>process and rake handles all necessary builds.
This makes some sense, but if you see the only option is to either load
everything all at once and then run, or to load one rakefile and run,
with no alternative in between, then I disagree.
>One problem with really large projects is the possibility of name
>collisions amoung the different sub-directories.
Yup, Boost.Build handled this by fully resolving all the defined
tasks/files/etc into, effectively, unique namespaces based on the fully
>The CVS head of Rake now
>supports namespaces so each subdirectory can create tasks in its own
>namespace without conflict. (The code is ready for a beta-release ... I
>just haven't had time to document the new features yet).
I definitely think it's a good idea that you intend to support building
the dependency graph based on rakefiles from multiple directories. It
could definitely prove useful in some cases.
However, if you intend on this being the one, true way, then I think
you're being a tad short-sighted.
The paper you sighted (which is 8 years old) talks about a project
containing 3,000 files as a hypothetical large system [at least that's
the impression I had]. That's so small it's kind of laughable.
A single branch of our codebase contains *at least* 100,000+ files. :\
*Patrick Bennett* | Software Engineer
phone & fax +1.317.715.8302 | patrick.bennett at inin.com
*Interactive Intelligence Inc.*
More information about the Rake-devel