[Rubygems-developers] Current bugs and loadpath_manager.rb

Gavin Sinclair gsinclair at soyabean.com.au
Sat Aug 28 07:16:13 EDT 2004

On Saturday, August 28, 2004, 1:44:48 PM, Richard wrote:

> On 8/27/04 7:40 PM, "Chad Fowler" <chad at chadfowler.com> wrote:

>> preferably to _any_ of our library stubs.  If we were to discuss and
>> agree, perhaps we could just rip all of the stub stuff back out?  Rich,
>> would you like to introduce how this works?
>> Chad

> OK...here goes.  The stub stuff has been bothering me for a while.  The main
> issue is we are 'dirtying up' the site_ruby directory with all these stubs,
> which was one thing we wanted to avoid when doing gems to begin with.

I quite agree that the library stub stuff is in a sense undesirable, and am
happy to see it replaced with the scheme you've come up with, which seems
pretty clever.  (A little too clever for me right now, but I'll get
there :)

If the goal is allowing easy access to gems for non-Rubygems-aware
programs, I think library stubs meet that goal better.  But at a
price, obviously.

> Anyway, the problem was managing how to 'fault in' gems when they are
> needed.  There were various stabs at a 'require hack' that attempted to do
> this, but I realized for the require hack to work right, it would need to be
> fast, efficient, and as transparent as possible.  It would need to allow you
> to require ANY file in ANY gem (with the highest version first).  What we
> effectively needed was something that scanned all the paths in all the gems
> (sorted in highest-version-first gem order) for a file match, and then
> require_gem'd the gem the file was in, then required that file.

> This is what the load_path_manager.rb does.  It contains a require hack, and
> a module that quickly searches for a required file first in the LOAD_PATH,
> then in all the gems.  To achieve speed it builds Dir.glob pattern that
> contains all the directories to search (both for the LOAD_PATH and
> Gem.path).

> Here is an example of the jabber4r gem dir:

> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/jabber4r/jid.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/rexml_1.8_patch.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/session.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/jabber4r.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/protocol.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/roster.rb
> ruby/gems/1.8/gems/jabber4r-0.8.0/lib/vcard.rb

> In the .gemspec it has:

> ...
>   s.require_paths = ["lib"]

> So, once you require 'rubygems' and
> 'rubygems/load_path_manager.rb'...

> require 'jabber4r/jid'

> The LoadPathManager class searches the LOAD_PATH for a matching file, and
> fines none.  It then searches all the require_paths of all the gems for a
> matching file.  I actually have two versions of jabber4r installed, so it
> ends up finding jabber4r/jid in two places.  It picks the first, then from
> the list and from there determines which gem it is (jabber4r version 0.8.0)
> and does a:

> require_gem 'jabber4r', '= 0.8.0'

> Once that gem is 'required' the LOAD_PATH has been modified, and so the
> original require (aliased as require__) is called with 'jabber4r/jid' which
> it now has in the LOAD_PATH.  One consequence of this is autorequire is no
> longer needed because files are required after gem_require.

I don't understand this consequence.  Oh, hang on -- require
'jabber4r' goes and finds 'jabber4r.rb' (if it exists).  OK.

> Performance wise, this is really fast.  I built an optimized way of loading
> all the gemspec files (I created a 'mock' Gem::Specification to speed things
> up because I only need certain data).  From there, I build a Dir.glob
> pattern, which when called is executed in C...and is quite fast.

That's kind of surprising.  Dir.glob may be implemented well, but
accessing lots of files doesn't sound fast in any language.

> The only performance hit we get right now is loading the
> lib/specification.rb file because that file has a lot of cool ruby
> hackery in it (clever, but a tad slow).

I'm happy to put effort into speeding specification.rb up.  My only
concern with that file is that it's easy to verify and document.  I'm
sure performance can be improved while still meeting that goal.

> So again, this is a generally useful technique for allowing require to work
> with rubygems transparently, and if you want a particular version, you can
> still do a require_gem on whatever you like.  The problem, of course, is
> getting the rubygems require in there to begin with.  As with the original
> require hacks, we could have folks use the ENV variable RUBYOPT and specify:

> set RUBYOPT='rrubygems'

That's a really fragile and undesirable condition.  Then again,
library stubs are probably more fragile, as they can be: not
installed, partially installed, incorrectly installed, buggy, or
properly installed; all on a case by case basis.

At least RUBYOPT is all or nothing.

It's a big bridge to cross, though, in terms of probable user

> Then when you run ruby the require is already overridden.  Anyway, that is
> the way is stands now.  It is not currently enabled until I check in a
> change in the rubygems.rb file that requires the
> rubygems/load_path_manager.rb file.

Any other alternative you can think of to RUBYOPT?

> Let me know comments,

I'm happy to see this go ahead, and will be able to work on improving
performance of specification.rb, and removing the library stub stuff.
(Not now; after we've worked out what's going on.)

I won't bother fixing those errors I reported about either -- yet.  It
just doesn't seem important at the moment :)

Let's hear other comments!


More information about the Rubygems-developers mailing list