[Rubygems-developers] Gem autoloading problems

Donavan Pantke avatar at spellboundnet.com
Sat Apr 19 10:22:00 EDT 2008


On Friday 18 April 2008 04:56:44 pm Eric Hodel wrote:
> On Apr 18, 2008, at 06:07 AM, Donavan Pantke wrote:
> > On Friday 18 April 2008 04:01:01 am Eric Hodel wrote:
> >> On Apr 17, 2008, at 22:38 PM, Chad Woolley wrote:
> >>> On Thu, Apr 17, 2008 at 8:23 PM, Eric Hodel <drbrain at segment7.net>
> >>> wrote:
> >>>> A user of RubyGems can already control which gem gets activated  
> >>>> with
> >>>> Kernel#gem.  However, it is up to the user to activate the gems  
> >>>> they
> >>>> need in dependency order.  `gem lock` can help with this.
> >>>
> >>> Try GemInstaller.  It should be able to provide all the control you
> >>> need over gem activation, and autoinstall all required versions as
> >>> well.  If you can't make it do what you need, let me know.
> >>
> >> This problem is, given a-1, a-2, and b-1 which depends on any a:
> >>
> >> gem 'b'
> >>
> >> fork {
> >>   gem 'a', '1' # raises as a-2 is active
> >> }
> >
> > Correct. I also tested to ensure that the behavior was consistent,  
> > and it was
> > all the way back to 0.9.0, and it is consistent as you suggested.  
> > Sorry bout
> > that.
> >
> >> The solutions I see are:
> >>
> >> Don't list 'a' as a dependency.
> > Easy to do, but at that point the metadata doesn't describe the data  
> > well.
> >
> >> exec after fork to clear memory.
> > Well, since the entire point of preloading in this case is to avoid  
> > an exec
> > and the memory overhead associated with loading the libraries again,  
> > that
> > doesn't work well.
> 
> How well does this work?  Doesn't the garbage collector touch the  
> majority of your pages the first time it runs un-sharing most of the  
> memory saved?  I've never actually measured it, but common knowledge  
> seems to be that fork() to save memory via COW usually doesn't save  
> much with Ruby.

Not to get too far OT, but Hongli Lai has come up with a patch for MRI Ruby 
that moves the mark table outside the heap space itself, so GC becomes much 
more COW friendly. Part of the advantages with Passenger and the GC patch is 
that multiple versions of Rails can be loaded in child threads, then a 
GC.start is called right before a fork(), so the heaps have a much longer 
lifespan with COW.

> 
> >> Be careful not to require any files from 'a' and remove 'a' from the
> >> set of active gems.  (Hard)
> >
> > Actually, some of this harkens back to an earlier thread about  
> > categories of
> > dependencies and their handling. In this case, the desired behavior of
> > add_dependency in the specification was to influence the installer  
> > to ensure
> > that the right gems with the right versions were installed.  
> > Installed, but
> > not necessarily activated.
> 
> I'm not coming up with a good solution that fails in a sensible way.   
> Your usecase is unique in this respect, and I think RubyGems simply  
> fails it.  :)

Well, although this usecase is unique for now, with interpreters that handle 
memory better, and more frameworks like passenger come along, it may not that 
way for long.

That being said, what scenarios are you thinking of that would make sensible 
failing hard? My initial plan is to simply make it such that the activation 
code skips dependencies specified as install_only. DependencyInstaller will 
continue as normal. Am I missing something here?


> 
> > My suggestion right away is just to remove the
> > dependency, but this may make a small, easy to implement portion of  
> > the
> > system. The code should be much simpler because we're not really  
> > complicating
> > things much.
> 
> I think most people know that Passenger requires Rails, so this may be  
> the best solution. :)

Possibly, but if we can make the situation better and more seemless.... :)

> 
> 
> _______________________________________________
> Rubygems-developers mailing list
> Rubygems-developers at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rubygems-developers
> 




More information about the Rubygems-developers mailing list