[Rubygems-developers] Size of gem index

Eric Hodel drbrain at segment7.net
Thu Aug 30 00:30:00 EDT 2007


On Aug 29, 2007, at 08:10, Nick Sieger wrote:

> The gem index yaml file is starting to get pretty big.  Folks are
> reporting Ruby's memory size gets in the range of 128-150MB when
> loading up the index.  Does anyone else think this is starting to get
> problematic?
>
> It's especially a problem for JRuby since we need to specify the
> memory size of the Java VM up front, so as the gem index grows,
> eventually JRuby gem install fails with OutOfMemoryErrors.  It seems
> wasteful to us to simply hike the max memory of the VM up to 512MB
> (currently the max is 256, but gem install is starting to err
> consistently for some even at that level).
>
> Would something like a two-tier index work, with the main yaml/yaml.Z
> just containing a list of all available gems/versions with pointers to
> individual gemspec files available on the web server,

You just described the "quick index" perfectly.

There's a problem with the index-building script on RubyForge that  
has been fixed in trunk but not deployed yet that causes bulk updates  
(yaml.Z/yaml) when they aren't necessary.

> or could gem even get by w/o the gemspec before a gem is actually  
> downloaded and
> installed?

The index is needed to resolve versions and platforms across  
dependencies.

> Or is the gem index not something to be mucked with at this stage?

I think the fix to the indexer script will solve the problem unless  
the source index is extremely out of date.  I get annoying bulk index  
updates when I'm doing things at just the right times of day.

Still, we're open to improvements.

Previously Ryan Davis proposed an index of just the latest gems,  
since 90% of the time that's all people want.  I don't think it was  
fully implemented.

--
Poor workers blame their tools. Good workers build better tools. The
best workers get their tools to do the work for them. -- Syndicate Wars




More information about the Rubygems-developers mailing list