[Rubygems-developers] New Beta Gem with Better Memory Footprint (was: Bulk updates)

Ryan Davis ryand-ruby at zenspider.com
Fri Jun 1 11:36:27 EDT 2007

On Jun 1, 2007, at 08:24 , Jim Weirich wrote:

> I've taken Ryan's changes (removing a copy step in the bulk download)
> and added a few more for a beta release of RubyGems.
> Here are the changes:
> * Bulk downloads no longer copy the YAML string before loading it.
> * Bulk downloads don't kick in until more than 500 gems need resyncing
>    (the old limit was 50).
> * Users may set the bulk download threshhold in the command line (-B)
>    or in the .gemrc file.

Is this going to thrash rubyforge.org with the number of connections?

> Here are the memory stats using Ruby 1.8.5 on Mac OSX:
> Old Version Bulk:       107Mb Real,  146Mb Virtual
> New Version Bulk:        98Mb Real,  130Mb Virtual
> New Version Incremental  36Mb Real,   63Mb Virtual

For the record, I got much better numbers than this. I'm not sure  
what the differentiating factor is. I had 120Mb -> 73Mb for the first  
two steps.

I'm not sure how/why the incremental would be that much better. I  
proposed sticking with bulk but letting go of the previous cache for  
that server so it can be GC'd. I think this would be a good balance  
between # of socket connections and memory consumption.

Also Jim, I just committed a change to properly skip those files with  
the bad names. Please scoop that up as well.

More information about the Rubygems-developers mailing list