[Rubygems-developers] New Beta Gem with Better Memory Footprint (was: Bulk updates)

Jim Weirich jim.weirich at gmail.com
Fri Jun 1 13:24:59 EDT 2007

On 6/1/07, Ryan Davis <ryand-ruby at zenspider.com> wrote:
> On Jun 1, 2007, at 08:24 , Jim Weirich wrote:
> > I've taken Ryan's changes (removing a copy step in the bulk download)
> > and added a few more for a beta release of RubyGems.
> >
> > Here are the changes:
> >
> > * Bulk downloads no longer copy the YAML string before loading it.
> > * Bulk downloads don't kick in until more than 500 gems need resyncing
> >    (the old limit was 50).
> > * Users may set the bulk download threshhold in the command line (-B)
> >    or in the .gemrc file.
> Is this going to thrash rubyforge.org with the number of connections?

I don't recall making the connections concurrent ... although I
thought about it.  If it is non-concurrent, then at any given time you
don't have any more connections, its just that the lifetime of the
connections are shorter.

> For the record, I got much better numbers than this. I'm not sure
> what the differentiating factor is. I had 120Mb -> 73Mb for the first
> two steps.

I think memory footprint measurements are still a black art.  FWIW, I
added a --bench command to gem that will pause it just before it exits
so you can get the final number from activity monitor.

> I'm not sure how/why the incremental would be that much better. I
> proposed sticking with bulk but letting go of the previous cache for
> that server so it can be GC'd. I think this would be a good balance
> between # of socket connections and memory consumption.

Although not in the beta, I did try this and didn't see any marked
difference.  Of course, I could have done something wrong, so please
double check it.

> Also Jim, I just committed a change to properly skip those files with
> the bad names. Please scoop that up as well.

Will do.  Thanks.

-- Jim Weirich    jim at weirichhouse.org     http://onestepback.org
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

More information about the Rubygems-developers mailing list