[Rubygems-developers] New Beta Gem with Better Memory Footprint (was: Bulk updates)
jim.weirich at gmail.com
Fri Jun 1 13:24:59 EDT 2007
On 6/1/07, Ryan Davis <ryand-ruby at zenspider.com> wrote:
> On Jun 1, 2007, at 08:24 , Jim Weirich wrote:
> > I've taken Ryan's changes (removing a copy step in the bulk download)
> > and added a few more for a beta release of RubyGems.
> > Here are the changes:
> > * Bulk downloads no longer copy the YAML string before loading it.
> > * Bulk downloads don't kick in until more than 500 gems need resyncing
> > (the old limit was 50).
> > * Users may set the bulk download threshhold in the command line (-B)
> > or in the .gemrc file.
> Is this going to thrash rubyforge.org with the number of connections?
I don't recall making the connections concurrent ... although I
thought about it. If it is non-concurrent, then at any given time you
don't have any more connections, its just that the lifetime of the
connections are shorter.
> For the record, I got much better numbers than this. I'm not sure
> what the differentiating factor is. I had 120Mb -> 73Mb for the first
> two steps.
I think memory footprint measurements are still a black art. FWIW, I
added a --bench command to gem that will pause it just before it exits
so you can get the final number from activity monitor.
> I'm not sure how/why the incremental would be that much better. I
> proposed sticking with bulk but letting go of the previous cache for
> that server so it can be GC'd. I think this would be a good balance
> between # of socket connections and memory consumption.
Although not in the beta, I did try this and didn't see any marked
difference. Of course, I could have done something wrong, so please
double check it.
> Also Jim, I just committed a change to properly skip those files with
> the bad names. Please scoop that up as well.
Will do. Thanks.
-- Jim Weirich jim at weirichhouse.org http://onestepback.org
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)
More information about the Rubygems-developers