[Rubygems-developers] faster rubygems.rb loading by way of deferred plugin loads, and other resource usage enhancements

James Tucker jftucker at gmail.com
Fri Jul 9 13:40:00 EDT 2010


On 9 Jul 2010, at 17:25, Luis Lavena wrote:

> On Wed, Jul 7, 2010 at 12:27 PM, James Tucker <jftucker at gmail.com> wrote:
>> Hey all,
>> 
>> I've started work on a branch to reduce the runtime resource usage by loading rubygems.rb. Some of you are already aware of this [1].
>> 
>> The main 'cheat' in the branch so far is to move plugin loads into the command manager, but I'm not entirely satisfied with this, and so I'm polling for interest in what features plugin authors actually need / want, and where those can be injected. Most specifically, I want to see if they can be done so more lightly.
>> 
>> Initially, probably the most commonly used plugin is the gemcutter plugin, which provides commands. At present it deliberately loads the CommandManager, which results in loading in about 37 files and the spec cache into memory whenever rubygems.rb is loaded. The difference in startup performance is significantly over an order of magnitude on common developer systems (with largish numbers of gems installed).
>> 
>> A solution for command plugins would be to have plugins simply register a list of files that should be loaded when commands are going to be invoked. A continuation of this idea leads to an API something along the lines of:
>> 
>> Gem.register_plugin(:install, 'some_plugin/gem/installer')
>> Gem.register_plugin(:command, 'some_plugin/gem/command')
>> 
> 
> Wouldn't be more appropriate Gem.register_command(:push,
> 'path/to/gemcutter/plugin') ?
> 
> After all, you're registering commands, since the plugin loader has
> already been started.

Special casing for some hooks is fine, yes. Obviously the hooks for plugin initializations don't override or replace hooks for specific things, like pre_install / post_install, etc. I'm also looking into adding new hooks for certain things so that other things can be registered more correctly (e.g. http://github.com/erikh/hanna/blob/master/lib/rubygems_plugin.rb)

>> Such that rubygems could contain simple internal hooks to load these on demand, rather than ahead of time, avoiding pre-activation of plugins and the current increased cost of loading rubygems.
>> 
> 
> I like this, a lot.
> 
>> My end goal with this is to aim to get rubygems.rb fast and light enough that it could be used for the core integration in 1.9, to avoid the remaining and otherwise unavoidable bugs in gem_prelude.rb.
>> 
>> Other work that will be included in this, is indexing the paths to rubygems_plugin.rb files, and reducing the resource requirements for activation, by having a lightweight version and index set that can be used by Gem.activate in place of the current heavyweight Gem.spec.
>> 
> 
> Wouldn't rubygems_plugin.rb do the Gem.register* command and deffer
> the actual plugin loading script for the command invitation?

Yes, it will defer calling activate and modifying the load path.

> What is the benefit on indexing those, since these were already loaded?

To avoid needing to load Gem.spec in order to execute Gem.find_files, which is insanely expensive (relatively speaking).



More information about the Rubygems-developers mailing list