[Rubygems-developers] Optimisation suggestion (was: Updating source index is slow)

Gavin Sinclair gsinclair at soyabean.com.au
Wed Nov 10 19:55:34 EST 2004

On Thursday, November 11, 2004, 12:17:59 AM, Richard wrote:

> I will look into downloading just the updated gemspecs and see if that can
> be used to speed things up.  It would obviously be a cgi...wanted to stay
> away from that...but CPU is cheeper than bandwidth right now!

Here's a way you could do it.  Have an index file that looks like

  #  Gem Name              MD5
  1  rake-0.4.4            fadflhdflkashflaewjhalewkjhf
  2  rake-0.4.8            fkasjhflkjehflkaht4luhdlkfjl

(Or use YAML.)  That can obviously be downloaded quickly.

Then each gemspec is in its own file (rake-0.4.4.gemspec, etc.) and is
downloaded separately.

The client sees what's in its current index, and assumes that it has
all the corresponding gemspec files.  It examines the fresh index and
sees which numbers it's missing.  It sends a request to the server,
containing a list of the numbers.  The server responds with a
datastream of the corresponding gemspecs.  The client then writes all
the gemspec files.

As a further optimisation, the index file could include the project
name and summary, making the list/search command much faster.

The 'gem check' command can verify the soundness of the client-side
information, and if necessary throw it all out and get fresh
information from the server.

Does all this sound plausible?


More information about the Rubygems-developers mailing list