[Rubygems-developers] Updating source index is slow
eivind at FreeBSD.ORG
Thu Nov 11 08:36:39 EST 2004
On Wed, Nov 10, 2004 at 02:39:46PM -0500, Tom Clarke wrote:
> I've been lurking here for a bit, and just let me say that ruby gems
> is looking great and everything I wanted for the now defunct
> RAAInstall (which I gave life to and subsequently killed).
> The issue I encounted with raa install was that for an RAA sized
> database (which gems isn't yet) it was very difficult to get the
> information down to a reasonable size. I ended up producing a
> drastically compressed format (in terms of what data I put in the
> download file). The gems I've seen are comparatively verbose.
> I would suggest that the end solution probably needs to be either of:
> * A gentoo like system where each package has it's own file, and be
> synced with rsync or a clone (this has the benefit of being easy to
> mirror in an efficient manner)
> * A client server system where package details are queried on demand.
It also works to have large metadata and have the users sync it occasionally
using the rsync algorithm in some fashion.
We do this for FreeBSD using cvsup, and it scales to the 12000 ports
we maintain there.
WRT packaging DBI: This will require binary packages if it is to work well,
I think. David Ross is coordinating a project to try to make this work for
Windows; he's probably primarily targetting RPA (as he is a fan of ours), but
it should probably be possible to use a lot of the results for RubyGems, too.
The problem turned out to be much more icky than at least I thought; I've
been used to doing binary packages and having that be only a minor part
of the problem in packaging. However, the issue of platforms without a
standardized build environment (Windows, especially) turns this nasty.
More information about the Rubygems-developers