RubyForge Mirrors

In order to save bandwidth on the main RubyForge site, we have a mirror system for serving large files and RubyGems. Here's how it works.

File mirroring

When a person releases a file, it gets written to a directory and the database stores the path to that file. Every 10 minutes, a cron job runs that checks for new files and pings a "mirror master" server if any are found. The mirror master then rsyncs over any new files and pushes them out to all the folks hosting file mirrors. Later, when someone tries to download a file, some custom code we hacked into GForge redirects the request to a mirror server.

RubyGems mirroring

When a person releases a RubyGem, it gets written to the filesystem just like a regular file - since to GForge, that's all it is. Every 5 minutes, a file checks for new RubyGem releases. If it finds any it rebuilds the Gem index (both the complete and the incremental indexes), pings the mirror master, and from there it's just like file mirroring.

When someone tries to download a Gem (via gem install rails) the gems.rubyforge.org virtual host uses mod_rewrite to issue an HTTP redirect to a randomly selected mirror.

Setup

If you'd like to host a mirror, please join the support-mirrors mailing list. After that, you'll need to coordinate with Dennis Oelkers, who runs the mirror master, to allow him to push changes to your server. On your end, you'll need to:

• Allocate some disk space for the files. As of 1/20/2009, this is 15 GB for file mirrors and 4 GB for RubyGem mirrors. Also, being a mirror will chew up about 300 GB of bandwidth per month.
• Open up port 873 (rsync) on your filewall.
• Set up two rsync entries (e.g., in rsyncd.conf). These directory names are only examples:

  [gem-mirror]
     path = /var/www/rubyforge-gems/
     auth users = rubyforge
     secrets file = /var/www/etc/rsync.secrets
     read only = no
     hosts allow = master.mirror.rubyforge.org
     hosts deny = *
  

  [file-mirror]
     path = /var/www/rubyforge-files/
     auth users = rubyforge
     secrets file = /var/www/etc/rsync.secrets
     read only = no
     hosts allow = master.mirror.rubyforge.org
     hosts deny = *
  

• Set up two virtual hosts: filesmirror.yourhost.com (pointing to /var/www/rubyforge-files/) and gemmirror.yourhost.com (pointing to /var/www/rubyforge-gems/). The vhosts can be named slightly differently if you want.
• Send the contents of your /var/www/etc/rsync.secrets file to Dennis using some sort of encryption mechanism - like PGP. Note that the rsync.secrets file should not be world-readable, but should be readable by the user running the rsync process.
• Once Dennis gets your information he'll do an initial sync to your server and then we'll add it to the rotation. Then you'll receive props here and you'll bask in the knowledge that you're helping to keep RubyForge speedy.

Internal mirrors

If you just want to get the gems and the files (say, for an internal mirror), Dennis has set up an rsync endpoint which you can get to like this:

$ rsync -av rsync://master.mirror.rubyforge.org/gems/ .
receiving file list ... done
./
ActiveRecord-JDBC-0.0.1.gem
ActiveRecord-JDBC-0.2.0.gem
[ ... etc ... ]

Same sort of thing for the files:

$ rsync -av rsync://master.mirror.rubyforge.org/files/ .

TODO

• What happens when a file or a gem gets deleted?

Questions? Comments? Post to the support-mirrors mailing list, or just send an email to Tom Copeland.