"Federating" rubygems.org

Charles Oliver Nutter headius at headius.com
Wed Jan 18 18:09:57 EST 2012


On Wed, Jan 18, 2012 at 11:49 AM, Evan Phoenix <evan at phx.io> wrote:
> While thinking about the gem size and dependency naming issues last night, I had a thought for a feature.
>
> What if the following were possible:
>
>  gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem
>
> This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url.
>
> This allows rubygems.org to be a sort of head-in or primary federation server for gems. 99% of people will just push their gem like normal, but for the people that have very large gems that no longer fit within our size limit, they can simply host the gem file themselves. This provides a best case scenario where users can continue to simply install a gem like normal, but the costs of hosting the large gems is offset by the creator.

I wonder if Github or someone else with a tiered service would be
interested in partnering with RubyGems.org?

My concern about an arbitrary federation system is that there's a lot
of details involved. We're pushing the storage onto an arbitrary
service that may be unreliable, may be insecure, may allow the files
to change (more on security below). If we partnered directly with
someone like Github or folks that host Maven artifacts, we'd have a
directly integrated service that already manages sizes, tiering,
authentication, and so on.

Being able to push a gem with its guts on an arbitrary server is nice,
but I'd prefer to see something that just say "offload my gems to
Github" and figures out the rest for you.

> I was worried about security slightly, since the file at the url could be replaced with different code after the fact. But this is not dissimilar from someone pushing a new version that introduces a security issue. Since most people use ~> or > for their deps, they'd pick up that new version.

Security is definitely a tricky bit. Perhaps what actually gets pushed
to the store is not the raw gem but a gem plus a hash. Basically,
you've pushed the gem based on a particular file and it must remain
that file if you want the gem to keep working. Then the onus is on the
user to not replace files they've pushed before.

If course if you have a direct tie-in with something like GH, RG can
mangle the filename any way it wants, like
rake-1.2.3.gem.SOMELONGHASH. If the hash doesn't match the contents,
it won't be used, and if you're just using RG to do all the pushes
anyway you'll never care.

- Charlie


More information about the RubyGems-Developers mailing list