From thewoolleyman at gmail.com Wed Oct 11 15:54:40 2006 From: thewoolleyman at gmail.com (Chad Woolley) Date: Wed, 11 Oct 2006 12:54:40 -0700 Subject: [Support-mirrors] Mirroring without hosting? Message-ID: Hello, We'd like to have a local mirror of the gems for situations where rubyforge is down. However, we don't have the bandwidth available to be a public server. Is it still possible for us to rsync the files down, even if we don't host them? We might even consider being a public mirror if the bandwidth requirement wasn't quite as high (we may have 100-200 gig free if we know it won't go over that much). Thanks, Chad From thewoolleyman at gmail.com Wed Oct 11 15:59:25 2006 From: thewoolleyman at gmail.com (Chad Woolley) Date: Wed, 11 Oct 2006 12:59:25 -0700 Subject: [Support-mirrors] Questions on RubyForge outages Message-ID: Yesterday I noticed rubyforge was down, as well as gem downloads. I'm just curious of the details behind this, and why the gem mirrors didn't help this situation. Was the central server that is doing the round-robin down, or the whole rubyforge domain? Also, what happens if everything is up on rubyforge, but one of the mirrors is down. Is the mirror selection algorithm smart enough to remove a server from the round-robin if it is down, or will people just get a failure if they happen to get assigned to that server when it's down? Thanks, Chad From tom at infoether.com Wed Oct 11 16:20:59 2006 From: tom at infoether.com (Tom Copeland) Date: Wed, 11 Oct 2006 16:20:59 -0400 Subject: [Support-mirrors] Mirroring without hosting? In-Reply-To: Message-ID: <009701c6ed72$c498a2f0$0301000a@tomhplaptop> > We'd like to have a local mirror of the gems for situations > where rubyforge is down. However, we don't have the > bandwidth available to be a public server. Is it still > possible for us to rsync the files down, even if we don't > host them? Hi Chad - Hm, Dennis, what do you think? > We might even consider being a public mirror if > the bandwidth requirement wasn't quite as high (we may have > 100-200 gig free if we know it won't go over that much). We could set things up so that you're only a RubyGems mirror; the load there is far lighter than for a files mirror. And we can always throttle usage if it's too high... Yours, Tom From tom at infoether.com Wed Oct 11 16:23:34 2006 From: tom at infoether.com (Tom Copeland) Date: Wed, 11 Oct 2006 16:23:34 -0400 Subject: [Support-mirrors] Questions on RubyForge outages In-Reply-To: Message-ID: <009801c6ed73$20fe0c60$0301000a@tomhplaptop> > Yesterday I noticed rubyforge was down, as well as gem > downloads. I'm just curious of the details behind this, and > why the gem mirrors didn't help this situation. Hi Chad - It's because the gem index isn't mirrored out, so you can't do something like "gem install rails --source http://foo_bar". At least, I don't think the index is getting mirrored out. But maybe it should be. Hm, is that all that's necessary to run a gem server? I think so... must investigate. > Was the central server that is doing the round-robin down, or > the whole rubyforge domain? Just the one machine that hosts rubyforge.org. > Also, what happens if everything is up on rubyforge, but one > of the mirrors is down. Is the mirror selection algorithm > smart enough to remove a server from the round-robin if it is > down, or will people just get a failure if they happen to get > assigned to that server when it's down? The latter - people will just get an error if they're redirected to the server that's down. Dennis has written a script to fix this; I need to integrate it into the gem redirector thingy. Yours, Tom From thewoolleyman at gmail.com Wed Oct 11 18:33:25 2006 From: thewoolleyman at gmail.com (Chad Woolley) Date: Wed, 11 Oct 2006 15:33:25 -0700 Subject: [Support-mirrors] Mirroring without hosting? In-Reply-To: <009701c6ed72$c498a2f0$0301000a@tomhplaptop> References: <009701c6ed72$c498a2f0$0301000a@tomhplaptop> Message-ID: On 10/11/06, Tom Copeland wrote: > > > We might even consider being a public mirror if > > the bandwidth requirement wasn't quite as high (we may have > > 100-200 gig free if we know it won't go over that much). > > We could set things up so that you're only a RubyGems mirror; the load > there is far lighter than for a files mirror. And we can always > throttle usage if it's too high... That would work. How do you throttle usage? From tom at infoether.com Wed Oct 11 19:12:08 2006 From: tom at infoether.com (Tom Copeland) Date: Wed, 11 Oct 2006 19:12:08 -0400 Subject: [Support-mirrors] Mirroring without hosting? In-Reply-To: References: <009701c6ed72$c498a2f0$0301000a@tomhplaptop> Message-ID: <1160608328.17520.4.camel@bugs.hal> On Wed, 2006-10-11 at 15:33 -0700, Chad Woolley wrote: > > We could set things up so that you're only a RubyGems mirror; the load > > there is far lighter than for a files mirror. And we can always > > throttle usage if it's too high... > > That would work. How do you throttle usage? We use a RewriteMap that round robins between the gem mirrors, and hits more frequently listed mirrors more often. For example, if the map data was: a|a|a|b Then server "a" gets 3x as many requests as server "b". Yours, Tom