From tom at infoether.com Tue Nov 1 08:28:59 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 01 Nov 2005 08:28:59 -0500 Subject: [Support-mirrors] Gem mirror Message-ID: <1130851739.11439.63.camel@hal> Hi all - Some more folks have subscribed to the list - folks who are considering hosting RubyGem mirrors. Thanks to Chad Fowler for coordinating all this! I've added a new rsync endpoint "gem-mirror" that points to /var/www/gems/gems on RubyForge; this directory is 193 MB. Hopefully we can put together a RewriteMap thingy in the gems.rubyforge.org virtual host that can redirect traffic between all the gem mirrors. For starters, Chad, what's the URL that you've copied all the gems to so I can try to redirect some traffic there? Thanks, Tom From tom at infoether.com Tue Nov 1 08:32:53 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 01 Nov 2005 08:32:53 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130851739.11439.63.camel@hal> References: <1130851739.11439.63.camel@hal> Message-ID: <1130851973.11439.69.camel@hal> On Tue, 2005-11-01 at 08:28 -0500, Tom Copeland wrote: > Hi all - > > Some more folks have subscribed to the list - folks who are considering > hosting RubyGem mirrors. Thanks to Chad Fowler for coordinating all > this! > > I've added a new rsync endpoint "gem-mirror" that points > to /var/www/gems/gems on RubyForge; this directory is 193 MB. Hopefully > we can put together a RewriteMap thingy in the gems.rubyforge.org > virtual host that can redirect traffic between all the gem mirrors. Chad also suggested adding another rsync point (or modifying this one) to include the Gem index. That sounds like a good idea to me; let me get this rewrite thing working and then we'll move on to that. Thanks, Tom From chad at chadfowler.com Tue Nov 1 09:01:57 2005 From: chad at chadfowler.com (Chad Fowler) Date: Tue, 1 Nov 2005 09:01:57 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130851739.11439.63.camel@hal> References: <1130851739.11439.63.camel@hal> Message-ID: On 01-Nov-05, at 8:28 AM, Tom Copeland wrote: > Hi all - > > Some more folks have subscribed to the list - folks who are > considering > hosting RubyGem mirrors. Thanks to Chad Fowler for coordinating all > this! > > I've added a new rsync endpoint "gem-mirror" that points > to /var/www/gems/gems on RubyForge; this directory is 193 MB. > Hopefully > we can put together a RewriteMap thingy in the gems.rubyforge.org > virtual host that can redirect traffic between all the gem mirrors. > > For starters, Chad, what's the URL that you've copied all the gems > to so > I can try to redirect some traffic there? > Hi Tom. Here's the URL: http://rubygems.org/gems/ Thanks! Chad From robby.lists at planetargon.com Tue Nov 1 10:04:52 2005 From: robby.lists at planetargon.com (Robby Russell) Date: Tue, 01 Nov 2005 07:04:52 -0800 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130851739.11439.63.camel@hal> References: <1130851739.11439.63.camel@hal> Message-ID: <1130857492.30609.605.camel@linus> On Tue, 2005-11-01 at 08:28 -0500, Tom Copeland wrote: > Hi all - > > Some more folks have subscribed to the list - folks who are considering > hosting RubyGem mirrors. Thanks to Chad Fowler for coordinating all > this! > > I've added a new rsync endpoint "gem-mirror" that points > to /var/www/gems/gems on RubyForge; this directory is 193 MB. Hopefully > we can put together a RewriteMap thingy in the gems.rubyforge.org > virtual host that can redirect traffic between all the gem mirrors. > > For starters, Chad, what's the URL that you've copied all the gems to so > I can try to redirect some traffic there? > > Thanks, > > Tom PLANET ARGON has the follwing setup: http://rubyforge.planetargon.com/gems.rubyforge.org/ -- /****************************************************** * Robby Russell, Owner.Developer.Geek * PLANET ARGON, Open Source Solutions & Web Hosting * Portland, Oregon | p: 503.351.4730 | f: 815.642.4068 * www.planetargon.com | www.robbyonrails.com * Programming Rails | www.programmingrails.com *******************************************************/ From dennis at lauschmusik.de Tue Nov 1 10:15:55 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Tue, 1 Nov 2005 16:15:55 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130857492.30609.605.camel@linus> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey folks, Am 01.11.2005 um 16:04 schrieb Robby Russell: > PLANET ARGON has the follwing setup: > > http://rubyforge.planetargon.com/gems.rubyforge.org/ Our mirror of the repository will be accessible via: http://rubyforge.lauschmusik.de/gems.rubyforge.org/ I am in the middle of mirroring the files, should be finished soon. The cron-job for rsync will be run every 30 mins, 20/50 minutes after the hour. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDZ4awcT9mJF158yMRAvE5AJ9xLveJCNq/8vdG0MT8P0wwdbJ6QACg0au1 4f0vejT+fpWMfZEH052aLXE= =5i+C -----END PGP SIGNATURE----- From chad at chadfowler.com Tue Nov 1 10:24:55 2005 From: chad at chadfowler.com (Chad Fowler) Date: Tue, 1 Nov 2005 10:24:55 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: On 01-Nov-05, at 10:15 AM, Dennis Oelkers wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hey folks, > > Am 01.11.2005 um 16:04 schrieb Robby Russell: > >> PLANET ARGON has the follwing setup: >> >> http://rubyforge.planetargon.com/gems.rubyforge.org/ > > > Our mirror of the repository will be accessible via: > http://rubyforge.lauschmusik.de/gems.rubyforge.org/ > > I am in the middle of mirroring the files, should be finished > soon. The cron-job for rsync will be run every 30 mins, 20/50 > minutes after the hour. > Oh cool, that reminds me. I'm running every 15 minutes at 1,15,30, 45 (can you do 0 with cron? I'm never sure.) :) We should probably establish a minimum interval. The problem with even 15 minutes is that we might end up with 404s on attempted gem installs. Any ideas on how to deal with that? Robby, are you using rsync or just straight HTTP get? I see that you have the yaml indexes too. Chad From dpiddy at gmail.com Tue Nov 1 10:28:27 2005 From: dpiddy at gmail.com (Dan Peterson) Date: Tue, 1 Nov 2005 08:28:27 -0700 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: <56d3286d0511010728k5f8ccb7fta2ebc9293a727fbc@mail.gmail.com> I think he had mirrored my setup so that's why he has the indexes. I will be setting up http://gems.dpiddy.net/gems.rubyforge.org to be rsync'd later today. -d. On 11/1/05, Chad Fowler wrote: > Robby, are you using rsync or just straight HTTP get? I see that you > have the yaml indexes too. From dennis at lauschmusik.de Tue Nov 1 10:33:29 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Tue, 1 Nov 2005 16:33:29 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 01.11.2005 um 16:24 schrieb Chad Fowler: > Oh cool, that reminds me. I'm running every 15 minutes at 1,15,30, > 45 (can you do 0 with cron? I'm never sure.) :) You can :) > > We should probably establish a minimum interval. The problem with > even 15 minutes is that we might end up with 404s on attempted gem > installs. Any ideas on how to deal with that? > I would be fine with decreasing the interval. On the other hand we should think about establishing some sort of push- instead of pull-mechanism to update the mirrors. I do not have a clue how new/updated gems find their way to gems.rubyforge.org (ftp?), but maybe we should automate the process of updating the mirrors after a new/updated gem has been uploaded. (Maybe via rsync too?) It would not solve the problem of 404s perfectly, but it is as close as we can get with a decentralised setup. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDZ4rLcT9mJF158yMRAqmUAJoCVSPIbZLveedxf8bEBUmXPtEilgCg3weN pvGY1Pm3voH940c4kU25cpo= =pVnN -----END PGP SIGNATURE----- From robby.lists at planetargon.com Tue Nov 1 10:34:09 2005 From: robby.lists at planetargon.com (Robby Russell) Date: Tue, 01 Nov 2005 07:34:09 -0800 Subject: [Support-mirrors] Gem mirror In-Reply-To: <56d3286d0511010728k5f8ccb7fta2ebc9293a727fbc@mail.gmail.com> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <56d3286d0511010728k5f8ccb7fta2ebc9293a727fbc@mail.gmail.com> Message-ID: <1130859250.30609.612.camel@linus> On Tue, 2005-11-01 at 08:28 -0700, Dan Peterson wrote: > I think he had mirrored my setup so that's why he has the indexes. I > will be setting up http://gems.dpiddy.net/gems.rubyforge.org to be > rsync'd later today. > > -d. > Yep, that was exactly what I did. :-) (thanks again dan, it's been nice having my own gem mirror) -Robby -- /****************************************************** * Robby Russell, Owner.Developer.Geek * PLANET ARGON, Open Source Solutions & Web Hosting * Portland, Oregon | p: 503.351.4730 | f: 815.642.4068 * www.planetargon.com | www.robbyonrails.com * Programming Rails | www.programmingrails.com *******************************************************/ From chad at chadfowler.com Tue Nov 1 10:42:12 2005 From: chad at chadfowler.com (Chad Fowler) Date: Tue, 1 Nov 2005 10:42:12 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130859250.30609.612.camel@linus> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <56d3286d0511010728k5f8ccb7fta2ebc9293a727fbc@mail.gmail.com> <1130859250.30609.612.camel@linus> Message-ID: <54149C11-73FF-46CE-B63F-AD0945A8A12A@chadfowler.com> On 01-Nov-05, at 10:34 AM, Robby Russell wrote: > On Tue, 2005-11-01 at 08:28 -0700, Dan Peterson wrote: >> I think he had mirrored my setup so that's why he has the indexes. I >> will be setting up http://gems.dpiddy.net/gems.rubyforge.org to be >> rsync'd later today. >> >> -d. >> > > > Yep, that was exactly what I did. :-) > > (thanks again dan, it's been nice having my own gem mirror) > > Awesome. It's really great of you guys to help out with this! Chad From tom at infoether.com Wed Nov 2 08:51:22 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 08:51:22 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> Message-ID: <1130939482.25367.58.camel@hal> On Tue, 2005-11-01 at 09:01 -0500, Chad Fowler wrote: > > For starters, Chad, what's the URL that you've copied all the gems > > to so > > I can try to redirect some traffic there? > > > > Hi Tom. Here's the URL: http://rubygems.org/gems/ Super! OK, all the gems are now being served from rubygems.org - the index is still being served from RubyForge. Here's the RewriteRule I'm using: RewriteMap gemmirrors rnd:/usr/local/apache/conf/gemmirrors.map RewriteRule ^/gems/(.*) http://${gemmirrors:server}/gems/$1 [R] Comment/improvements are most welcome. Now to get some of the other folks into the rotation, figure out the index thing, etc, etc... Yours, tom From tom at infoether.com Wed Nov 2 08:52:40 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 08:52:40 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130857492.30609.605.camel@linus> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: <1130939560.25367.60.camel@hal> On Tue, 2005-11-01 at 07:04 -0800, Robby Russell wrote: > On Tue, 2005-11-01 at 08:28 -0500, Tom Copeland wrote: > > Hi all - > > > > Some more folks have subscribed to the list - folks who are considering > > hosting RubyGem mirrors. Thanks to Chad Fowler for coordinating all > > this! > > > > I've added a new rsync endpoint "gem-mirror" that points > > to /var/www/gems/gems on RubyForge; this directory is 193 MB. Hopefully > > we can put together a RewriteMap thingy in the gems.rubyforge.org > > virtual host that can redirect traffic between all the gem mirrors. > > > > For starters, Chad, what's the URL that you've copied all the gems to so > > I can try to redirect some traffic there? > > > > Thanks, > > > > Tom > > PLANET ARGON has the follwing setup: > > http://rubyforge.planetargon.com/gems.rubyforge.org/ Cool, OK, hm, let me see... hm. Could you move them in a "/gems/" directory, perhaps? That'll make my redirecting easier on this end... my Apache-fu is weak... Thanks, tom From tom at infoether.com Wed Nov 2 08:54:32 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 08:54:32 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: <1130939672.25367.63.camel@hal> On Tue, 2005-11-01 at 16:15 +0100, Dennis Oelkers wrote: > Our mirror of the repository will be accessible via: > http://rubyforge.lauschmusik.de/gems.rubyforge.org/ Thanks Dennis! Think I could get you to put them in a gems/ as well? Although maybe I can figure out a way to to it from here... seems like it should be possible... I'm just not sure if I can put just a hostname or what in that map file... > I am in the middle of mirroring the files, should be finished > soon. The cron-job for rsync will be run every 30 mins, 20/50 > minutes after the hour. Super - yup, once every half hour should be sufficient, I think... Tom From tom at infoether.com Wed Nov 2 08:55:52 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 08:55:52 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: <1130939752.25367.66.camel@hal> On Tue, 2005-11-01 at 10:24 -0500, Chad Fowler wrote: > Oh cool, that reminds me. I'm running every 15 minutes at 1,15,30, > 45 (can you do 0 with cron? I'm never sure.) :) > > We should probably establish a minimum interval. The problem with > even 15 minutes is that we might end up with 404s on attempted gem > installs. Any ideas on how to deal with that? I think that's OK... I think it's just a matter of setting expectations with an email to the list saying "hey, after you deploy a gem, there may be a 30 minute delay". At least, I'm not sure of a better way, unless we switch to server push. Yours, Tom From dennis at lauschmusik.de Wed Nov 2 08:57:14 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 02 Nov 2005 14:57:14 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130939672.25367.63.camel@hal> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> Message-ID: <4368C5BA.7060100@lauschmusik.de> Tom Copeland wrote: >On Tue, 2005-11-01 at 16:15 +0100, Dennis Oelkers wrote: > > >>Our mirror of the repository will be accessible via: >>http://rubyforge.lauschmusik.de/gems.rubyforge.org/ >> >> > >Thanks Dennis! Think I could get you to put them in a gems/ as well? > > Done. Kind regards, Dennis Oelkers From tom at infoether.com Wed Nov 2 09:02:09 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:02:09 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> Message-ID: <1130940129.25367.72.camel@hal> On Tue, 2005-11-01 at 16:33 +0100, Dennis Oelkers wrote: > I would be fine with decreasing the interval. Hm, I just checked RubyForge and there were about 30 rsync processes, so I think some cron jobs were starting to catch up with each other. We may want to rsync no more than once or twice an hour... > On the other hand we > should think about establishing > some sort of push- instead of pull-mechanism to update the mirrors. I > do not have a clue how new/updated > gems find their way to gems.rubyforge.org (ftp?) When a user releases a file that ends with ".gem" it gets copied to /var/www/gems/gems by a cronjob and the index gets rebuilt. > , but maybe we should > automate the process of updating > the mirrors after a new/updated gem has been uploaded. (Maybe via > rsync too?) > It would not solve the problem of 404s perfectly, but it is as close > as we can get with a decentralised setup. Hm. Server push is sounding better... I haven't set up that before, but seems like it should be somewhat straightforward... Yours, Tom From tom at infoether.com Wed Nov 2 09:03:43 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:03:43 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <54149C11-73FF-46CE-B63F-AD0945A8A12A@chadfowler.com> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <56d3286d0511010728k5f8ccb7fta2ebc9293a727fbc@mail.gmail.com> <1130859250.30609.612.camel@linus> <54149C11-73FF-46CE-B63F-AD0945A8A12A@chadfowler.com> Message-ID: <1130940223.25367.75.camel@hal> On Tue, 2005-11-01 at 10:42 -0500, Chad Fowler wrote: > Awesome. It's really great of you guys to help out with this! Right on, yup, thanks everybody! Soon RubyForge traffic will consist entirely of HTTP 302 :-) Yours, Tom From tom at infoether.com Wed Nov 2 09:06:52 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:06:52 -0500 Subject: [Support-mirrors] rsync pull options Message-ID: <1130940412.25367.80.camel@hal> Hi all - Since there's a lot of rsync stuff going on now, we might want to settle on the "best" options to invoke rysnc with to minimize load on RubyForge computation and bandwidth-wise. For example, is anyone using the "--whole-file" option? From the rsync docs, that option lets rsync "copy files whole (without rsync algorithm)". Sounds like that might be more efficient for what we're doing... anyone familiar with the effect of that? Thanks, Tom From tom at infoether.com Wed Nov 2 09:12:11 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:12:11 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <4368C5BA.7060100@lauschmusik.de> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> <4368C5BA.7060100@lauschmusik.de> Message-ID: <1130940731.25367.82.camel@hal> On Wed, 2005-11-02 at 14:57 +0100, Dennis Oelkers wrote: > Tom Copeland wrote: > > >On Tue, 2005-11-01 at 16:15 +0100, Dennis Oelkers wrote: > > > > > >>Our mirror of the repository will be accessible via: > >>http://rubyforge.lauschmusik.de/gems.rubyforge.org/ > >> > >> > > > >Thanks Dennis! Think I could get you to put them in a gems/ as well? > > > > > Done. > > Kind regards, > Dennis Oelkers Great, thanks Dennis! OK, just to make sure my random mirror thingy is working, I just tried 5-6 installs of a small Gem... Dennis, can you see if your logs have any hits to your gems directory? Thanks, Tom From dennis at lauschmusik.de Wed Nov 2 09:14:33 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 02 Nov 2005 15:14:33 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130940731.25367.82.camel@hal> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> <4368C5BA.7060100@lauschmusik.de> <1130940731.25367.82.camel@hal> Message-ID: <4368C9C9.6010000@lauschmusik.de> Tom Copeland wrote: >OK, just to make sure my random mirror thingy is working, I just tried >5-6 installs of a small Gem... Dennis, can you see if your logs have any >hits to your gems directory? > > $ grep -ic \/gems\/ rubyforge.lauschmusik.de-access_log 32 Seems like it's working! :) Kind regards, Dennis Oelkers From tom at infoether.com Wed Nov 2 09:17:47 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:17:47 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <4368C9C9.6010000@lauschmusik.de> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> <4368C5BA.7060100@lauschmusik.de> <1130940731.25367.82.camel@hal> <4368C9C9.6010000@lauschmusik.de> Message-ID: <1130941067.25367.85.camel@hal> On Wed, 2005-11-02 at 15:14 +0100, Dennis Oelkers wrote: > Tom Copeland wrote: > > >OK, just to make sure my random mirror thingy is working, I just tried > >5-6 installs of a small Gem... Dennis, can you see if your logs have any > >hits to your gems directory? > > > > > $ grep -ic \/gems\/ rubyforge.lauschmusik.de-access_log > 32 > > Seems like it's working! :) Excellent, I'm sure Chad's network connection appreciates you sharing the load :-) Thanks for the confirmation, Tom From chad at chadfowler.com Wed Nov 2 09:29:45 2005 From: chad at chadfowler.com (Chad Fowler) Date: Wed, 2 Nov 2005 09:29:45 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130941067.25367.85.camel@hal> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> <4368C5BA.7060100@lauschmusik.de> <1130940731.25367.82.camel@hal> <4368C9C9.6010000@lauschmusik.de> <1130941067.25367.85.camel@hal> Message-ID: <05DD5C78-DA44-4500-81D0-85C0A8847B1F@chadfowler.com> On 02-Nov-05, at 9:17 AM, Tom Copeland wrote: > On Wed, 2005-11-02 at 15:14 +0100, Dennis Oelkers wrote: >> Tom Copeland wrote: >> >>> OK, just to make sure my random mirror thingy is working, I just >>> tried >>> 5-6 installs of a small Gem... Dennis, can you see if your logs >>> have any >>> hits to your gems directory? >>> >>> >> $ grep -ic \/gems\/ rubyforge.lauschmusik.de-access_log >> 32 >> >> Seems like it's working! :) > > Excellent, I'm sure Chad's network connection appreciates you sharing > the load :-) > Yes, very much! Tom, can you get to the server push thing fairly quickly? If so, I'll wait to configure my rsync settings. I don't much like the idea of an hour lag between a gem deploy and it being installable without an error. Especially when the gem_index script is already a lag. Thanks! Chad From chad at chadfowler.com Wed Nov 2 09:36:05 2005 From: chad at chadfowler.com (Chad Fowler) Date: Wed, 2 Nov 2005 09:36:05 -0500 Subject: [Support-mirrors] rsync pull options In-Reply-To: <1130940412.25367.80.camel@hal> References: <1130940412.25367.80.camel@hal> Message-ID: <240FE031-3619-491E-8BBB-EE8227CF1628@chadfowler.com> On 02-Nov-05, at 9:06 AM, Tom Copeland wrote: > Hi all - > > Since there's a lot of rsync stuff going on now, we might want to > settle > on the "best" options to invoke rysnc with to minimize load on > RubyForge > computation and bandwidth-wise. > > For example, is anyone using the "--whole-file" option? From the > rsync > docs, that option lets rsync "copy files whole (without rsync > algorithm)". Sounds like that might be more efficient for what we're > doing... anyone familiar with the effect of that? > I'm completely rsync-ignorant, so I just copied and pasted the command you posted in an email last week :) Chad From tom at infoether.com Wed Nov 2 09:37:09 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:37:09 -0500 Subject: [Support-mirrors] rsync pull options In-Reply-To: <9e7db9110511020632o4d06ce77sa1d30758af6b26d2@mail.gmail.com> References: <1130940412.25367.80.camel@hal> <9e7db9110511020632o4d06ce77sa1d30758af6b26d2@mail.gmail.com> Message-ID: <1130942229.25367.90.camel@hal> On Wed, 2005-11-02 at 09:32 -0500, Austin Ziegler wrote: > Might it be better to do a round-robin approach, if possible, since > RubyForge is behind such a small pipe? > > That is, the mirror (or two mirrors) with the biggest pipes rsync > directly from RubyForge every five minutes or so. Everyone else rsyncs > from them. (And if I ever figure out how to run rsync on my mirror, > I'll happily add my mirror to the list for RubyGems, too.) That's a super idea! That would certainly help out with RubyForge's bandwidth. Anyone want to volunteer to be the RubyForge mirror-server-mirror? Yours, Tom From dennis at lauschmusik.de Wed Nov 2 09:39:00 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 2 Nov 2005 15:39:00 +0100 Subject: [Support-mirrors] rsync pull options In-Reply-To: <1130942229.25367.90.camel@hal> References: <1130940412.25367.80.camel@hal> <9e7db9110511020632o4d06ce77sa1d30758af6b26d2@mail.gmail.com> <1130942229.25367.90.camel@hal> Message-ID: <5BC44E08-B2CB-43F2-AA77-97CA76A7BB15@lauschmusik.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 02.11.2005 um 15:37 schrieb Tom Copeland: > On Wed, 2005-11-02 at 09:32 -0500, Austin Ziegler wrote: >> Might it be better to do a round-robin approach, if possible, since >> RubyForge is behind such a small pipe? >> >> That is, the mirror (or two mirrors) with the biggest pipes rsync >> directly from RubyForge every five minutes or so. Everyone else >> rsyncs >> from them. (And if I ever figure out how to run rsync on my mirror, >> I'll happily add my mirror to the list for RubyGems, too.) > > That's a super idea! That would certainly help out with RubyForge's > bandwidth. > > Anyone want to volunteer to be the RubyForge mirror-server-mirror? > > Yours, > > Tom Should be no problem for me to do that. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDaM+IcT9mJF158yMRAss6AJ9U7zhO+6QZsCSSkJ4Ts8PG00HUbwCfVlWQ uMP/QFfcggAgKx7lCNhW5uw= =6QZP -----END PGP SIGNATURE----- From chad at chadfowler.com Wed Nov 2 09:41:23 2005 From: chad at chadfowler.com (Chad Fowler) Date: Wed, 2 Nov 2005 09:41:23 -0500 Subject: [Support-mirrors] rsync pull options In-Reply-To: <1130942229.25367.90.camel@hal> References: <1130940412.25367.80.camel@hal> <9e7db9110511020632o4d06ce77sa1d30758af6b26d2@mail.gmail.com> <1130942229.25367.90.camel@hal> Message-ID: On 02-Nov-05, at 9:37 AM, Tom Copeland wrote: > On Wed, 2005-11-02 at 09:32 -0500, Austin Ziegler wrote: >> Might it be better to do a round-robin approach, if possible, since >> RubyForge is behind such a small pipe? >> >> That is, the mirror (or two mirrors) with the biggest pipes rsync >> directly from RubyForge every five minutes or so. Everyone else >> rsyncs >> from them. (And if I ever figure out how to run rsync on my mirror, >> I'll happily add my mirror to the list for RubyGems, too.) > > That's a super idea! That would certainly help out with RubyForge's > bandwidth. > > Anyone want to volunteer to be the RubyForge mirror-server-mirror? Probably not a good idea for my server unfortunately. But I think the general idea is great. Chad From tom at infoether.com Wed Nov 2 09:50:01 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 09:50:01 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <05DD5C78-DA44-4500-81D0-85C0A8847B1F@chadfowler.com> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939672.25367.63.camel@hal> <4368C5BA.7060100@lauschmusik.de> <1130940731.25367.82.camel@hal> <4368C9C9.6010000@lauschmusik.de> <1130941067.25367.85.camel@hal> <05DD5C78-DA44-4500-81D0-85C0A8847B1F@chadfowler.com> Message-ID: <1130943001.25367.114.camel@hal> On Wed, 2005-11-02 at 09:29 -0500, Chad Fowler wrote: > Tom, can you get to the server push thing fairly quickly? If so, > I'll wait to configure my rsync settings. I don't much like the idea > of an hour lag between a gem > deploy and it being installable without an error. Especially when > the gem_index script is already a lag. Cool, yup, you're right, it's not such a good thing. OK, I'll see what I can do... Yours, tom From halostatue at gmail.com Wed Nov 2 09:54:05 2005 From: halostatue at gmail.com (Austin Ziegler) Date: Wed, 2 Nov 2005 09:54:05 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) Message-ID: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> Tom and fellow mirror-providers, In following the discussion on mirroring RubyGems files, I made the suggestion that we perhaps have a large pipe mirror get the initial seed of a file and then the rest of us get it from the large pipe mirror. This would save Tom, for example, from having to push out six copies of Instant Rails when Curt Hibbs updates it at *35 Mb. Unfortunately, while I have a good bandwidth allocation (100Gb per month), I don't have the ability to rsync or otherwise pull. I have locked down the RubyForge mirror directory, so I have no problem with the mirror-mirror having the same authentication information that Tom currently has for pushing. That is, Tom pushes to the mirror-mirror (which seems to be likely to be Dennis at this point) and then we give our mirror-auth information to Dennis and he can push to the rest of us. How does everyone else feel about that? -austin -- Austin Ziegler * halostatue at gmail.com * Alternate: austin at halostatue.ca From chad at chadfowler.com Wed Nov 2 10:00:14 2005 From: chad at chadfowler.com (Chad Fowler) Date: Wed, 2 Nov 2005 10:00:14 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> Message-ID: <31ED19E2-2465-4CEB-998A-0A349FCF8A73@chadfowler.com> On 02-Nov-05, at 9:54 AM, Austin Ziegler wrote: > Tom and fellow mirror-providers, > > In following the discussion on mirroring RubyGems files, I made the > suggestion that we perhaps have a large pipe mirror get the initial > seed of a file and then the rest of us get it from the large pipe > mirror. This would save Tom, for example, from having to push out six > copies of Instant Rails when Curt Hibbs updates it at *35 Mb. > > Unfortunately, while I have a good bandwidth allocation (100Gb per > month), I don't have the ability to rsync or otherwise pull. I have > locked down the RubyForge mirror directory, so I have no problem with > the mirror-mirror having the same authentication information that Tom > currently has for pushing. That is, Tom pushes to the mirror-mirror > (which seems to be likely to be Dennis at this point) and then we give > our mirror-auth information to Dennis and he can push to the rest of > us. > > How does everyone else feel about that? > That would work for me. Chad From dennis at lauschmusik.de Wed Nov 2 10:01:34 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 2 Nov 2005 16:01:34 +0100 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 02.11.2005 um 15:54 schrieb Austin Ziegler: > Tom and fellow mirror-providers, > > In following the discussion on mirroring RubyGems files, I made the > suggestion that we perhaps have a large pipe mirror get the initial > seed of a file and then the rest of us get it from the large pipe > mirror. This would save Tom, for example, from having to push out six > copies of Instant Rails when Curt Hibbs updates it at *35 Mb. > > Unfortunately, while I have a good bandwidth allocation (100Gb per > month), I don't have the ability to rsync or otherwise pull. I have > locked down the RubyForge mirror directory, so I have no problem with > the mirror-mirror having the same authentication information that Tom > currently has for pushing. That is, Tom pushes to the mirror-mirror > (which seems to be likely to be Dennis at this point) and then we give > our mirror-auth information to Dennis and he can push to the rest of > us. > > How does everyone else feel about that? Good idea imo. I'm fine with it! Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDaNTRcT9mJF158yMRAgzGAKCRepS/jEwhB3uZNEPRrWhashMx+wCdGTR/ iC7uL2rjqkGkI0x+J0nbZhc= =gSbe -----END PGP SIGNATURE----- From james.britt at gmail.com Wed Nov 2 10:02:06 2005 From: james.britt at gmail.com (James Britt) Date: Wed, 02 Nov 2005 08:02:06 -0700 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130939560.25367.60.camel@hal> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939560.25367.60.camel@hal> Message-ID: <4368D4EE.3010506@gmail.com> Tom Copeland wrote: ... > Cool, OK, hm, let me see... hm. Could you move them in a "/gems/" > directory, perhaps? That'll make my redirecting easier on this end... > my Apache-fu is weak... > I'm waiting to get the 30secondrule.com domain moved to a new server, so I don't have the URL ready. What is the preferred format for the URL? Just something that ends in '/gems/'? SUuh as: http://www.30secondrule.com/ruby/mirror/gems What happens if a user goes to install a gem, gets a redirection to a mirror, and the mirror is down? What should my cron task look like to fetch the gems and index? (I think I can find this info by digging through past E-mail, but I sort of hoping we can start assembling a FAQ on this, so having all this info in one response in the archives might be a start on that. Make it easier for others to start mirroring) Do people/companies offering up mirrors get some public acknowledgment on the rubyforge.org home page( e.g. 'Gem hosting mirrors provided by ...')? Anything else a mirror site needs to know or do? Thanks, James From tom at infoether.com Wed Nov 2 10:03:41 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 10:03:41 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> Message-ID: <1130943821.25367.127.camel@hal> On Wed, 2005-11-02 at 09:54 -0500, Austin Ziegler wrote: > Tom and fellow mirror-providers, > > In following the discussion on mirroring RubyGems files, I made the > suggestion that we perhaps have a large pipe mirror get the initial > seed of a file and then the rest of us get it from the large pipe > mirror. This would save Tom, for example, from having to push out six > copies of Instant Rails when Curt Hibbs updates it at *35 Mb. > > Unfortunately, while I have a good bandwidth allocation (100Gb per > month), I don't have the ability to rsync or otherwise pull. I have > locked down the RubyForge mirror directory, so I have no problem with > the mirror-mirror having the same authentication information that Tom > currently has for pushing. That is, Tom pushes to the mirror-mirror > (which seems to be likely to be Dennis at this point) and then we give > our mirror-auth information to Dennis and he can push to the rest of > us. > > How does everyone else feel about that? +1 - although, Dennis, perhaps you can continue to pull from RubyForge for a bit while I get the push thing sorted out? And you can push to others as you and others see fit... Yours, tom From tom at infoether.com Wed Nov 2 10:04:35 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 10:04:35 -0500 Subject: [Support-mirrors] rsync pull options In-Reply-To: <240FE031-3619-491E-8BBB-EE8227CF1628@chadfowler.com> References: <1130940412.25367.80.camel@hal> <240FE031-3619-491E-8BBB-EE8227CF1628@chadfowler.com> Message-ID: <1130943875.25367.129.camel@hal> On Wed, 2005-11-02 at 09:36 -0500, Chad Fowler wrote: > I'm completely rsync-ignorant, so I just copied and pasted the > command you posted in an email last week :) By the time this settles out we'll all have new resume bullets :-) Tom From chad at chadfowler.com Wed Nov 2 10:05:45 2005 From: chad at chadfowler.com (Chad Fowler) Date: Wed, 2 Nov 2005 10:05:45 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <1130943821.25367.127.camel@hal> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> Message-ID: On 02-Nov-05, at 10:03 AM, Tom Copeland wrote: > On Wed, 2005-11-02 at 09:54 -0500, Austin Ziegler wrote: >> Tom and fellow mirror-providers, >> >> In following the discussion on mirroring RubyGems files, I made the >> suggestion that we perhaps have a large pipe mirror get the initial >> seed of a file and then the rest of us get it from the large pipe >> mirror. This would save Tom, for example, from having to push out six >> copies of Instant Rails when Curt Hibbs updates it at *35 Mb. >> >> Unfortunately, while I have a good bandwidth allocation (100Gb per >> month), I don't have the ability to rsync or otherwise pull. I have >> locked down the RubyForge mirror directory, so I have no problem with >> the mirror-mirror having the same authentication information that Tom >> currently has for pushing. That is, Tom pushes to the mirror-mirror >> (which seems to be likely to be Dennis at this point) and then we >> give >> our mirror-auth information to Dennis and he can push to the rest of >> us. >> >> How does everyone else feel about that? > > +1 - although, Dennis, perhaps you can continue to pull from RubyForge > for a bit while I get the push thing sorted out? And you can push to > others as you and others see fit... > Can someone tell me what I need to do to be the recipient of a push? Chad From tom at infoether.com Wed Nov 2 10:08:44 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 10:08:44 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <4368D4EE.3010506@gmail.com> References: <1130851739.11439.63.camel@hal> <1130857492.30609.605.camel@linus> <1130939560.25367.60.camel@hal> <4368D4EE.3010506@gmail.com> Message-ID: <1130944124.25367.134.camel@hal> On Wed, 2005-11-02 at 08:02 -0700, James Britt wrote: > What is the preferred format for the URL? Just something that ends in > '/gems/'? SUuh as: > > http://www.30secondrule.com/ruby/mirror/gems Precisely. > What happens if a user goes to install a gem, gets a redirection to a > mirror, and the mirror is down? Hm, good question, not sure how to deal with that. Maybe we need a monitor thingy somewhere that tracks mirrors... > What should my cron task look like to fetch the gems and index? (I > think I can find this info by digging through past E-mail, but I sort of > hoping we can start assembling a FAQ on this, so having all this info in > one response in the archives might be a start on that. Make it easier > for others to start mirroring) That's a good idea... although it might be changing now as we are moving towards server push. > Do people/companies offering up mirrors get some public acknowledgment > on the rubyforge.org home page( e.g. 'Gem hosting mirrors provided by ...')? Yup, they're listed here: http://rubyforge.org/credits/ Pls let me know if I missed anyone... > Anything else a mirror site needs to know or do? Hm, you're right, I do need to write up a mirror doc. Yours, Tom From halostatue at gmail.com Wed Nov 2 10:13:24 2005 From: halostatue at gmail.com (Austin Ziegler) Date: Wed, 2 Nov 2005 10:13:24 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> Message-ID: <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> On 11/2/05, Chad Fowler wrote: > Can someone tell me what I need to do to be the recipient of a push? Well, I think that the next step is for Dennis to get in touch with us individually so that we can give him FTP access credentials. He'll need to distinguish between the full mirrors (like me, willing to do file+gem), file mirrors (I don't know if we have any) and gem mirrors (I'm guessing like you), but that's more or less what would be needed. After Dennis gets his end worked out and the credentials needed to log into our FTP servers, then Tom should stop pushing to each mirror as appropriate. -austin -- Austin Ziegler * halostatue at gmail.com * Alternate: austin at halostatue.ca From tom at infoether.com Wed Nov 2 10:16:10 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 10:16:10 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> Message-ID: <1130944570.25367.137.camel@hal> On Wed, 2005-11-02 at 10:13 -0500, Austin Ziegler wrote: > On 11/2/05, Chad Fowler wrote: > > Can someone tell me what I need to do to be the recipient of a push? > > Well, I think that the next step is for Dennis to get in touch with us > individually so that we can give him FTP access credentials. He'll > need to distinguish between the full mirrors (like me, willing to do > file+gem), file mirrors (I don't know if we have any) and gem mirrors > (I'm guessing like you), but that's more or less what would be needed. > > After Dennis gets his end worked out and the credentials needed to log > into our FTP servers, then Tom should stop pushing to each mirror as > appropriate. Hi Austin - Currently yours is the only mirror which I'm pushing files via FTP - everyone else is pulling data from RubyForge using rysnc. But yup, as Dennis gets a chance, he can coordinate pushing files to the other folks and they can stop pulling from RubyForge. Good times... Yours, Tom From james.britt at gmail.com Wed Nov 2 10:48:15 2005 From: james.britt at gmail.com (James Britt) Date: Wed, 02 Nov 2005 08:48:15 -0700 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <1130944570.25367.137.camel@hal> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> Message-ID: <4368DFBF.9030005@gmail.com> Tom Copeland wrote: ... > > > Hi Austin - > > Currently yours is the only mirror which I'm pushing files via FTP - > everyone else is pulling data from RubyForge using rysnc. > > But yup, as Dennis gets a chance, he can coordinate pushing files to the > other folks and they can stop pulling from RubyForge. Good times... > What are the pros and cons of ftp vs. rsync? Is the idea that as new/changed files appear, then are ftp'd from one mirror to another? How do we handle deleted gems? How many gem versions should a mirror hold? For example, does each mirror need to have versions 0.1.0 to 0.25.0, and onwards, of Nitro? I believe an advantage to rsync over ftp is that file deletions, when occurring at a source, are invoked at the sync client as well (or at at least that is an option, so that both ends are indeed mirrors). James Note to self: This is one of those lists where one must remember to hit 'reply to all' every single time. :( From halostatue at gmail.com Wed Nov 2 11:12:20 2005 From: halostatue at gmail.com (Austin Ziegler) Date: Wed, 2 Nov 2005 11:12:20 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <4368DFBF.9030005@gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> Message-ID: <9e7db9110511020812t77d1e751pb1427a235a701b76@mail.gmail.com> On 11/2/05, James Britt wrote: > Tom Copeland wrote: > > Hi Austin - > > Currently yours is the only mirror which I'm pushing files via FTP - > > everyone else is pulling data from RubyForge using rysnc. > > > > But yup, as Dennis gets a chance, he can coordinate pushing files to the > > other folks and they can stop pulling from RubyForge. Good times... > What are the pros and cons of ftp vs. rsync? FTP has a low overhead; rsync has a high overhead (in terms of CPU). Some of us, however, don't have an option for rsync on the pipe we're using. (When my current contract term for halostatue.info is up -- 24 months to go, I think -- I'll look at upgrading to something I could use with rsync; as it is now, it's WAY too cheap to upgrade now.) I'd prefer rsync or even zsync (where the CPU load is placed mostly on the client), but that's just not an option for this host. -austin -- Austin Ziegler * halostatue at gmail.com * Alternate: austin at halostatue.ca From dennis at lauschmusik.de Wed Nov 2 11:33:59 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 02 Nov 2005 17:33:59 +0100 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <9e7db9110511020812t77d1e751pb1427a235a701b76@mail.gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <9e7db9110511020812t77d1e751pb1427a235a701b76@mail.gmail.com> Message-ID: <4368EA77.6070401@lauschmusik.de> Austin Ziegler wrote: >FTP has a low overhead; rsync has a high overhead (in terms of CPU). >Some of us, however, don't have an option for rsync on the pipe we're >using. (When my current contract term for halostatue.info is up -- 24 >months to go, I think -- I'll look at upgrading to something I could >use with rsync; as it is now, it's WAY too cheap to upgrade now.) > >I'd prefer rsync or even zsync (where the CPU load is placed mostly on >the client), but that's just not an option for this host. > >-austin > I strongly suggest using rsync, because it's just made for jobs like this, but it should not be a problem to support both ftp- and rsync-mirrors from my side. I will not be able to set this up today though, but I will start tomorrow. Kind regards, Dennis Oelkers From tom at infoether.com Wed Nov 2 16:58:17 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 02 Nov 2005 16:58:17 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <4368DFBF.9030005@gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> Message-ID: <1130968697.8367.26.camel@hal> On Wed, 2005-11-02 at 08:48 -0700, James Britt wrote: > Tom Copeland wrote: > ... > > > > > > Hi Austin - > > > > Currently yours is the only mirror which I'm pushing files via FTP - > > everyone else is pulling data from RubyForge using rysnc. > > > > But yup, as Dennis gets a chance, he can coordinate pushing files to the > > other folks and they can stop pulling from RubyForge. Good times... > > > > What are the pros and cons of ftp vs. rsync? rsync only transfers files if they've changed, not sure if FTP has some sort of flag for doing this. > Is the idea that as new/changed files appear, then are ftp'd from one > mirror to another? Yup. > How do we handle deleted gems? Good question... > How many gem versions should a mirror hold? For example, does each > mirror need to have versions 0.1.0 to 0.25.0, and onwards, of Nitro? Sure keeps things simpler if everyone's got everything. > I believe an advantage to rsync over ftp is that file deletions, when > occurring at a source, are invoked at the sync client as well (or at at > least that is an option, so that both ends are indeed mirrors). Yup, right on, "rsync --delete" for client pull. Yours, Tom From tom at infoether.com Thu Nov 3 08:38:00 2005 From: tom at infoether.com (Tom Copeland) Date: Thu, 03 Nov 2005 08:38:00 -0500 Subject: [Support-mirrors] A nice traffic reduction on gems.rubyforge.org Message-ID: <1131025080.8367.128.camel@hal> Cool, look at the dropoff in traffic on gems.rubyforge.org: http://gems.rubyforge.org/usage/usage_200511.html#DAYSTATS Less RubyForge traffic and faster downloads for users... good times... Yours, tom From chad at chadfowler.com Thu Nov 3 09:09:44 2005 From: chad at chadfowler.com (chad@chadfowler.com) Date: Thu, 3 Nov 2005 09:09:44 -0500 (EST) Subject: [Support-mirrors] A nice traffic reduction on gems.rubyforge.org In-Reply-To: <1131025080.8367.128.camel@hal> References: <1131025080.8367.128.camel@hal> Message-ID: <4667.12.208.248.218.1131026984.squirrel@12.208.248.218> > Cool, look at the dropoff in traffic on gems.rubyforge.org: > > http://gems.rubyforge.org/usage/usage_200511.html#DAYSTATS > > Less RubyForge traffic and faster downloads for users... good times... > > Yours, > Great! Tom, I noticed this referrer was the top in the logs: http://gems.rubyforge.org/gems/\\r\\n\\r\\n Anyone know what that is? Looks like a misconfigured script to me. Chad From dennis at lauschmusik.de Thu Nov 3 09:37:24 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Thu, 03 Nov 2005 15:37:24 +0100 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <1130968697.8367.26.camel@hal> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <1130968697.8367.26.camel@hal> Message-ID: <436A20A4.4090401@lauschmusik.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey folks, I set up a rudimentary mirroring script for the push mechanism, now all I need is the mirroring info for all mirrors who wish to receive the updates. Please do not forget to encrypt any sensitive data you send to me, my key-id is 0x5D79F323 and available on subkeys.pgp.net. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDaiCkcT9mJF158yMRAiLkAKCgRQe59RYolLBJlmXJ1p2jyt3NKACgvL4d f0O04ceuhr32BQMo6lSc7rc= =EHJh -----END PGP SIGNATURE----- From tom at infoether.com Fri Nov 4 08:38:59 2005 From: tom at infoether.com (Tom Copeland) Date: Fri, 04 Nov 2005 08:38:59 -0500 Subject: [Support-mirrors] A nice traffic reduction on gems.rubyforge.org In-Reply-To: <4667.12.208.248.218.1131026984.squirrel@12.208.248.218> References: <1131025080.8367.128.camel@hal> <4667.12.208.248.218.1131026984.squirrel@12.208.248.218> Message-ID: <1131111539.24884.85.camel@hal> On Thu, 2005-11-03 at 09:09 -0500, chad at chadfowler.com wrote: > Tom, I noticed this referrer was the top in the logs: > > http://gems.rubyforge.org/gems/\\r\\n\\r\\n > > Anyone know what that is? Looks like a misconfigured script to me. I think it's people hitting gems.rubyforge.org/gems/ - and that's followed by two carriage returns to complete the HTTP request. Yours, Tom From tom at infoether.com Fri Nov 4 08:40:11 2005 From: tom at infoether.com (Tom Copeland) Date: Fri, 04 Nov 2005 08:40:11 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <436A20A4.4090401@lauschmusik.de> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <1130968697.8367.26.camel@hal> <436A20A4.4090401@lauschmusik.de> Message-ID: <1131111611.24884.88.camel@hal> On Thu, 2005-11-03 at 15:37 +0100, Dennis Oelkers wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hey folks, > > I set up a rudimentary mirroring script for the push mechanism, > now all I need is the mirroring info for all mirrors who wish to > receive the updates. > > Please do not forget to encrypt any sensitive data you send to > me, my key-id is 0x5D79F323 and available on subkeys.pgp.net. Great! If folks can go ahead and do that, I can then reconfigure RubyForge to only sync with Dennis' server - I'll just only accept rsync connections from his machine. That'll further reduce RubyForge network traffic... good stuff! Yours, Tom From chad at chadfowler.com Fri Nov 4 08:48:42 2005 From: chad at chadfowler.com (Chad Fowler) Date: Fri, 4 Nov 2005 08:48:42 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <1131111611.24884.88.camel@hal> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <1130968697.8367.26.camel@hal> <436A20A4.4090401@lauschmusik.de> <1131111611.24884.88.camel@hal> Message-ID: On 04-Nov-05, at 8:40 AM, Tom Copeland wrote: > On Thu, 2005-11-03 at 15:37 +0100, Dennis Oelkers wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hey folks, >> >> I set up a rudimentary mirroring script for the push mechanism, >> now all I need is the mirroring info for all mirrors who wish to >> receive the updates. >> >> Please do not forget to encrypt any sensitive data you send to >> me, my key-id is 0x5D79F323 and available on subkeys.pgp.net. > > Great! If folks can go ahead and do that, I can then reconfigure > RubyForge to only sync with Dennis' server - I'll just only accept > rsync > connections from his machine. > > That'll further reduce RubyForge network traffic... good stuff! > I'm stuck in encryption configuration land for now. Will let people know when I figure out how to configure my mail client to encrypt a message (I had it working before upgrading to Tiger, but I use it so infrequently that I don't remember). Chad From chad at chadfowler.com Fri Nov 4 08:50:56 2005 From: chad at chadfowler.com (Chad Fowler) Date: Fri, 4 Nov 2005 08:50:56 -0500 Subject: [Support-mirrors] A nice traffic reduction on gems.rubyforge.org In-Reply-To: <1131111539.24884.85.camel@hal> References: <1131025080.8367.128.camel@hal> <4667.12.208.248.218.1131026984.squirrel@12.208.248.218> <1131111539.24884.85.camel@hal> Message-ID: <40E34B84-B983-40A1-95DA-A506B1F92BC8@chadfowler.com> On 04-Nov-05, at 8:38 AM, Tom Copeland wrote: > On Thu, 2005-11-03 at 09:09 -0500, chad at chadfowler.com wrote: >> Tom, I noticed this referrer was the top in the logs: >> >> http://gems.rubyforge.org/gems/\\r\\n\\r\\n >> >> Anyone know what that is? Looks like a misconfigured script to me. > > I think it's people hitting gems.rubyforge.org/gems/ - and that's > followed by two carriage returns to complete the HTTP request. > Yea, sorry. I wasn't totally clear in my question. It just looks suspect in that the CR/LFs are coming through and escaped in my referrer logs. Made me think someone had something misconfigured somewhere. Chad From james.britt at gmail.com Fri Nov 4 09:16:15 2005 From: james.britt at gmail.com (James Britt) Date: Fri, 04 Nov 2005 07:16:15 -0700 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <1131111611.24884.88.camel@hal> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <1130968697.8367.26.camel@hal> <436A20A4.4090401@lauschmusik.de> <1131111611.24884.88.camel@hal> Message-ID: <436B6D2F.90800@gmail.com> Tom Copeland wrote: > On Thu, 2005-11-03 at 15:37 +0100, Dennis Oelkers wrote: > >>-----BEGIN PGP SIGNED MESSAGE----- >>Hash: SHA1 >> >>Hey folks, >> >>I set up a rudimentary mirroring script for the push mechanism, >>now all I need is the mirroring info for all mirrors who wish to >>receive the updates. >> >>Please do not forget to encrypt any sensitive data you send to >>me, my key-id is 0x5D79F323 and available on subkeys.pgp.net. > > > Great! If folks can go ahead and do that, I can then reconfigure > RubyForge to only sync with Dennis' server - I'll just only accept rsync > connections from his machine. Now I'm confused. I was geting set to do the periodic (every 1/3 hour) rsync with RubyForge. Now, instead, I do not rsync, but have to open up some means to get data pushed to my server? What exactly is that? James From tom at infoether.com Mon Nov 7 08:50:12 2005 From: tom at infoether.com (Tom Copeland) Date: Mon, 07 Nov 2005 08:50:12 -0500 Subject: [Support-mirrors] A nice traffic reduction on gems.rubyforge.org In-Reply-To: <40E34B84-B983-40A1-95DA-A506B1F92BC8@chadfowler.com> References: <1131025080.8367.128.camel@hal> <4667.12.208.248.218.1131026984.squirrel@12.208.248.218> <1131111539.24884.85.camel@hal> <40E34B84-B983-40A1-95DA-A506B1F92BC8@chadfowler.com> Message-ID: <1131371412.13243.160.camel@hal> On Fri, 2005-11-04 at 08:50 -0500, Chad Fowler wrote: > On 04-Nov-05, at 8:38 AM, Tom Copeland wrote: > > > On Thu, 2005-11-03 at 09:09 -0500, chad at chadfowler.com wrote: > >> Tom, I noticed this referrer was the top in the logs: > >> > >> http://gems.rubyforge.org/gems/\\r\\n\\r\\n > >> > >> Anyone know what that is? Looks like a misconfigured script to me. > > > > I think it's people hitting gems.rubyforge.org/gems/ - and that's > > followed by two carriage returns to complete the HTTP request. > > > > Yea, sorry. I wasn't totally clear in my question. It just looks > suspect in that the CR/LFs are coming > through and escaped in my referrer logs. Made me think someone had > something misconfigured somewhere. Hm. Maybe... I wonder if my RewriteMap is picking up those /gems/ requests... maybe I can change it to /(.*).gem or something like that. I'll fiddle with it. Thanks, tom From tom at infoether.com Mon Nov 7 08:51:25 2005 From: tom at infoether.com (Tom Copeland) Date: Mon, 07 Nov 2005 08:51:25 -0500 Subject: [Support-mirrors] Distributed Mirroring ;) In-Reply-To: <436B6D2F.90800@gmail.com> References: <9e7db9110511020654i163dd44dt261236d55678a344@mail.gmail.com> <1130943821.25367.127.camel@hal> <9e7db9110511020713k3fe3b06dldd4681753cf11b41@mail.gmail.com> <1130944570.25367.137.camel@hal> <4368DFBF.9030005@gmail.com> <1130968697.8367.26.camel@hal> <436A20A4.4090401@lauschmusik.de> <1131111611.24884.88.camel@hal> <436B6D2F.90800@gmail.com> Message-ID: <1131371485.13243.163.camel@hal> On Fri, 2005-11-04 at 07:16 -0700, James Britt wrote: > > Great! If folks can go ahead and do that, I can then reconfigure > > RubyForge to only sync with Dennis' server - I'll just only accept rsync > > connections from his machine. > > > Now I'm confused. I was geting set to do the periodic (every 1/3 hour) > rsync with RubyForge. Yup, that's the way things are working at the moment. > Now, instead, I do not rsync, but have to open up > some means to get data pushed to my server? Yup, that seems like what we're moving towards. It'll lighten the network load on RubyForge and won't require you to set up a cron job; all the work gets transferred to Dennis' shoulders :-) Yours, tom From james.britt at gmail.com Tue Nov 8 16:39:36 2005 From: james.britt at gmail.com (James Britt) Date: Tue, 8 Nov 2005 14:39:36 -0700 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1130939482.25367.58.camel@hal> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> Message-ID: <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> I'm hoping to finally get set-up for mirroring off the correct server, and tried running this command so as to fetch the gems while I wait for nameserver changes to happen. rsync -r -z -t anonymous at rubyforge.org::gem-mirror I see a long list of file names shoot by, but no files are fetched. Any ideas what I'm doing wrong? Thanks, James From tom at infoether.com Wed Nov 9 10:25:34 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 09 Nov 2005 10:25:34 -0500 Subject: [Support-mirrors] Switching over to mirroring from Dennis' server Message-ID: <1131549934.30395.62.camel@hal> Hi all - I think Dennis is all set up with the "master mirror" server as proposed/discussed here: http://rubyforge.org/pipermail/support-mirrors/2005-November/000051.html So I plan to modify RubyForge's xinetd.conf to only allow rsync connections from Dennis' machine. Then I'll remove everyone from the mirror rotation except for Dennis and Austin, who are (as far as I know?) the only ones that are set up under the new scheme. As everyone gets their rsync information to Dennis, he can add them to his syncing script and let me know. Then I can readd them to the mirror rotation to share the load! I'll keep track of things by adding folks back to this page: http://rubyforge.org/credits/ as they're added back to the rotation. Questions/comments are welcome, Thanks, Tom From tom at infoether.com Wed Nov 9 10:27:03 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 09 Nov 2005 10:27:03 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> Message-ID: <1131550023.30395.64.camel@hal> On Tue, 2005-11-08 at 14:39 -0700, James Britt wrote: > I'm hoping to finally get set-up for mirroring off the correct server, > and tried running this command so as to fetch the gems while I wait > for nameserver changes to happen. > > rsync -r -z -t anonymous at rubyforge.org::gem-mirror > > I see a long list of file names shoot by, but no files are fetched. > > Any ideas what I'm doing wrong? Hm, I think you may need to do something like: rsync -r -z -t anonymous at rubyforge.org::gem-mirror . i.e., specify a target directory. But anyhow, you may want to switch things around to allow Dennis' machine to sync to yours instead... that's the wave of the future! Yours, Tom From james.britt at gmail.com Wed Nov 9 10:33:43 2005 From: james.britt at gmail.com (James Britt) Date: Wed, 09 Nov 2005 08:33:43 -0700 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1131550023.30395.64.camel@hal> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> Message-ID: <437216D7.7050606@gmail.com> Tom Copeland wrote: > On Tue, 2005-11-08 at 14:39 -0700, James Britt wrote: > >>I'm hoping to finally get set-up for mirroring off the correct server, >>and tried running this command so as to fetch the gems while I wait >>for nameserver changes to happen. >> >>rsync -r -z -t anonymous at rubyforge.org::gem-mirror >> >>I see a long list of file names shoot by, but no files are fetched. >> >>Any ideas what I'm doing wrong? > > > Hm, I think you may need to do something like: > > rsync -r -z -t anonymous at rubyforge.org::gem-mirror . > > i.e., specify a target directory. > > But anyhow, you may want to switch things around to allow Dennis' > machine to sync to yours instead... that's the wave of the future! What does that mean, "allow Dennis' machine to sync to yours "? What specifically do I have to do to get rsync playing nice with his machine? James From tom at infoether.com Wed Nov 9 11:28:45 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 09 Nov 2005 11:28:45 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <437216D7.7050606@gmail.com> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> Message-ID: <1131553725.30395.104.camel@hal> On Wed, 2005-11-09 at 08:33 -0700, James Britt wrote: > > But anyhow, you may want to switch things around to allow Dennis' > > machine to sync to yours instead... that's the wave of the future! > > > What does that mean, "allow Dennis' machine to sync to yours "? > What specifically do I have to do to get rsync playing nice with his > machine? Cool, let's nail this down so's I can put it in the mirroring docs. I _think_ that you need to open up port 873 and allow Dennis' machine to push new files to your mirror directory. So you would need to add a new rsync endpoint in /etc/rsyncd.conf. Dennis, does that sound right? Do you know what that entry would look like? I've got plenty of examples of "pull files from me" rsyncd.conf entries, but none of "push files to me" entries... Thanks, Tom From dennis at lauschmusik.de Wed Nov 9 14:45:04 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 9 Nov 2005 20:45:04 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1131553725.30395.104.camel@hal> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> Message-ID: <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 09.11.2005 um 17:28 schrieb Tom Copeland: > Cool, let's nail this down so's I can put it in the mirroring docs. I > _think_ that you need to open up port 873 and allow Dennis' machine to > push new files to your mirror directory. So you would need to add > a new > rsync endpoint in /etc/rsyncd.conf. > > Dennis, does that sound right? Do you know what that entry would look > like? I've got plenty of examples of "pull files from me" rsyncd.conf > entries, but none of "push files to me" entries... In fact it is quite similar to a read-only entry, you just have to add a secrets file and a list of allowed users. One thing you have to bear in mind is, that the user rsync is running as has to be allowed to read and write to the directories and files I am supposed to update. A definition like this should be sufficient: # this is the head of the config file uid = rubyforge-owner gid = rubyforge-owner # this is the entry for the rubyforge repository [file-mirror] path = /var/www/rubyforge/htdocs auth users = rubyforge secrets file = /var/www/rubyforge/etc/rsync.secrets hosts allow = rubyforge.lauschmusik.de hosts deny = * where rubyforge-owner is the user/group which owns (or at least has read/write-access to) the path and /var/www/rubyforge/etc/rsync.secrets looks like: rubyforge:mightysecretpassword One thing you could add is the chroot option for the entry of the repository, so that rsyncd chroot()s to the path of it when it is accessed, YMMV though. I hope this helps a bit, if there is need for further support, please feel free to contact me. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDclHIcT9mJF158yMRAnW1AKDDnxnnzWMRxvc55gnGTT71mFeZTACgpMDZ SoFZAyflO3l0wzRTGDSpdPg= =1NMF -----END PGP SIGNATURE----- From james.britt at gmail.com Wed Nov 9 17:36:20 2005 From: james.britt at gmail.com (James Britt) Date: Wed, 09 Nov 2005 15:36:20 -0700 Subject: [Support-mirrors] Gem mirror In-Reply-To: <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> Message-ID: <437279E4.5050009@gmail.com> Dennis Oelkers wrote: > > In fact it is quite similar to a read-only entry, you just have to > add a secrets > file and a list of allowed users. One thing you have to bear in mind > is, that > the user rsync is running as has to be allowed to read and write to the > directories and files I am supposed to update. This is really quite foreign to me; what happened to the idea of mirrors simply fetching new/changed files by running an rsync cron job? > > A definition like this should be sufficient: > > # this is the head of the config file > uid = rubyforge-owner > gid = rubyforge-owner A definition "like this" goes where? > > # this is the entry for the rubyforge repository > [file-mirror] > path = /var/www/rubyforge/htdocs > auth users = rubyforge > secrets file = /var/www/rubyforge/etc/rsync.secrets > hosts allow = rubyforge.lauschmusik.de > hosts deny = * In what file? > > where rubyforge-owner is the user/group which owns (or at least has > read/write-access to) the path and /var/www/rubyforge/etc/rsync.secrets > looks like: > > rubyforge:mightysecretpassword > > One thing you could add is the chroot option for the entry of the > repository, > so that rsyncd chroot()s to the path of it when it is accessed, YMMV > though. I can see I'll be playing sys admin here, not my favorite role. > > I hope this helps a bit, if there is need for further support, please > feel free > to contact me. Do I need to have rsyncd running at all times? Do I need to create a special user account for the rsync client? Do I need to explicitly open certain ports? Does my mirror directory need special permissions? Thanks, James From dennis at lauschmusik.de Wed Nov 9 17:46:24 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 9 Nov 2005 23:46:24 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <437279E4.5050009@gmail.com> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> <437279E4.5050009@gmail.com> Message-ID: <7805011A-5659-443C-9312-55275E88160C@lauschmusik.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 09.11.2005 um 23:36 schrieb James Britt: > This is really quite foreign to me; what happened to the idea of > mirrors > simply fetching new/changed files by running an rsync cron job? We decided to advance to a push-mechanism to keep the load on the main repository and the time until a file has been distributed to all mirrors low. (Especially the last part is important to establish a good quality of service) > A definition "like this" goes where? > In what file? Both parts belong to $whatever/etc/rsyncd.conf The usual command to start rsyncd would be something like "rsyncd --daemon --config=$whatever/rsyncd.conf" > > Do I need to have rsyncd running at all times? Yes. > Do I need to create a special user account for the rsync client? No, it has its own user management. > Do I need to explicitly open certain ports? If you have a packet filter ("firewall") running, you should allow incoming connections from rubyforge.lauschmusik.de to 873/tcp. > Does my mirror directory need special permissions? It has to be writable by rsyncd, just like the files and directories it contains. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDcnxEcT9mJF158yMRAtRtAKDYk5eiCyvtBYtYqDEvQXMN0yIa/QCeJcda WNzqM163BUUkx4m9I8S/2kU= =zbGQ -----END PGP SIGNATURE----- From tom at infoether.com Wed Nov 9 17:50:53 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 09 Nov 2005 17:50:53 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: <7805011A-5659-443C-9312-55275E88160C@lauschmusik.de> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> <437279E4.5050009@gmail.com> <7805011A-5659-443C-9312-55275E88160C@lauschmusik.de> Message-ID: <1131576653.30395.183.camel@hal> On Wed, 2005-11-09 at 23:46 +0100, Dennis Oelkers wrote: > > Do I need to have rsyncd running at all times? > > Yes. Hm... could it be kicked off with xinetd? Yours, Tom From dennis at lauschmusik.de Wed Nov 9 17:53:01 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 9 Nov 2005 23:53:01 +0100 Subject: [Support-mirrors] Gem mirror In-Reply-To: <1131576653.30395.183.camel@hal> References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> <437279E4.5050009@gmail.com> <7805011A-5659-443C-9312-55275E88160C@lauschmusik.de> <1131576653.30395.183.camel@hal> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 09.11.2005 um 23:50 schrieb Tom Copeland: > On Wed, 2005-11-09 at 23:46 +0100, Dennis Oelkers wrote: >>> Do I need to have rsyncd running at all times? >> >> Yes. > > Hm... could it be kicked off with xinetd? Uhm, okay, I was quite unprecise there. It has to be _available_ at all times, but it can be run either standalone or by a superserver like (x)inetd. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDcn3NcT9mJF158yMRAjeEAJwJC6UeN1s0JenL2j801BWkYEKUewCgluBj /u0vksoXFU7I02EGfDsGDtA= =3ipr -----END PGP SIGNATURE----- From tom at infoether.com Wed Nov 9 21:13:30 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 09 Nov 2005 21:13:30 -0500 Subject: [Support-mirrors] Gem mirror In-Reply-To: References: <1130851739.11439.63.camel@hal> <1130939482.25367.58.camel@hal> <5b1d0c600511081339p3b1fa854pfd2a44ce499cabda@mail.gmail.com> <1131550023.30395.64.camel@hal> <437216D7.7050606@gmail.com> <1131553725.30395.104.camel@hal> <835147E5-A305-4779-97D4-D01F88F9E68A@lauschmusik.de> <437279E4.5050009@gmail.com> <7805011A-5659-443C-9312-55275E88160C@lauschmusik.de> <1131576653.30395.183.camel@hal> Message-ID: <1131588810.30395.218.camel@hal> On Wed, 2005-11-09 at 23:53 +0100, Dennis Oelkers wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Am 09.11.2005 um 23:50 schrieb Tom Copeland: > > > On Wed, 2005-11-09 at 23:46 +0100, Dennis Oelkers wrote: > >>> Do I need to have rsyncd running at all times? > >> > >> Yes. > > > > Hm... could it be kicked off with xinetd? > > Uhm, okay, I was quite unprecise there. It has to be > _available_ at all times, but it can be run either > standalone or by a superserver like (x)inetd. Cool, OK, right on. Dennis, also, is Austin's gem mirror set up? I've got his server in the file mirror rotation, but I haven't put it in the gem mirror rotation yet. Thanks, Tom From tom at infoether.com Wed Nov 16 13:12:29 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 16 Nov 2005 13:12:29 -0500 Subject: [Support-mirrors] Documentation updated, and 2 gem/file mirrors now online Message-ID: <1132164749.27790.151.camel@hal> Hi all - I just noticed that Austin's Gems seem to be sync'd up, so I've added him to the RubyGem mirror rotation; now it's both Dennis and Auston for both RubyGem and file mirrors: http://rubyforge.org/credits/ Everyone else, as you have a chance, please contact Dennis with your server info so he can push files your way. Thanks much! Also, I updated the "RubyForge Mirroring" document here: http://rubyforge.org/docman/?group_id=5 to include information on the new architecture. Comments/improvements/questions are welcome! Thanks, Tom From james.britt at gmail.com Wed Nov 16 13:50:36 2005 From: james.britt at gmail.com (James Britt) Date: Wed, 16 Nov 2005 11:50:36 -0700 Subject: [Support-mirrors] Documentation updated, and 2 gem/file mirrors now online In-Reply-To: <1132164749.27790.151.camel@hal> References: <1132164749.27790.151.camel@hal> Message-ID: <437B7F7C.8000806@gmail.com> Tom Copeland wrote: > Hi all - > > I just noticed that Austin's Gems seem to be sync'd up, so I've added > him to the RubyGem mirror rotation; now it's both Dennis and Auston for > both RubyGem and file mirrors: > > http://rubyforge.org/credits/ > > Everyone else, as you have a chance, please contact Dennis with your > server info so he can push files your way. Thanks much! > > Also, I updated the "RubyForge Mirroring" document here: > > http://rubyforge.org/docman/?group_id=5 > > to include information on the new architecture. > Comments/improvements/questions are welcome! Very good. I've set up, perhaps even correctly, the rsync stuff on 30secondrule.com, so I should be able to mirror soon. Is there a good way to test my configuration? Thanks, James -- http://www.ruby-doc.org - Ruby Help & Documentation http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted http://www.rubystuff.com - The Ruby Store for Ruby Stuff http://www.jamesbritt.com - Playing with Better Toys http://www.30secondrule.com - Building Better Tools From tom at infoether.com Wed Nov 16 14:05:30 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 16 Nov 2005 14:05:30 -0500 Subject: [Support-mirrors] Documentation updated, and 2 gem/file mirrors now online In-Reply-To: <437B7F7C.8000806@gmail.com> References: <1132164749.27790.151.camel@hal> <437B7F7C.8000806@gmail.com> Message-ID: <1132167930.27790.161.camel@hal> On Wed, 2005-11-16 at 11:50 -0700, James Britt wrote: > Very good. I've set up, perhaps even correctly, the rsync stuff on > 30secondrule.com, so I should be able to mirror soon. Cool! That'll be great; right now Dennis and Austin are shouldering the entire load. > Is there a good way to test my configuration? Hm. You could try pushing some files on there, i.e.: rsync -av local_directory rubyforge at remotehost:target_directory and see if it can connect and copy up the files. I didn't try the above snippet, but it's something along those lines; Dennis, please correct me if I'm presenting that incorrectly... Thanks, Tom From tom at infoether.com Tue Nov 22 11:57:19 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 22 Nov 2005 11:57:19 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems Message-ID: <005c01c5ef85$ce13f370$0201a8c0@TOMHP> Hi all - It seems like new gems aren't being pushed to Austin's server... when I do a "gem install gruff" against halostatue I get an HTTP 300. Maybe I'm redirecting to the wrong directory? I'm redirecting Gem requests to rubyforge.halostatue.info/gems/, Austin, is that the directory that's getting sync'd? Also, shouldn't this just return an HTTP 404 since the gem's not there? Or maybe you don't have direct control over that, Austin? http://rubyforge.halostatue.info/gems/gruff-0.0.5.gem Thanks, Tom From dennis at lauschmusik.de Tue Nov 22 12:48:15 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Tue, 22 Nov 2005 18:48:15 +0100 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <005c01c5ef85$ce13f370$0201a8c0@TOMHP> References: <005c01c5ef85$ce13f370$0201a8c0@TOMHP> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey folks, Am 22.11.2005 um 17:57 schrieb Tom Copeland: > Hi all - > > It seems like new gems aren't being pushed to Austin's server... > when I > do a "gem install gruff" against halostatue I get an HTTP 300. Maybe > I'm redirecting to the wrong directory? I'm redirecting Gem > requests to > rubyforge.halostatue.info/gems/, Austin, is that the directory that's > getting sync'd? > > Also, shouldn't this just return an HTTP 404 since the gem's not > there? > Or maybe you don't have direct control over that, Austin? > > http://rubyforge.halostatue.info/gems/gruff-0.0.5.gem > Uhm, the problem is on my side, the script I used to trigger the sync of the other mirrors was flaky, I am gonna fix it asap. Austin's host returns a 300, because it (mod_speling?) presents an option for other choices if it does not find the object specified. Btw, Tom, we should talk about setting up the mechanism which pushes the updates from your host to mine soon. Kind regards, Dennis Oelkers -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (Darwin) iD8DBQFDg1nkcT9mJF158yMRAsI1AJ9JzEDvWhx8/mUO5E4MKxUrFedw+ACgltmZ HO0pklmwb7dWRfO4D4V24pc= =wvKl -----END PGP SIGNATURE----- From tom at infoether.com Tue Nov 22 13:48:05 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 22 Nov 2005 13:48:05 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: Message-ID: <002601c5ef95$46ed75a0$0201a8c0@TOMHP> > Uhm, the problem is on my side, the script I used to trigger > the sync of the other mirrors was flaky, I am gonna fix it asap. Ah, cool, thanks much! > Austin's host returns a 300, because it (mod_speling?) presents an > option > for other choices if it does not find the object specified. Hm, yup, OK. Austin, any chance you can turn that off? It's probably better just to fail with a 404 rather than present a list of options in this case... and it'll save CPU time on your end, too... > Btw, Tom, we should talk about setting up the mechanism which > pushes the updates from your host to mine soon. Right you are! Hm. OK, on my end, it's a Ruby cron job which finds new items... how can I notify your machine of a new file? I can hit a URL if that would be helpful... or send an email... or something else, not sure, whatever you think best. Yours, Tom From tom at infoether.com Tue Nov 22 14:03:39 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 22 Nov 2005 14:03:39 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <9e7db9110511221050x34214676ne1edd66b8b211811@mail.gmail.com> Message-ID: <003e01c5ef97$72c4d6d0$0201a8c0@TOMHP> > -----Original Message----- > From: Austin Ziegler [mailto:halostatue at gmail.com] > Sent: Tuesday, November 22, 2005 1:50 PM > To: Tom Copeland > Subject: Re: [Support-mirrors] HTTP 300 error on newer Gems > > > On 11/22/05, Tom Copeland wrote: > > > Austin's host returns a 300, because it (mod_speling?) > presents an > > > option for other choices if it does not find the object specified. > > Hm, yup, OK. Austin, any chance you can turn that off? > It's probably > > better just to fail with a 404 rather than present a list > of options > > in this case... and it'll save CPU time on your end, too... > > Nope. Wish I could, but this is a full virtual hosting system. Ah, OK. Ah well, no big deal... it should work OK most of the time. Thanks, Tom From james.britt at gmail.com Tue Nov 22 15:04:05 2005 From: james.britt at gmail.com (James Britt) Date: Tue, 22 Nov 2005 13:04:05 -0700 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <003e01c5ef97$72c4d6d0$0201a8c0@TOMHP> References: <003e01c5ef97$72c4d6d0$0201a8c0@TOMHP> Message-ID: <438379B5.2060401@gmail.com> Tom Copeland wrote: >>-----Original Message----- >>From: Austin Ziegler [mailto:halostatue at gmail.com] >>Sent: Tuesday, November 22, 2005 1:50 PM >>To: Tom Copeland >>Subject: Re: [Support-mirrors] HTTP 300 error on newer Gems >> >> >>On 11/22/05, Tom Copeland wrote: >> >>>>Austin's host returns a 300, because it (mod_speling?) >> >>presents an >> >>>>option for other choices if it does not find the object specified. >>> >>>Hm, yup, OK. Austin, any chance you can turn that off? >> >>It's probably >> >>>better just to fail with a 404 rather than present a list >> >>of options >> >>>in this case... and it'll save CPU time on your end, too... >> >>Nope. Wish I could, but this is a full virtual hosting system. > > > Ah, OK. Ah well, no big deal... it should work OK most of the time. But is this connected to gem install failures where one gets the message: ERROR: While executing gem ... (OpenURI::HTTPError) 300 Multiple Choices James From tom at infoether.com Tue Nov 22 15:08:54 2005 From: tom at infoether.com (Tom Copeland) Date: Tue, 22 Nov 2005 15:08:54 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <438379B5.2060401@gmail.com> Message-ID: <00cc01c5efa0$91105660$0201a8c0@TOMHP> > > Ah, OK. Ah well, no big deal... it should work OK most of the time. > > But is this connected to gem install failures where one gets > the message: > > ERROR: While executing gem ... (OpenURI::HTTPError) > 300 Multiple Choices Right, but if Austin turned off mod_speling, that would just be a 404 error rather than a 300 error. The underlying problem is that Austin's machine is not getting sync'd properly, and Dennis is working on fixing that... Yours, Tom From tom at infoether.com Wed Nov 23 14:05:56 2005 From: tom at infoether.com (Tom Copeland) Date: Wed, 23 Nov 2005 14:05:56 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: Message-ID: <01e001c5f060$efa40e40$0201a8c0@TOMHP> > > http://rubyforge.halostatue.info/gems/gruff-0.0.5.gem > > > > Uhm, the problem is on my side, the script I used to trigger > the sync of the other mirrors was flaky, I am gonna fix it asap. This seems to be happening again, i.e.: http://rubyforge.halostatue.info/gems/gruff-0.0.6.gem This Gem was released last night, so it should have been sync'd over by now... hm. Dennis, can you take a look at that script again? Thanks, Tom From dennis at lauschmusik.de Thu Nov 24 05:24:22 2005 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Thu, 24 Nov 2005 11:24:22 +0100 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <01e001c5f060$efa40e40$0201a8c0@TOMHP> References: <01e001c5f060$efa40e40$0201a8c0@TOMHP> Message-ID: <438594D6.90108@lauschmusik.de> Tom Copeland wrote: >This seems to be happening again, i.e.: > >http://rubyforge.halostatue.info/gems/gruff-0.0.6.gem > >This Gem was released last night, so it should have been sync'd over by >now... hm. Dennis, can you take a look at that script again? > Should be fixed now. Sorry for the inconvenience. Btw, I will contact you with the details of your part of the push mechanism soon. You will have to trigger the resync of the mirrors by hitting some url authenticating yourself using either a certificate or credentials. I don't know client you would use to do that, so please tell me what your preferred way of authentication is. Kind regards, Dennis Oelkers From tom at infoether.com Fri Nov 25 09:33:13 2005 From: tom at infoether.com (Tom Copeland) Date: Fri, 25 Nov 2005 09:33:13 -0500 Subject: [Support-mirrors] HTTP 300 error on newer Gems In-Reply-To: <438594D6.90108@lauschmusik.de> References: <01e001c5f060$efa40e40$0201a8c0@TOMHP> <438594D6.90108@lauschmusik.de> Message-ID: <1132929193.4479.37.camel@hal> On Thu, 2005-11-24 at 11:24 +0100, Dennis Oelkers wrote: > Should be fixed now. Sorry for the inconvenience. Cool, yup, working now, thanks! > Btw, I will contact you with the details of your part of the push > mechanism soon. You will have to > trigger the resync of the mirrors by hitting some url authenticating > yourself using either a certificate > or credentials. I don't know client you would use to do that, so please > tell me what your preferred > way of authentication is. Great! Hm. I'm running a Ruby script to check for new files... so anything I can do from Ruby works for me. How about HTTP basic authentication - would that work for you? Thanks, Tom