From dennis at lauschmusik.de Mon Nov 12 15:38:09 2007 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Mon, 12 Nov 2007 21:38:09 +0100 Subject: [Support-admins] Change proposal regarding the Rubyforge distribution infrastructure - your input is needed Message-ID: <4738B9B1.5030702@lauschmusik.de> Hello rubyforge, hello rubycentral, For those of you who do not know me: I am the current "maintainer" of the general mirroring architecture of rubyforge, running the master mirror and the infrastructure to push out the changes of the gems and rubyforge files repositories to all the subscribed mirrors. I am currently living in Berlin, Germany, where I study computer sciences (starting my diploma thesis in about a year), and working for Nokia Germany. In our constant efforts to improve the quality of rubyforge, we had to struggle with several crashes of the main (now retired) server hosting the website and the project repositories. At that time, we noticed that the webserver is a crucial component of the distribution chain, because all clients have to contact the main server via http, and are distributed to the different mirrors by following the redirect which is then issued. Although the current hardware is running fine so far, we would like to change this in the long run. The current idea is to transfer the load balancing to dns, and maintain the dns entries using a modified version of the script which is currently generating the apache rewrite map. This is all fine, but has one big drawback: We are not able to do centraslised accouting of the downloads, like we are with the current solution. Gathering the logfiles from the different mirrors is a solution which seems to be too painful to do at the moment. Abstaining from the statistics we are able to generate from the centralised logfiles we currently have, is not an option either. This is the main reason why I wrote this mail: what is your opinion about the general proposal? Do you have a clever idea to solve the logging problem? Is there anything else that you see as a problem/improvable point of the current rubyforge infrastructure? Looking forward to your replies and valued input, kind regards, Dennis Oelkers From tom at infoether.com Wed Nov 14 09:56:49 2007 From: tom at infoether.com (Tom Copeland) Date: Wed, 14 Nov 2007 09:56:49 -0500 Subject: [Support-admins] Change proposal regarding the Rubyforge distribution infrastructure - your input is needed In-Reply-To: <4738B9B1.5030702@lauschmusik.de> References: <4738B9B1.5030702@lauschmusik.de> Message-ID: <1195052209.26046.20.camel@bugs.hal> On Mon, 2007-11-12 at 21:38 +0100, Dennis Oelkers wrote: > Hello rubyforge, hello rubycentral, > > For those of you who do not know me: I am the current "maintainer" of > the general mirroring architecture of rubyforge, running the master > mirror and the infrastructure to push out the changes of the gems and > rubyforge files repositories to all the subscribed mirrors. > I am currently living in Berlin, Germany, where I study computer > sciences (starting my diploma thesis in about a year), and working for > Nokia Germany. > > In our constant efforts to improve the quality of rubyforge, we had to > struggle with several crashes of the main (now retired) server hosting > the website and the project repositories. At that time, we noticed that > the webserver is a crucial component of the distribution chain, because > all clients have to contact the main server via http, and are > distributed to the different mirrors by following the redirect which is > then issued. > > Although the current hardware is running fine so far, we would like to > change this in the long run. The current idea is to transfer the load > balancing to dns, and maintain the dns entries using a modified version > of the script which is currently generating the apache rewrite map. > This is all fine, but has one big drawback: We are not able to do > centraslised accouting of the downloads, like we are with the current > solution. Gathering the logfiles from the different mirrors is a > solution which seems to be too painful to do at the moment. Abstaining > from the statistics we are able to generate from the centralised > logfiles we currently have, is not an option either. > > This is the main reason why I wrote this mail: what is your opinion > about the general proposal? Do you have a clever idea to solve the > logging problem? Is there anything else that you see as a > problem/improvable point of the current rubyforge infrastructure? Another option might be to ensure the gem index is being sync'd out to all the mirrors... that would let people do things like this when RubyForge is down: gem install ikko --source http://rubyforge.vm.bytemark.co.uk/ Hm, I just tried that and it worked... maybe we're doing this already? Of course, people would still have to know the proper URL for the gem server... Yours, Tom From dennis at lauschmusik.de Wed Nov 14 11:04:26 2007 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Wed, 14 Nov 2007 17:04:26 +0100 Subject: [Support-admins] Change proposal regarding the Rubyforge distribution infrastructure - your input is needed In-Reply-To: <1195052209.26046.20.camel@bugs.hal> References: <4738B9B1.5030702@lauschmusik.de> <1195052209.26046.20.camel@bugs.hal> Message-ID: <20071114160426.GE61767@valhalla.lauschmusik.de> On Wed, Nov 14, 2007 at 09:56:49AM -0500, Tom Copeland wrote: > gem install ikko --source http://rubyforge.vm.bytemark.co.uk/ > > Hm, I just tried that and it worked... maybe we're doing this already? > > Of course, people would still have to know the proper URL for the gem > server... Yes, this is already common practice among the rubygems users who experienced the downtimes of rubyforge. That is also the reason why there is always the url to a rubyforge mirror in the topic of #ruby-lang on freenode. Nevertheless in my opinion it is an obvious advantage if this would be handled transparently for the user, so even novice (or users who just do not use irc/mailing lists) are able to cope with downtimes of rubyforge (and I am not only talking about crashes / hardware / network problems, but we also have to deal with planned outages due to software updates or maintenance work done by the provider). Kind regards, Dennis Oelkers From tom at infoether.com Thu Nov 15 10:08:25 2007 From: tom at infoether.com (Tom Copeland) Date: Thu, 15 Nov 2007 10:08:25 -0500 Subject: [Support-admins] Change proposal regarding the Rubyforge distribution infrastructure - your input is needed In-Reply-To: <20071114160426.GE61767@valhalla.lauschmusik.de> References: <4738B9B1.5030702@lauschmusik.de> <1195052209.26046.20.camel@bugs.hal> <20071114160426.GE61767@valhalla.lauschmusik.de> Message-ID: <1195139305.13674.13.camel@bugs.hal> On Wed, 2007-11-14 at 17:04 +0100, Dennis Oelkers wrote: > On Wed, Nov 14, 2007 at 09:56:49AM -0500, Tom Copeland wrote: > > gem install ikko --source http://rubyforge.vm.bytemark.co.uk/ > > > > Hm, I just tried that and it worked... maybe we're doing this already? > > > > Of course, people would still have to know the proper URL for the gem > > server... > > Yes, this is already common practice among the rubygems users who experienced the downtimes of rubyforge. That is also the reason why there is always the url to a rubyforge mirror in the topic of #ruby-lang on freenode. Nevertheless in my opinion it is an obvious advantage if this would be handled transparently for the user, so even novice (or users who just do not use irc/mailing lists) are able to cope with downtimes of rubyforge (and I am not only talking about crashes / hardware / network problems, but we also have to deal with planned outages due to software updates or maintenance work done by the provider). Yup. So perhaps the RubyGems client would fetch the mirrors list from somewhere and then pick one? Maybe we'd have the mirrors list in a couple of places and hardcode that list into the client, something like that? Yours, Tom From tom at infoether.com Mon Nov 19 17:56:14 2007 From: tom at infoether.com (Tom Copeland) Date: Mon, 19 Nov 2007 17:56:14 -0500 Subject: [Support-admins] Change proposal regarding the Rubyforge distribution infrastructure - your input is needed In-Reply-To: References: <4738B9B1.5030702@lauschmusik.de> <1195052209.26046.20.camel@bugs.hal> <20071114160426.GE61767@valhalla.lauschmusik.de> <1195139305.13674.13.camel@bugs.hal> Message-ID: <1195512974.29742.4.camel@bugs.hal> On Thu, 2007-11-15 at 09:22 -0700, Chad Fowler wrote: > > Yup. So perhaps the RubyGems client would fetch the mirrors list from > > somewhere and then pick one? Maybe we'd have the mirrors list in a > > couple of places and hardcode that list into the client, something > > like > > that? > > > > I think this is a good idea. This is what the Sources gem was > originally meant to do. We should probably just do it with a simple > text file, though. Sounds like we should move this over to rubygems-developers. Maybe the client can just do a HEAD on that mirrors file to get updates... and it would only use the mirror list if the main server was down, is that right? That way the stats would still be mostly accurate as long as rubyforge.org was online most of the time... Yours, Tom From dennis at lauschmusik.de Fri Nov 23 11:23:10 2007 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Fri, 23 Nov 2007 17:23:10 +0100 Subject: [Support-admins] Setting up Ruby mirror In-Reply-To: <4746E625.4020003@xaop.com> References: <46B05688.4050309@xaop.com> <20070801122711.GA90098@valhalla.lauschmusik.de> <46B07DE4.1060006@xaop.com> <4746E625.4020003@xaop.com> Message-ID: <20071123162310.GB5171@valhalla.lauschmusik.de> Hi Peter, On Fri, Nov 23, 2007 at 03:39:33PM +0100, Peter Vanbroekhoven wrote: > >> @ERROR: access denied to gem-mirror from rubyforge.lauschmusik.de (80.237.222.133) > >> > >> I guess you used the wrong (even wrong in the docs) hostname? Please use rubyforge.lauschmusik.de in stead of rubyforge.rubyuser.de. > > > > Could you try again? Thanks. > > Bump? I just re-enabled your mirror in our local config, and during sync I get lots of: [23/Nov/2007:17:16:18 CET]: xaop rsync: mkstemp "/.zyps-0.6.2.gem.LxNaEL" (in gem-mirror) failed: Permission denied (13) and a final: [23/Nov/2007:17:16:18 CET]: xaop rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] Could you please check the permissions of the directories where the gems are located? Kind regards, Dennis Oelkers From dennis at lauschmusik.de Fri Nov 23 11:29:45 2007 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Fri, 23 Nov 2007 17:29:45 +0100 Subject: [Support-admins] Touching a file in the gems/main repositories on each sync Message-ID: <20071123162945.GC5171@valhalla.lauschmusik.de> Hey Tom, is it possible for you to touch a file in the gem/main repositories before you trigger a sync on the mirror master? This way I am able to fetch this file from each of the mirrors and check if they are up to date. I will replace the current functionality (just fetching the index of the repositories) in the mirrorwatch script with this, so we are not only checking if the mirrors are alive, but also if the syncing of them works properly. Kind regards, Dennis From tom at infoether.com Sat Nov 24 13:41:03 2007 From: tom at infoether.com (Tom Copeland) Date: Sat, 24 Nov 2007 13:41:03 -0500 Subject: [Support-admins] Touching a file in the gems/main repositories on each sync In-Reply-To: <20071123162945.GC5171@valhalla.lauschmusik.de> References: <20071123162945.GC5171@valhalla.lauschmusik.de> Message-ID: <1195929663.6294.14.camel@bugs.hal> On Fri, 2007-11-23 at 17:29 +0100, Dennis Oelkers wrote: > Hey Tom, > > is it possible for you to touch a file in the gem/main repositories before you trigger a sync on the mirror master? > This way I am able to fetch this file from each of the mirrors and check if they are up to date. I will replace the current functionality (just fetching the index of the repositories) in the mirrorwatch script with this, so we are not only checking if the mirrors are alive, but also if the syncing of them works properly. Sounds good! What do you want the file to be called? Yours, Tom From dennis at lauschmusik.de Mon Nov 26 11:11:47 2007 From: dennis at lauschmusik.de (Dennis Oelkers) Date: Mon, 26 Nov 2007 17:11:47 +0100 Subject: [Support-admins] Setting up Ruby mirror In-Reply-To: <474AEED0.2030009@xaop.com> References: <46B05688.4050309@xaop.com> <20070801122711.GA90098@valhalla.lauschmusik.de> <46B07DE4.1060006@xaop.com> <4746E625.4020003@xaop.com> <20071123162310.GB5171@valhalla.lauschmusik.de> <474AD9D0.6080700@xaop.com> <20071126144145.GA57120@valhalla.lauschmusik.de> <474AE176.5070108@xaop.com> <20071126153755.GC57120@valhalla.lauschmusik.de> <474AEED0.2030009@xaop.com> Message-ID: <20071126161147.GD57120@valhalla.lauschmusik.de> On Mon, Nov 26, 2007 at 05:05:36PM +0100, Peter Vanbroekhoven wrote: > Better now? Very much, I just added you to the rotation, so you should already get some traffic now. Thank you very much! Kind regards, Dennis Oelkers