From tom at infoether.com Fri Dec 1 11:24:05 2006 From: tom at infoether.com (Tom Copeland) Date: Fri, 1 Dec 2006 11:24:05 -0500 Subject: [Support-admins] Images served elsewhere Message-ID: <001d01c71565$1fa2ed80$0301000a@tomhplaptop> Hi all - I've modified Bruce Williams' theme so that all the images are served off the bytemark mirror; I set up a new vhost, copied over the images, etc. This reduces the RubyForge load by about 250 hits per minute, whew! Yours, Tom From tom at infoether.com Sat Dec 2 15:50:44 2006 From: tom at infoether.com (Tom Copeland) Date: Sat, 02 Dec 2006 15:50:44 -0500 Subject: [Support-admins] Host UP alert for rubyforge.org! In-Reply-To: <20061202203748.DEC303C03B8@infoether.com> References: <20061202203748.DEC303C03B8@infoether.com> Message-ID: <1165092644.23149.15.camel@bugs.hal> This outage - it lasted about 30 minutes - was due to scheduled network downtime by our hosting provider... all seems well now. Yours, Tom On Sat, 2006-12-02 at 14:37 -0600, nagios at infoether.com wrote: > ***** Nagios ***** > > Notification Type: RECOVERY > Host: rubyforge.org > State: UP > Address: 205.234.109.18 > Info: OK - HTTP/1.1 302 Found - 0.075 second response time > > Date/Time: Sat Dec 2 14:37:48 CST 2006 From tom at infoether.com Mon Dec 11 12:10:21 2006 From: tom at infoether.com (Tom Copeland) Date: Mon, 11 Dec 2006 12:10:21 -0500 Subject: [Support-admins] FW: ** PROBLEM alert - rubyforge.org/SMTP is CRITICAL ** Message-ID: <000f01c71d47$3dc96200$0301000a@tomhplaptop> Hi all - RubyForge is currently "under attack" in some sense; we're getting tons of SMTP connections, presumably from spammers. I've got Postfix capped at 100 processes, and all those are being used up by connections from random IP addresses. The emails themselves are getting caught by our HELO checks and greylisting, but it's still using up all the processes. I could bump Postfix up to 200 processes, but I think they'll just get filled up too. Any ideas? Thanks, Tom -----Original Message----- From: nagios at infoether.com [mailto:nagios at infoether.com] Sent: Monday, December 11, 2006 11:53 AM To: tom at infoether.com Subject: ** PROBLEM alert - rubyforge.org/SMTP is CRITICAL ** ***** Nagios ***** Notification Type: PROBLEM Service: SMTP Host: rubyforge.org Address: 205.234.109.18 State: CRITICAL Date/Time: Mon Dec 11 10:53:13 CST 2006 Additional Info: CRITICAL - Socket timeout after 10 seconds From tom at infoether.com Mon Dec 11 15:55:19 2006 From: tom at infoether.com (Tom Copeland) Date: Mon, 11 Dec 2006 15:55:19 -0500 Subject: [Support-admins] PostgreSQL 8.2 upgrade complete... Message-ID: <002e01c71d66$abb712c0$0301000a@tomhplaptop> ...seems like all's well, but if anyone notices anything awry, please let me know... Thanks, Tom From tom at infoether.com Thu Dec 14 14:39:17 2006 From: tom at infoether.com (Tom Copeland) Date: Thu, 14 Dec 2006 14:39:17 -0500 Subject: [Support-admins] FW: RubyForge offline Message-ID: <002701c71fb7$8b6d70b0$0301000a@tomhplaptop> Everything came back up after a reboot. I'm not sure what went wrong; there's nothing very interesting in the logs except some network card errors about an hour before it went down. And it seems to have recovered from those just fine; the last message in the log before the shutdown is: Dec 14 12:56:04 rubyforge kernel: NETDEV WATCHDOG: eth0: transmit timed out Dec 14 12:56:05 rubyforge kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex The load today wasn't very high... I don't know. Yours, tom -----Original Message----- From: Tom Copeland [mailto:tom at infoether.com] Sent: Thursday, December 14, 2006 2:21 PM To: ruby-talk ML Subject: RubyForge offline Hi all - RubyForge is offline; we're investigating it. Mirror information is here: http://tomcopeland.blogs.com/juniordeveloper/2006/10/rubyforge_down.html Thanks for your patience, Tom From tom at infoether.com Fri Dec 15 15:15:33 2006 From: tom at infoether.com (Tom Copeland) Date: Fri, 15 Dec 2006 15:15:33 -0500 Subject: [Support-admins] FW: RubyForge offline Message-ID: <002901c72085$c6d8baa0$0301000a@tomhplaptop> FYI, this outage appears to have been caused by a lot of bots hitting the ViewVC pages and triggering lots of complicated diff operations. This drove the load average up to ridiculous levels, which caused the machine to go offline, etc. I've disabled ViewVC until I can get a good solution to this problem; hopefully we can figure out a way to disable only the more expensive operations. We shall see... Yours, Tom -----Original Message----- From: Tom Copeland [mailto:tom at infoether.com] Sent: Thursday, December 14, 2006 2:39 PM To: 'support-admins at rubyforge.org' Subject: FW: RubyForge offline Everything came back up after a reboot. I'm not sure what went wrong; there's nothing very interesting in the logs except some network card errors about an hour before it went down. And it seems to have recovered from those just fine; the last message in the log before the shutdown is: Dec 14 12:56:04 rubyforge kernel: NETDEV WATCHDOG: eth0: transmit timed out Dec 14 12:56:05 rubyforge kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex The load today wasn't very high... I don't know. Yours, tom -----Original Message----- From: Tom Copeland [mailto:tom at infoether.com] Sent: Thursday, December 14, 2006 2:21 PM To: ruby-talk ML Subject: RubyForge offline Hi all - RubyForge is offline; we're investigating it. Mirror information is here: http://tomcopeland.blogs.com/juniordeveloper/2006/10/rubyforge_down.html Thanks for your patience, Tom From tom at infoether.com Wed Dec 20 09:28:50 2006 From: tom at infoether.com (Tom Copeland) Date: Wed, 20 Dec 2006 09:28:50 -0500 Subject: [Support-admins] RubyForge offline Message-ID: <000001c72443$2b142de0$0301000a@tomhplaptop> > FYI, this outage appears to have been caused by a lot of bots > hitting the ViewVC pages and triggering lots of complicated > diff operations. This drove the load average up to > ridiculous levels, which caused the machine to go offline, > etc. I've disabled ViewVC until I can get a good solution to > this problem; hopefully we can figure out a way to disable > only the more expensive operations. We shall see... I've "fixed" this by commenting out various bits of ViewVC. This should let folks browse their source code while preventing evil diff commands. Half a loaf is better than none... Yours, Tom