From eden at hulu.com  Thu May  1 01:33:34 2008
From: eden at hulu.com (Eden Li)
Date: Thu, 1 May 2008 13:33:34 +0800
Subject: [Mongrel] Mongrel hangs in production. gdb stack included
In-Reply-To: <355679A4-A176-4BB7-BECA-8B06F077E670@gmail.com>
References: <5E151013-6346-40E5-B1FB-588F4C6EB014@gmail.com>
	<94A91EF2-5A28-4674-9975-4B3A204E7103@cheney.net>
	<197640DD-71D3-466E-A2A8-80BE8BE7673D@gmail.com>
	<020A3A62-778C-439E-8B24-5BA1AE2EB8E9@gmail.com>
	<355679A4-A176-4BB7-BECA-8B06F077E670@gmail.com>
Message-ID: <1dd361e10804302233h77ac970bta6c183e3e24f51bd@mail.gmail.com>

What's your global mysql interactive_timeout set to?  This is the
timeout after which mysql will close any inactive connections.  The
default is 28800 (8 hours).  Try setting it to 24 hours or 1 week.
It's probable that your app is timed out waiting on a closed mysql
connection.

On Thu, May 1, 2008 at 3:09 AM, Erik Morton <eimorton at gmail.com> wrote:
> How can be 100% certain beyond having the gem installed and having the stack
> from gdb include:
>
>
> >
> > #8  0x04cc9881 in mysql_store_result (mysql=0x913eb10) at client.c:2697
> >
> > #8  0x01d6a881 in mysql_store_result (mysql=0x93e71a8) at client.c:2697
> >
>
>  Thanks
>
>  Erik
>
>
>
>  On Apr 30, 2008, at 2:52 PM, Ezra Zygmuntowicz wrote:
>
> >
> >        Also are you 100% certain you are using the native C mysql-ruby
> bindings and not the built into rails pure ruby ones?
> >
> > -Ezra
> >
> >
> >
> > On Apr 30, 2008, at 6:28 AM, Erik Morton wrote:
> >
> > > It happens at night usually so I haven't had a chance to run show innodb
> status yet. I will say that once when it happened show processlist still
> showed all of the expected connections.
> > >
> > > I have no large hooks. Just a call to MyObject.find(:all, :include =>
> :parent) that generates an SQL statement approximately 2,000-3,000
> characters long. My max_packet is set to 16mb
> > >
> > > Erik
> > > On Apr 30, 2008, at 8:50 AM, Dave Cheney wrote:
> > >
> > > > What does
> > > > mysql> show innodb status\G
> > > > mysql> show processlist;
> > > >
> > > > show during the hang, it may be a deadlock in the db. We had similar
> problems with large after_save hooks that would deadlock updating multiple
> tables.
> > > >
> > > > Cheers
> > > >
> > > > Dave
> > > >
> > > > On 30/04/2008, at 12:31 PM, Erik Morton wrote:
> > > >
> > > >
> > > > > I am getting hanging Mongrels daily under light volume.
> > > > >
> > > > > Mongrel Web Server 1.1.4
> > > > > Apache 2.2 w/mod_proxy
> > > > > RedHat EL4 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686
> i686 i386 GNU/Linux
> > > > > ruby 1.8.5 (2006-08-25) [i686-linux]
> > > > > fastthread (1.0.1)
> > > > > Mysql 5.0.45
> > > > >
> > > > > The mongrels are hanging with 0% CPU, their database connections are
> still being reported as open by mysql. Attaching gdb to the processes yields
> similar results (included below). I have no idea about what the next steps
> are in figuring out what is going on -- short of upgrading to ruby 1.8.6!
> > > > >
> > > > > Help and pointers are much appreciated.
> > > > >
> > > > > (gdb) whe
> > > > > #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> > > > > #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
> > > > > #2  0x04ccc27f in vio_read (vio=0xfffffe00, buf=0xfffffe00 <Address
> 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
> > > > > #3  0x04ccc356 in vio_read_buff (vio=0x9088ae0,
> > > > > buf=0xb5d471df "e Travel Plaza, located on I-90 eastbound between
> Interchanges 48A (Pembroke) and 48 (Batavia), at milepost 397. The
> renovation projects scheduled at Ulster and Pembroke travel plazas, in
> addition to "..., size=4412) at viosocket.c:95
> > > > > #4  0x04ccd8f6 in my_real_read (net=0x913eb10, complen=0xbfed5e98)
> at net.c:804
> > > > > #5  0x04ccdc30 in my_net_read (net=0x913eb10) at net.c:978
> > > > > #6  0x04cc6b78 in net_safe_read (mysql=0x913eb10) at client.c:596
> > > > > #7  0x04cc792f in cli_read_rows (mysql=0x913eb10,
> mysql_fields=0xad6e1a0, fields=59) at client.c:1343
> > > > > #8  0x04cc9881 in mysql_store_result (mysql=0x913eb10) at
> client.c:2697
> > > > > #9  0x04ca299f in store_result (obj=3076071240) at mysql.c:677
> > > > > #10 0x0805bcb4 in rb_call0 (klass=3076091220, recv=3076071240,
> id=27825, oid=4294966784, argc=1, argv=0xbfed6160, body=0xb7596458, flags=0)
> at eval.c:5665
> > > > > #11 0x0805c017 in rb_call (klass=3076091220, recv=3076071240,
> mid=27825, argc=1, argv=0xbfed6160, scope=0) at eval.c:6048
> > > > > #12 0x08058127 in rb_eval (self=3076059900, n=0x113c) at ruby.h:654
> > > > > #13 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
> flags=0, avalue=0) at eval.c:4987
> > > > > #14 0x08058475 in rb_eval (self=3076059900, n=0x113c) at eval.c:3248
> > > > > #15 0x08058952 in rb_eval (self=3076059900, n=0x113c) at eval.c:3624
> > > > > #16 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
> flags=0, avalue=0) at eval.c:4987
> > > > > #17 0x08058475 in rb_eval (self=3082341360, n=0x113c) at eval.c:3248
> > > > > #18 0x0805b9ce in rb_call0 (klass=3082341260, recv=3082341360,
> id=53385, oid=4294966784, argc=0, argv=0x0, body=0xb7b931f8, flags=0) at
> eval.c:5954
> > > > > #19 0x0805c017 in rb_call (klass=3082341260, recv=3082341360,
> mid=53385, argc=0, argv=0x0, scope=0) at eval.c:6048
> > > > > #20 0x08058127 in rb_eval (self=3082341360, n=0x113c) at ruby.h:654
> > > > >
> > > > >
> > > > > (gdb) whe
> > > > > #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> > > > > #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
> > > > > #2  0x01d6d27f in vio_read (vio=0xfffffe00, buf=0xfffffe00 <Address
> 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
> > > > > #3  0x01d6d356 in vio_read_buff (vio=0x93e7c30,
> > > > > buf=0xb6f00db4
> "~\202\215^-?\213\201?QJ\237\203\201\230D?\232V\201M8\233\224?\201\021-o\217[\200?#?\211\236\200?\203??0\213\021wC?h\211\236j\234?\215\210F]?\232\207\006Q\b\236\220\205?Dv\231t\204?8\220\224
> \203?-\214\216\236\202?$\026\210?\201?\203Y?u\221?v???\217?jS??\215?]\206??\213?P?\235?\212\023DS\230?\210 at 8\210\223d\206o-?\215?\204?$R\210z\202?\203\017??\230\236v??\026\226",
> size=503704)
> > > > > at viosocket.c:95
> > > > > #4  0x01d6e8f6 in my_real_read (net=0x93e71a8, complen=0xbff9efa8)
> at net.c:804
> > > > > #5  0x01d6ec30 in my_net_read (net=0x93e71a8) at net.c:978
> > > > > #6  0x01d67b78 in net_safe_read (mysql=0x93e71a8) at client.c:596
> > > > > #7  0x01d6892f in cli_read_rows (mysql=0x93e71a8,
> mysql_fields=0xaec8f00, fields=51) at client.c:1343
> > > > > #8  0x01d6a881 in mysql_store_result (mysql=0x93e71a8) at
> client.c:2697
> > > > > #9  0x01d4399f in store_result (obj=3071130920) at mysql.c:677
> > > > > #10 0x0805bcb4 in rb_call0 (klass=3071150400, recv=3071130920,
> id=27825, oid=4294966784, argc=1, argv=0xbff9f270, body=0xb70e0058, flags=0)
> at eval.c:5665
> > > > > #11 0x0805c017 in rb_call (klass=3071150400, recv=3071130920,
> mid=27825, argc=1, argv=0xbff9f270, scope=0) at eval.c:6048
> > > > > #12 0x08058127 in rb_eval (self=3071126200, n=0x7af98) at ruby.h:654
> > > > >
> > > > > _______________________________________________
> > > > > Mongrel-users mailing list
> > > > > Mongrel-users at rubyforge.org
> > > > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > > > >
> > > >
> > > > _______________________________________________
> > > > Mongrel-users mailing list
> > > > Mongrel-users at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > > >
> > >
> > > _______________________________________________
> > > Mongrel-users mailing list
> > > Mongrel-users at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > >
> >
> > - Ezra Zygmuntowicz
> > -- Founder & Software Architect
> > -- ezra at engineyard.com
> > -- EngineYard.com
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users

From eimorton at gmail.com  Thu May  1 09:18:24 2008
From: eimorton at gmail.com (Erik Morton)
Date: Thu, 1 May 2008 09:18:24 -0400
Subject: [Mongrel] Mongrel hangs in production. gdb stack included
In-Reply-To: <1dd361e10804301844y7c4de0cak7644a6213b7c98b3@mail.gmail.com>
References: <5E151013-6346-40E5-B1FB-588F4C6EB014@gmail.com>
	<94A91EF2-5A28-4674-9975-4B3A204E7103@cheney.net>
	<197640DD-71D3-466E-A2A8-80BE8BE7673D@gmail.com>
	<020A3A62-778C-439E-8B24-5BA1AE2EB8E9@gmail.com>
	<1dd361e10804301844y7c4de0cak7644a6213b7c98b3@mail.gmail.com>
Message-ID: <C753C6A4-D20C-42A7-909E-A7027AF3ACE7@gmail.com>

Thanks for your response. I agree that it is not a Mongrel issue. All  
of our tables are innodb. The site is an "enterprisesy" site so it  
doesn't get a lot of traffic is real terms. However there are RSS  
feeds that pull from it all day and night and Google gives it a lot of  
love, so the connections are never really at risk of timing out. When  
the issue has occurred a show processlist will list all of the  
connections that I expect to have.

Thanks for the tips on isolation level queries. The initial exception  
that leads to the hanging Mongrel occurs on a method that generates a  
custom RSS feed. It mostly happens at night, but I'm not sure that  
there is any relation. Anyway, I have a band aid in place with a  
brutally mean Monit configuration and I will continue to troubleshoot.  
Thanks!

Erik
On Apr 30, 2008, at 9:44 PM, Eden Li wrote:
> This is not really related to mongrel anymore, but...
>
> The stack trace points to the C-version of the mysql-ruby library
> (mysql.c -> store_result).  it looks like it's hanging in the libmysql
> C-library provided (or compiled) with your mysql client installation
> in mysql_store_result.  This points to either a badly compiled
> libmysql (unlikely) or some other issue with how your database is
> configured.
>
> As mentioned, use `show engine innodb status\G` -- it'll show you any
> previous deadlocks even after they've occurred.  if you've got one in
> your app, you'll see it right away.  This only applies if your tables
> are InnoDB (mysql defaults to MyISAM).
>
> Also, if you're doing MyObject.find(:all, ...) without conditions and
> a huge include, and you're using MyISAM tables, you'll be triggering a
> huge table locks (one for my_objects and one for parents) every time
> you run that query which will stop up any other reads or writes
> causing a slow down in your app and potentially causing deadlocks.
>
> First, you should try to refactor your code so it doesn't need to pull
> in a huge dataset potentially causing lots of locks.  If you're
> crunched and you really need to do this, try switching to InnoDB
> should help if you're not there already.  Also:
>
> - make sure that the :parent association has an index set on the  
> join column
> - do the read on a replicated slave
> - and/or, run "SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED"
> before the query (set it back to "REPEATABLE READ" after you're done).
> this will prevent your select from locking tables when reading, but
> will potentially give you stale data.  i think it only works with
> innodb, but i'm not sure.
>
>
> On Thu, May 1, 2008 at 2:52 AM, Ezra Zygmuntowicz  
> <ezmobius at gmail.com> wrote:
>>
>>        Also are you 100% certain you are using the native C mysql- 
>> ruby
>> bindings and not the built into rails pure ruby ones?
>>
>> -Ezra
>>
>>
>>
>>
>>
>> On Apr 30, 2008, at 6:28 AM, Erik Morton wrote:
>>
>>> It happens at night usually so I haven't had a chance to run show  
>>> innodb
>> status yet. I will say that once when it happened show processlist  
>> still
>> showed all of the expected connections.
>>>
>>> I have no large hooks. Just a call to MyObject.find(:all, :include  
>>> =>
>> :parent) that generates an SQL statement approximately 2,000-3,000
>> characters long. My max_packet is set to 16mb
>>>
>>> Erik
>>> On Apr 30, 2008, at 8:50 AM, Dave Cheney wrote:
>>>
>>>> What does
>>>> mysql> show innodb status\G
>>>> mysql> show processlist;
>>>>
>>>> show during the hang, it may be a deadlock in the db. We had  
>>>> similar
>> problems with large after_save hooks that would deadlock updating  
>> multiple
>> tables.
>>>>
>>>> Cheers
>>>>
>>>> Dave
>>>>
>>>> On 30/04/2008, at 12:31 PM, Erik Morton wrote:
>>>>
>>>>
>>>>> I am getting hanging Mongrels daily under light volume.
>>>>>
>>>>> Mongrel Web Server 1.1.4
>>>>> Apache 2.2 w/mod_proxy
>>>>> RedHat EL4 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686  
>>>>> i686
>> i386 GNU/Linux
>>>>> ruby 1.8.5 (2006-08-25) [i686-linux]
>>>>> fastthread (1.0.1)
>>>>> Mysql 5.0.45
>>>>>
>>>>> The mongrels are hanging with 0% CPU, their database connections  
>>>>> are
>> still being reported as open by mysql. Attaching gdb to the  
>> processes yields
>> similar results (included below). I have no idea about what the  
>> next steps
>> are in figuring out what is going on -- short of upgrading to ruby  
>> 1.8.6!
>>>>>
>>>>> Help and pointers are much appreciated.
>>>>>
>>>>> (gdb) whe
>>>>> #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
>>>>> #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
>>>>> #2  0x04ccc27f in vio_read (vio=0xfffffe00, buf=0xfffffe00  
>>>>> <Address
>> 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
>>>>> #3  0x04ccc356 in vio_read_buff (vio=0x9088ae0,
>>>>> buf=0xb5d471df "e Travel Plaza, located on I-90 eastbound between
>> Interchanges 48A (Pembroke) and 48 (Batavia), at milepost 397. The
>> renovation projects scheduled at Ulster and Pembroke travel plazas,  
>> in
>> addition to "..., size=4412) at viosocket.c:95
>>>>> #4  0x04ccd8f6 in my_real_read (net=0x913eb10,  
>>>>> complen=0xbfed5e98) at
>> net.c:804
>>>>> #5  0x04ccdc30 in my_net_read (net=0x913eb10) at net.c:978
>>>>> #6  0x04cc6b78 in net_safe_read (mysql=0x913eb10) at client.c:596
>>>>> #7  0x04cc792f in cli_read_rows (mysql=0x913eb10,
>> mysql_fields=0xad6e1a0, fields=59) at client.c:1343
>>>>> #8  0x04cc9881 in mysql_store_result (mysql=0x913eb10) at
>> client.c:2697
>>>>> #9  0x04ca299f in store_result (obj=3076071240) at mysql.c:677
>>>>> #10 0x0805bcb4 in rb_call0 (klass=3076091220, recv=3076071240,
>> id=27825, oid=4294966784, argc=1, argv=0xbfed6160, body=0xb7596458,  
>> flags=0)
>> at eval.c:5665
>>>>> #11 0x0805c017 in rb_call (klass=3076091220, recv=3076071240,
>> mid=27825, argc=1, argv=0xbfed6160, scope=0) at eval.c:6048
>>>>> #12 0x08058127 in rb_eval (self=3076059900, n=0x113c) at ruby.h: 
>>>>> 654
>>>>> #13 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
>> flags=0, avalue=0) at eval.c:4987
>>>>> #14 0x08058475 in rb_eval (self=3076059900, n=0x113c) at eval.c: 
>>>>> 3248
>>>>> #15 0x08058952 in rb_eval (self=3076059900, n=0x113c) at eval.c: 
>>>>> 3624
>>>>> #16 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
>> flags=0, avalue=0) at eval.c:4987
>>>>> #17 0x08058475 in rb_eval (self=3082341360, n=0x113c) at eval.c: 
>>>>> 3248
>>>>> #18 0x0805b9ce in rb_call0 (klass=3082341260, recv=3082341360,
>> id=53385, oid=4294966784, argc=0, argv=0x0, body=0xb7b931f8,  
>> flags=0) at
>> eval.c:5954
>>>>> #19 0x0805c017 in rb_call (klass=3082341260, recv=3082341360,
>> mid=53385, argc=0, argv=0x0, scope=0) at eval.c:6048
>>>>> #20 0x08058127 in rb_eval (self=3082341360, n=0x113c) at ruby.h: 
>>>>> 654
>>>>>
>>>>>
>>>>> (gdb) whe
>>>>> #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
>>>>> #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
>>>>> #2  0x01d6d27f in vio_read (vio=0xfffffe00, buf=0xfffffe00  
>>>>> <Address
>> 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
>>>>> #3  0x01d6d356 in vio_read_buff (vio=0x93e7c30,
>>>>> buf=0xb6f00db4
>> "~\202\215^-?\213\201?QJ\237\203\201\230D?\232V\201M8\233\224? 
>> \201\021-o\217[\200?#?\211\236\200?\203??0\213\021wC?h\211\236j\234? 
>> \215\210F]?\232\207\006Q\b\236\220\205?Dv\231t\204?8\220\224
>> \203?-\214\216\236\202?$\026\210?\201?\203Y?u\221?v???\217?jS?? 
>> \215?]\206??\213?P?\235?\212\023DS\230?\210 at 8\210\223d\206o-?\215? 
>> \204?$R\210z\202?\203\017??\230\236v??\026\226",
>> size=503704)
>>>>> at viosocket.c:95
>>>>> #4  0x01d6e8f6 in my_real_read (net=0x93e71a8,  
>>>>> complen=0xbff9efa8) at
>> net.c:804
>>>>> #5  0x01d6ec30 in my_net_read (net=0x93e71a8) at net.c:978
>>>>> #6  0x01d67b78 in net_safe_read (mysql=0x93e71a8) at client.c:596
>>>>> #7  0x01d6892f in cli_read_rows (mysql=0x93e71a8,
>> mysql_fields=0xaec8f00, fields=51) at client.c:1343
>>>>> #8  0x01d6a881 in mysql_store_result (mysql=0x93e71a8) at
>> client.c:2697
>>>>> #9  0x01d4399f in store_result (obj=3071130920) at mysql.c:677
>>>>> #10 0x0805bcb4 in rb_call0 (klass=3071150400, recv=3071130920,
>> id=27825, oid=4294966784, argc=1, argv=0xbff9f270, body=0xb70e0058,  
>> flags=0)
>> at eval.c:5665
>>>>> #11 0x0805c017 in rb_call (klass=3071150400, recv=3071130920,
>> mid=27825, argc=1, argv=0xbff9f270, scope=0) at eval.c:6048
>>>>> #12 0x08058127 in rb_eval (self=3071126200, n=0x7af98) at ruby.h: 
>>>>> 654
>>>>>
>>>>> _______________________________________________
>>>>> Mongrel-users mailing list
>>>>> Mongrel-users at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mongrel-users mailing list
>>>> Mongrel-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>>
>>>
>>> _______________________________________________
>>> Mongrel-users mailing list
>>> Mongrel-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>
>>
>> - Ezra Zygmuntowicz
>> -- Founder & Software Architect
>> -- ezra at engineyard.com
>> -- EngineYard.com
>>
>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users


From jgeiger at gmail.com  Thu May  1 11:32:56 2008
From: jgeiger at gmail.com (Joey Geiger)
Date: Thu, 1 May 2008 10:32:56 -0500
Subject: [Mongrel] Mongrel hangs in production. gdb stack included
In-Reply-To: <C753C6A4-D20C-42A7-909E-A7027AF3ACE7@gmail.com>
References: <5E151013-6346-40E5-B1FB-588F4C6EB014@gmail.com>
	<94A91EF2-5A28-4674-9975-4B3A204E7103@cheney.net>
	<197640DD-71D3-466E-A2A8-80BE8BE7673D@gmail.com>
	<020A3A62-778C-439E-8B24-5BA1AE2EB8E9@gmail.com>
	<1dd361e10804301844y7c4de0cak7644a6213b7c98b3@mail.gmail.com>
	<C753C6A4-D20C-42A7-909E-A7027AF3ACE7@gmail.com>
Message-ID: <466af3440805010832r3f1863dge4daec9be663ef21@mail.gmail.com>

Actually you mention RSS feeds, and that reminded me of something.
I've noticed that my mongrels die at night sometimes as well, and it's
during something similar to feed creation, but it's not the app that's
causing the issue.

We're doing a mysql db optimize around that time, and it ends up
locking a bunch of the tables, which in turn causes the mongrels to
hang and eventually monit restarts them.

So if it happens mainly at night, maybe there is some maintenance
going on at the same time.

On Thu, May 1, 2008 at 8:18 AM, Erik Morton <eimorton at gmail.com> wrote:
> Thanks for your response. I agree that it is not a Mongrel issue. All of our
> tables are innodb. The site is an "enterprisesy" site so it doesn't get a
> lot of traffic is real terms. However there are RSS feeds that pull from it
> all day and night and Google gives it a lot of love, so the connections are
> never really at risk of timing out. When the issue has occurred a show
> processlist will list all of the connections that I expect to have.
>
>  Thanks for the tips on isolation level queries. The initial exception that
> leads to the hanging Mongrel occurs on a method that generates a custom RSS
> feed. It mostly happens at night, but I'm not sure that there is any
> relation. Anyway, I have a band aid in place with a brutally mean Monit
> configuration and I will continue to troubleshoot. Thanks!
>
>  Erik
>
>
>  On Apr 30, 2008, at 9:44 PM, Eden Li wrote:
>
> > This is not really related to mongrel anymore, but...
> >
> > The stack trace points to the C-version of the mysql-ruby library
> > (mysql.c -> store_result).  it looks like it's hanging in the libmysql
> > C-library provided (or compiled) with your mysql client installation
> > in mysql_store_result.  This points to either a badly compiled
> > libmysql (unlikely) or some other issue with how your database is
> > configured.
> >
> > As mentioned, use `show engine innodb status\G` -- it'll show you any
> > previous deadlocks even after they've occurred.  if you've got one in
> > your app, you'll see it right away.  This only applies if your tables
> > are InnoDB (mysql defaults to MyISAM).
> >
> > Also, if you're doing MyObject.find(:all, ...) without conditions and
> > a huge include, and you're using MyISAM tables, you'll be triggering a
> > huge table locks (one for my_objects and one for parents) every time
> > you run that query which will stop up any other reads or writes
> > causing a slow down in your app and potentially causing deadlocks.
> >
> > First, you should try to refactor your code so it doesn't need to pull
> > in a huge dataset potentially causing lots of locks.  If you're
> > crunched and you really need to do this, try switching to InnoDB
> > should help if you're not there already.  Also:
> >
> > - make sure that the :parent association has an index set on the join
> column
> > - do the read on a replicated slave
> > - and/or, run "SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED"
> > before the query (set it back to "REPEATABLE READ" after you're done).
> > this will prevent your select from locking tables when reading, but
> > will potentially give you stale data.  i think it only works with
> > innodb, but i'm not sure.
> >
> >
> > On Thu, May 1, 2008 at 2:52 AM, Ezra Zygmuntowicz <ezmobius at gmail.com>
> wrote:
> >
> > >
> > >       Also are you 100% certain you are using the native C mysql-ruby
> > > bindings and not the built into rails pure ruby ones?
> > >
> > > -Ezra
> > >
> > >
> > >
> > >
> > >
> > > On Apr 30, 2008, at 6:28 AM, Erik Morton wrote:
> > >
> > >
> > > > It happens at night usually so I haven't had a chance to run show
> innodb
> > > >
> > > status yet. I will say that once when it happened show processlist still
> > > showed all of the expected connections.
> > >
> > > >
> > > > I have no large hooks. Just a call to MyObject.find(:all, :include =>
> > > >
> > > :parent) that generates an SQL statement approximately 2,000-3,000
> > > characters long. My max_packet is set to 16mb
> > >
> > > >
> > > > Erik
> > > > On Apr 30, 2008, at 8:50 AM, Dave Cheney wrote:
> > > >
> > > >
> > > > > What does
> > > > > mysql> show innodb status\G
> > > > > mysql> show processlist;
> > > > >
> > > > > show during the hang, it may be a deadlock in the db. We had similar
> > > > >
> > > >
> > > problems with large after_save hooks that would deadlock updating
> multiple
> > > tables.
> > >
> > > >
> > > > >
> > > > > Cheers
> > > > >
> > > > > Dave
> > > > >
> > > > > On 30/04/2008, at 12:31 PM, Erik Morton wrote:
> > > > >
> > > > >
> > > > >
> > > > > > I am getting hanging Mongrels daily under light volume.
> > > > > >
> > > > > > Mongrel Web Server 1.1.4
> > > > > > Apache 2.2 w/mod_proxy
> > > > > > RedHat EL4 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686
> i686
> > > > > >
> > > > >
> > > >
> > > i386 GNU/Linux
> > >
> > > >
> > > > >
> > > > > > ruby 1.8.5 (2006-08-25) [i686-linux]
> > > > > > fastthread (1.0.1)
> > > > > > Mysql 5.0.45
> > > > > >
> > > > > > The mongrels are hanging with 0% CPU, their database connections
> are
> > > > > >
> > > > >
> > > >
> > > still being reported as open by mysql. Attaching gdb to the processes
> yields
> > > similar results (included below). I have no idea about what the next
> steps
> > > are in figuring out what is going on -- short of upgrading to ruby
> 1.8.6!
> > >
> > > >
> > > > >
> > > > > >
> > > > > > Help and pointers are much appreciated.
> > > > > >
> > > > > > (gdb) whe
> > > > > > #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> > > > > > #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
> > > > > > #2  0x04ccc27f in vio_read (vio=0xfffffe00, buf=0xfffffe00
> <Address
> > > > > >
> > > > >
> > > >
> > > 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
> > >
> > > >
> > > > >
> > > > > > #3  0x04ccc356 in vio_read_buff (vio=0x9088ae0,
> > > > > > buf=0xb5d471df "e Travel Plaza, located on I-90 eastbound between
> > > > > >
> > > > >
> > > >
> > > Interchanges 48A (Pembroke) and 48 (Batavia), at milepost 397. The
> > > renovation projects scheduled at Ulster and Pembroke travel plazas, in
> > > addition to "..., size=4412) at viosocket.c:95
> > >
> > > >
> > > > >
> > > > > > #4  0x04ccd8f6 in my_real_read (net=0x913eb10, complen=0xbfed5e98)
> at
> > > > > >
> > > > >
> > > >
> > > net.c:804
> > >
> > > >
> > > > >
> > > > > > #5  0x04ccdc30 in my_net_read (net=0x913eb10) at net.c:978
> > > > > > #6  0x04cc6b78 in net_safe_read (mysql=0x913eb10) at client.c:596
> > > > > > #7  0x04cc792f in cli_read_rows (mysql=0x913eb10,
> > > > > >
> > > > >
> > > >
> > > mysql_fields=0xad6e1a0, fields=59) at client.c:1343
> > >
> > > >
> > > > >
> > > > > > #8  0x04cc9881 in mysql_store_result (mysql=0x913eb10) at
> > > > > >
> > > > >
> > > >
> > > client.c:2697
> > >
> > > >
> > > > >
> > > > > > #9  0x04ca299f in store_result (obj=3076071240) at mysql.c:677
> > > > > > #10 0x0805bcb4 in rb_call0 (klass=3076091220, recv=3076071240,
> > > > > >
> > > > >
> > > >
> > > id=27825, oid=4294966784, argc=1, argv=0xbfed6160, body=0xb7596458,
> flags=0)
> > > at eval.c:5665
> > >
> > > >
> > > > >
> > > > > > #11 0x0805c017 in rb_call (klass=3076091220, recv=3076071240,
> > > > > >
> > > > >
> > > >
> > > mid=27825, argc=1, argv=0xbfed6160, scope=0) at eval.c:6048
> > >
> > > >
> > > > >
> > > > > > #12 0x08058127 in rb_eval (self=3076059900, n=0x113c) at
> ruby.h:654
> > > > > > #13 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
> > > > > >
> > > > >
> > > >
> > > flags=0, avalue=0) at eval.c:4987
> > >
> > > >
> > > > >
> > > > > > #14 0x08058475 in rb_eval (self=3076059900, n=0x113c) at
> eval.c:3248
> > > > > > #15 0x08058952 in rb_eval (self=3076059900, n=0x113c) at
> eval.c:3624
> > > > > > #16 0x0805d99b in rb_yield_0 (val=6, self=3076059900, klass=0,
> > > > > >
> > > > >
> > > >
> > > flags=0, avalue=0) at eval.c:4987
> > >
> > > >
> > > > >
> > > > > > #17 0x08058475 in rb_eval (self=3082341360, n=0x113c) at
> eval.c:3248
> > > > > > #18 0x0805b9ce in rb_call0 (klass=3082341260, recv=3082341360,
> > > > > >
> > > > >
> > > >
> > > id=53385, oid=4294966784, argc=0, argv=0x0, body=0xb7b931f8, flags=0) at
> > > eval.c:5954
> > >
> > > >
> > > > >
> > > > > > #19 0x0805c017 in rb_call (klass=3082341260, recv=3082341360,
> > > > > >
> > > > >
> > > >
> > > mid=53385, argc=0, argv=0x0, scope=0) at eval.c:6048
> > >
> > > >
> > > > >
> > > > > > #20 0x08058127 in rb_eval (self=3082341360, n=0x113c) at
> ruby.h:654
> > > > > >
> > > > > >
> > > > > > (gdb) whe
> > > > > > #0  0x0058b7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> > > > > > #1  0x0065ec93 in __read_nocancel () from /lib/tls/libc.so.6
> > > > > > #2  0x01d6d27f in vio_read (vio=0xfffffe00, buf=0xfffffe00
> <Address
> > > > > >
> > > > >
> > > >
> > > 0xfffffe00 out of bounds>, size=-512) at viosocket.c:44
> > >
> > > >
> > > > >
> > > > > > #3  0x01d6d356 in vio_read_buff (vio=0x93e7c30,
> > > > > > buf=0xb6f00db4
> > > > > >
> > > > >
> > > >
> > >
> "~\202\215^-?\213\201?QJ\237\203\201\230D?\232V\201M8\233\224?\201\021-o\217[\200?#?\211\236\200?\203??0\213\021wC?h\211\236j\234?\215\210F]?\232\207\006Q\b\236\220\205?Dv\231t\204?8\220\224
> > >
> \203?-\214\216\236\202?$\026\210?\201?\203Y?u\221?v???\217?jS??\215?]\206??\213?P?\235?\212\023DS\230?\210 at 8\210\223d\206o-?\215?\204?$R\210z\202?\203\017??\230\236v??\026\226",
> > > size=503704)
> > >
> > > >
> > > > >
> > > > > > at viosocket.c:95
> > > > > > #4  0x01d6e8f6 in my_real_read (net=0x93e71a8, complen=0xbff9efa8)
> at
> > > > > >
> > > > >
> > > >
> > > net.c:804
> > >
> > > >
> > > > >
> > > > > > #5  0x01d6ec30 in my_net_read (net=0x93e71a8) at net.c:978
> > > > > > #6  0x01d67b78 in net_safe_read (mysql=0x93e71a8) at client.c:596
> > > > > > #7  0x01d6892f in cli_read_rows (mysql=0x93e71a8,
> > > > > >
> > > > >
> > > >
> > > mysql_fields=0xaec8f00, fields=51) at client.c:1343
> > >
> > > >
> > > > >
> > > > > > #8  0x01d6a881 in mysql_store_result (mysql=0x93e71a8) at
> > > > > >
> > > > >
> > > >
> > > client.c:2697
> > >
> > > >
> > > > >
> > > > > > #9  0x01d4399f in store_result (obj=3071130920) at mysql.c:677
> > > > > > #10 0x0805bcb4 in rb_call0 (klass=3071150400, recv=3071130920,
> > > > > >
> > > > >
> > > >
> > > id=27825, oid=4294966784, argc=1, argv=0xbff9f270, body=0xb70e0058,
> flags=0)
> > > at eval.c:5665
> > >
> > > >
> > > > >
> > > > > > #11 0x0805c017 in rb_call (klass=3071150400, recv=3071130920,
> > > > > >
> > > > >
> > > >
> > > mid=27825, argc=1, argv=0xbff9f270, scope=0) at eval.c:6048
> > >
> > > >
> > > > >
> > > > > > #12 0x08058127 in rb_eval (self=3071126200, n=0x7af98) at
> ruby.h:654
> > > > > >
> > > > > > _______________________________________________
> > > > > > Mongrel-users mailing list
> > > > > > Mongrel-users at rubyforge.org
> > > > > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Mongrel-users mailing list
> > > > > Mongrel-users at rubyforge.org
> > > > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > Mongrel-users mailing list
> > > > Mongrel-users at rubyforge.org
> > > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > > >
> > > >
> > >
> > > - Ezra Zygmuntowicz
> > > -- Founder & Software Architect
> > > -- ezra at engineyard.com
> > > -- EngineYard.com
> > >
> > >
> > >
> > > _______________________________________________
> > > Mongrel-users mailing list
> > > Mongrel-users at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/mongrel-users
> > >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users

From lists at ruby-forum.com  Mon May  5 09:41:48 2008
From: lists at ruby-forum.com (Roger Pack)
Date: Mon, 5 May 2008 15:41:48 +0200
Subject: [Mongrel] mongrel rails output
Message-ID: <5ce9de1fc984859708fc90168bffc60e@ruby-forum.com>

I don't know if this is a mongrel thing, but
Completed in 0.61607 (1 reqs/sec) | Rendering: 0.59662 (96%) | DB:
0.00000 (0%) | 200 OK [http://ties/]

This line always confuses me.  And I'd imagine it confuses every newbie,
too.  Though useful, it doesn't add up to 100% so confuses slightly.
Maybe an option would be
Completed in 0.61607 (1 reqs/sec) | Rendering: 0.59 (96%) | DB: 0.00
(0%) | Controller: 0.04 (4%) | 200 OK [http://ties/]
Or something to try to help it add up to 100.
Thoughts?
-R
-- 
Posted via http://www.ruby-forum.com/.

From swindsor at gmail.com  Mon May  5 11:42:13 2008
From: swindsor at gmail.com (Scott Windsor)
Date: Mon, 5 May 2008 08:42:13 -0700
Subject: [Mongrel] mongrel rails output
In-Reply-To: <5ce9de1fc984859708fc90168bffc60e@ruby-forum.com>
References: <5ce9de1fc984859708fc90168bffc60e@ruby-forum.com>
Message-ID: <aa2e18ec0805050842s1573174ey522e707f7ec24172@mail.gmail.com>

This is coming from Rails, not from mongrel.
Are you running with log level info?

This bug may shed some light, if so...
http://dev.rubyonrails.org/ticket/10475

- scott

On Mon, May 5, 2008 at 6:41 AM, Roger Pack <lists at ruby-forum.com> wrote:

> I don't know if this is a mongrel thing, but
> Completed in 0.61607 (1 reqs/sec) | Rendering: 0.59662 (96%) | DB:
> 0.00000 (0%) | 200 OK [http://ties/]
>
> This line always confuses me.  And I'd imagine it confuses every newbie,
> too.  Though useful, it doesn't add up to 100% so confuses slightly.
> Maybe an option would be
> Completed in 0.61607 (1 reqs/sec) | Rendering: 0.59 (96%) | DB: 0.00
> (0%) | Controller: 0.04 (4%) | 200 OK [http://ties/]
> Or something to try to help it add up to 100.
> Thoughts?
> -R
> --
> Posted via http://www.ruby-forum.com/.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mongrel-users/attachments/20080505/4056d879/attachment.html>

From gissmog at humanized.de  Mon May  5 12:17:21 2008
From: gissmog at humanized.de (Jochen kaechelin)
Date: Mon, 5 May 2008 18:17:21 +0200
Subject: [Mongrel] mongrel_upload_progress_bar - where are my files?
Message-ID: <A9D5049A-3B29-4875-9097-7DAD019B494A@humanized.de>

> I just configured mongrel_upload_progess_bar following this doc:
>
>       http://mongrel.rubyforge.org/wiki/UploadProgress
>
> My "config/mongrel_upload_progress.conf" looks like:
>
> uri "/",
>  :handler => plugin("/handlers/upload",
>    :path_info => '/Users/railer/Documents/RAILS/dummy/files/upload',
>    :drb => 'druby://127.0.0.1:2999'),
>  :in_front => true
>
>
> Everything seems to work fine, progress bar, form, logfile ...
>
> But I could not find my uploaded files in the path...I only recognize
> some files in /var/folders/...bla ... bla ...
>
> development.log:
>
> Processing FilesController#upload (for 127.0.0.1 at 2008-05-05
> 01:50:19) [POST]
>  Session ID:
> BAh7BzoMY3NyZl9pZCIlNGZjYzE3M2I1M2VhZjA4NzU3NWRhODAwODY3MWQ1
> %0ANGUiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
> %0Ac2h7AAY6CkB1c2VkewA%3D--56884b6ccd656497d776379fa23d7909bc5afb83
>  Parameters: {"commit"=>"Upload",
> "authenticity_token"=>"6a06cc13fcb42087ee3a74e2e6a16c2c0d1df8f6",
> "action"=>"upload", "controller"=>"files", "upload_id"=>"1209944944",
> "data"=>#<File:/var/folders/wY/wYe0+otlEPq0TtDuamD7IU+++TM/-Tmp-/
> CGI808-2>}
> Completed in 0.00295 (339 reqs/sec) | Rendering: 0.00005 (1%) | DB:
> 0.00000 (0%) | 200 OK [http://127.0.0.1/files/upload?upload_id=1209944944
> ]
>
>
> What's wrong here?
>
> Thanx
>
> --
> Jochen

From gissmog at humanized.de  Mon May  5 12:43:19 2008
From: gissmog at humanized.de (Jochen Kaechelin)
Date: Mon, 5 May 2008 18:43:19 +0200
Subject: [Mongrel] mongrel_upload_progess_bar - where are my files?
Message-ID: <BE34C899-3B05-4C5F-B47A-DD2D881987F9@humanized.de>

> I just configured mongrel_upload_progess_bar following this doc:
>
>       http://mongrel.rubyforge.org/wiki/UploadProgress
>
> My "config/mongrel_upload_progress.conf" looks like:
>
> uri "/",
>  :handler => plugin("/handlers/upload",
>    :path_info => '/Users/railer/Documents/RAILS/dummy/files/upload',
>    :drb => 'druby://127.0.0.1:2999'),
>  :in_front => true
>
>
> Everything seems to work fine, progress bar, form, logfile ...
>
> But I could not find my uploaded files in the path...I only recognize
> some files in /var/folders/...bla ... bla ...
>
> development.log:
>
> Processing FilesController#upload (for 127.0.0.1 at 2008-05-05
> 01:50:19) [POST]
>  Session ID:
> BAh7BzoMY3NyZl9pZCIlNGZjYzE3M2I1M2VhZjA4NzU3NWRhODAwODY3MWQ1
> %0ANGUiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
> %0Ac2h7AAY6CkB1c2VkewA%3D--56884b6ccd656497d776379fa23d7909bc5afb83
>  Parameters: {"commit"=>"Upload",
> "authenticity_token"=>"6a06cc13fcb42087ee3a74e2e6a16c2c0d1df8f6",
> "action"=>"upload", "controller"=>"files", "upload_id"=>"1209944944",
> "data"=>#<File:/var/folders/wY/wYe0+otlEPq0TtDuamD7IU+++TM/-Tmp-/
> CGI808-2>}
> Completed in 0.00295 (339 reqs/sec) | Rendering: 0.00005 (1%) | DB:
> 0.00000 (0%) | 200 OK [http://127.0.0.1/files/upload?upload_id=1209944944
> ]
>
>
> What's wrong here?
>
> Thanx
>
> --
> Jochen

From cmdrclueless at gmail.com  Mon May  5 23:37:56 2008
From: cmdrclueless at gmail.com (Brian Weaver)
Date: Mon, 5 May 2008 23:37:56 -0400
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
Message-ID: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>

Hello All,

Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
changes to mongrel that I've written on behalf of
my employer, Raritan Computer, Inc. Also, these changes are released
under the existing Ruby license used for Mongrel.

Raritan is using mongrel internally for product development and was
not able to get the level of scalability desired due to limitations
Mongrel/Rails/Ruby. In particular the single threaded nature of
processing rails request is a serious bottleneck when one or more long
running requests could occur. Originally we used mongrel cluster to
spin up a small number of mongrel rails instance; Apache mod_proxy was
used to hand off requests to the various instances of mongrel.

Unfortunately a long running request could cause our entire web
application to "hang".  As these requests are being processing one or
more mongrel instances continued to accept and enqueue new connections
it can't immediately service.

Mongrel cluster didn't provide any features for dynamically sizing the
number of mongrel_rails instances necessary to keep the application
running smoothly. Also, mongrels nature of continuing to enqueue
connections while processing a long running request was a problem for
our application. In essence we needed a smarter mongrel that would
know which instances of a "cluster" were busy or free; if none were
free then it should spin up a new instance.

The attached patch does just that. It's definitely tailored for a Unix
derivative platform and depends on the functionality of the UNIXSocket
class to pass file descriptors between instances. It works in a
master-slave type of server setup. The "master" is the initial
instance of mongrel rails which goes through its normal initialization
gyrations before forking mulitple "slave" instances to do the actual
rails processing. The server listens on the defined TCP/IP address and
port accepting request (from Apache in our case), determines which
child if any is available, and then hands off the file descriptor for
processing.

The number of children can dynamically grow and shrink depending on
how busy the application is being kept. The patch has fixed our issue
with having long running requests hang the web application.

The patch isn't without any warts. Due to the way that mongrel
preforms initialization I found it difficult to follow the startup
logic. Because of the convoluted
contortions that occur during startup I was unable to
make deeper changes due to time constraints which means that the
following problem exists.

* If you start the server as root and the configuration causes a
setuid/setgid call to a non-root user then during shutdown mongrel may
not be able to remove its pid file.

I'm not aware of any other lurking problem, but I wouldn't be
surprised if one or more exists.

If you have any further question just send me a note or post to the
list. I'll stay subscribed for a few weeks at least. Also, for the
maintainers: I've added a copyright notice for Raritan Computer on
those files that had extensive changes. I believe that is the correct
way to handle major changes, let me know if I'm incorrect in my
understanding.

Enjoy

-- Brian Weaver

*sigh* here is the rest of the text I'm required to include

THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.



-- 

/* insert witty comment here */

From ezmobius at gmail.com  Tue May  6 00:30:49 2008
From: ezmobius at gmail.com (Ezra Zygmuntowicz)
Date: Mon, 5 May 2008 21:30:49 -0700
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
Message-ID: <2148D024-45B1-487A-8FAA-3D78A45DD813@gmail.com>

Hey Brian-

	This is pretty kick ass, I'll definitely take a look.

-Ezra	

On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
> Hello All,
>
> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> changes to mongrel that I've written on behalf of
> my employer, Raritan Computer, Inc. Also, these changes are released
> under the existing Ruby license used for Mongrel.
>
> Raritan is using mongrel internally for product development and was
> not able to get the level of scalability desired due to limitations
> Mongrel/Rails/Ruby. In particular the single threaded nature of
> processing rails request is a serious bottleneck when one or more long
> running requests could occur. Originally we used mongrel cluster to
> spin up a small number of mongrel rails instance; Apache mod_proxy was
> used to hand off requests to the various instances of mongrel.
>
> Unfortunately a long running request could cause our entire web
> application to "hang".  As these requests are being processing one or
> more mongrel instances continued to accept and enqueue new connections
> it can't immediately service.
>
> Mongrel cluster didn't provide any features for dynamically sizing the
> number of mongrel_rails instances necessary to keep the application
> running smoothly. Also, mongrels nature of continuing to enqueue
> connections while processing a long running request was a problem for
> our application. In essence we needed a smarter mongrel that would
> know which instances of a "cluster" were busy or free; if none were
> free then it should spin up a new instance.
>
> The attached patch does just that. It's definitely tailored for a Unix
> derivative platform and depends on the functionality of the UNIXSocket
> class to pass file descriptors between instances. It works in a
> master-slave type of server setup. The "master" is the initial
> instance of mongrel rails which goes through its normal initialization
> gyrations before forking mulitple "slave" instances to do the actual
> rails processing. The server listens on the defined TCP/IP address and
> port accepting request (from Apache in our case), determines which
> child if any is available, and then hands off the file descriptor for
> processing.
>
> The number of children can dynamically grow and shrink depending on
> how busy the application is being kept. The patch has fixed our issue
> with having long running requests hang the web application.
>
> The patch isn't without any warts. Due to the way that mongrel
> preforms initialization I found it difficult to follow the startup
> logic. Because of the convoluted
> contortions that occur during startup I was unable to
> make deeper changes due to time constraints which means that the
> following problem exists.
>
> * If you start the server as root and the configuration causes a
> setuid/setgid call to a non-root user then during shutdown mongrel may
> not be able to remove its pid file.
>
> I'm not aware of any other lurking problem, but I wouldn't be
> surprised if one or more exists.
>
> If you have any further question just send me a note or post to the
> list. I'll stay subscribed for a few weeks at least. Also, for the
> maintainers: I've added a copyright notice for Raritan Computer on
> those files that had extensive changes. I believe that is the correct
> way to handle major changes, let me know if I'm incorrect in my
> understanding.
>
> Enjoy
>
> -- Brian Weaver
>
> *sigh* here is the rest of the text I'm required to include
>
> THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
> WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
> MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> -- 
>
> /* insert witty comment here */
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users



From cmdrclueless at gmail.com  Tue May  6 01:36:32 2008
From: cmdrclueless at gmail.com (Brian Weaver)
Date: Tue, 6 May 2008 01:36:32 -0400
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <2148D024-45B1-487A-8FAA-3D78A45DD813@gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<2148D024-45B1-487A-8FAA-3D78A45DD813@gmail.com>
Message-ID: <7f4149bb0805052236x56ac4af0s5cb375897b441f2d@mail.gmail.com>

Thanks....

I was just reviewing the patch and I've notice several spelling errors
(and grammar problems) in my log messages that need to be fixed. Also,
there isn't any logic handling any kind of IO error on the TCP Server
socket for the UnixDispatchServer. That could be a problem, but I
don't really expect any IO errors on the actual TCP socket for
accepting incoming request from Apache (mod_proxy).

I'll try to review the code changes again after I get some sleep (I
really should have been in bed hours ago) and submit a new patch with
any fixes that I make; cosmetic or otherwise.

-- Weave

On Tue, May 6, 2008 at 12:30 AM, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> Hey Brian-
>
>         This is pretty kick ass, I'll definitely take a look.
>
>  -Ezra
>
>
>  On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
>
> >
> >
> >
> > Hello All,
> >
> > Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> > changes to mongrel that I've written on behalf of
> > my employer, Raritan Computer, Inc. Also, these changes are released
> > under the existing Ruby license used for Mongrel.
> >
> > Raritan is using mongrel internally for product development and was
> > not able to get the level of scalability desired due to limitations
> > Mongrel/Rails/Ruby. In particular the single threaded nature of
> > processing rails request is a serious bottleneck when one or more long
> > running requests could occur. Originally we used mongrel cluster to
> > spin up a small number of mongrel rails instance; Apache mod_proxy was
> > used to hand off requests to the various instances of mongrel.
> >
> > Unfortunately a long running request could cause our entire web
> > application to "hang".  As these requests are being processing one or
> > more mongrel instances continued to accept and enqueue new connections
> > it can't immediately service.
> >
> > Mongrel cluster didn't provide any features for dynamically sizing the
> > number of mongrel_rails instances necessary to keep the application
> > running smoothly. Also, mongrels nature of continuing to enqueue
> > connections while processing a long running request was a problem for
> > our application. In essence we needed a smarter mongrel that would
> > know which instances of a "cluster" were busy or free; if none were
> > free then it should spin up a new instance.
> >
> > The attached patch does just that. It's definitely tailored for a Unix
> > derivative platform and depends on the functionality of the UNIXSocket
> > class to pass file descriptors between instances. It works in a
> > master-slave type of server setup. The "master" is the initial
> > instance of mongrel rails which goes through its normal initialization
> > gyrations before forking mulitple "slave" instances to do the actual
> > rails processing. The server listens on the defined TCP/IP address and
> > port accepting request (from Apache in our case), determines which
> > child if any is available, and then hands off the file descriptor for
> > processing.
> >
> > The number of children can dynamically grow and shrink depending on
> > how busy the application is being kept. The patch has fixed our issue
> > with having long running requests hang the web application.
> >
> > The patch isn't without any warts. Due to the way that mongrel
> > preforms initialization I found it difficult to follow the startup
> > logic. Because of the convoluted
> > contortions that occur during startup I was unable to
> > make deeper changes due to time constraints which means that the
> > following problem exists.
> >
> > * If you start the server as root and the configuration causes a
> > setuid/setgid call to a non-root user then during shutdown mongrel may
> > not be able to remove its pid file.
> >
> > I'm not aware of any other lurking problem, but I wouldn't be
> > surprised if one or more exists.
> >
> > If you have any further question just send me a note or post to the
> > list. I'll stay subscribed for a few weeks at least. Also, for the
> > maintainers: I've added a copyright notice for Raritan Computer on
> > those files that had extensive changes. I believe that is the correct
> > way to handle major changes, let me know if I'm incorrect in my
> > understanding.
> >
> > Enjoy
> >
> > -- Brian Weaver
> >
> > *sigh* here is the rest of the text I'm required to include
> >
> > THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
> > WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
> > MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
> >
> >
> >
> > --
> >
> > /* insert witty comment here */
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
>
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users
>



-- 

/* insert witty comment here */

From ezmobius at gmail.com  Tue May  6 01:42:19 2008
From: ezmobius at gmail.com (Ezra Zygmuntowicz)
Date: Mon, 5 May 2008 22:42:19 -0700
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052236x56ac4af0s5cb375897b441f2d@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<2148D024-45B1-487A-8FAA-3D78A45DD813@gmail.com>
	<7f4149bb0805052236x56ac4af0s5cb375897b441f2d@mail.gmail.com>
Message-ID: <0020B433-EF93-4486-896C-505DFD18EC7B@gmail.com>


	How does this work with the AR database connections? Are they closed  
and reopened in the children? Or do the children load rails after they  
fork?

-Ezra



On May 5, 2008, at 10:36 PM, Brian Weaver wrote:
> Thanks....
>
> I was just reviewing the patch and I've notice several spelling errors
> (and grammar problems) in my log messages that need to be fixed. Also,
> there isn't any logic handling any kind of IO error on the TCP Server
> socket for the UnixDispatchServer. That could be a problem, but I
> don't really expect any IO errors on the actual TCP socket for
> accepting incoming request from Apache (mod_proxy).
>
> I'll try to review the code changes again after I get some sleep (I
> really should have been in bed hours ago) and submit a new patch with
> any fixes that I make; cosmetic or otherwise.
>
> -- Weave
>
> On Tue, May 6, 2008 at 12:30 AM, Ezra Zygmuntowicz  
> <ezmobius at gmail.com> wrote:
>> Hey Brian-
>>
>>        This is pretty kick ass, I'll definitely take a look.
>>
>> -Ezra
>>
>>
>> On May 5, 2008, at 8:37 PM, Brian Weaver wrote:
>>
>>>
>>>
>>>
>>> Hello All,
>>>
>>> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
>>> changes to mongrel that I've written on behalf of
>>> my employer, Raritan Computer, Inc. Also, these changes are released
>>> under the existing Ruby license used for Mongrel.
>>>
>>> Raritan is using mongrel internally for product development and was
>>> not able to get the level of scalability desired due to limitations
>>> Mongrel/Rails/Ruby. In particular the single threaded nature of
>>> processing rails request is a serious bottleneck when one or more  
>>> long
>>> running requests could occur. Originally we used mongrel cluster to
>>> spin up a small number of mongrel rails instance; Apache mod_proxy  
>>> was
>>> used to hand off requests to the various instances of mongrel.
>>>
>>> Unfortunately a long running request could cause our entire web
>>> application to "hang".  As these requests are being processing one  
>>> or
>>> more mongrel instances continued to accept and enqueue new  
>>> connections
>>> it can't immediately service.
>>>
>>> Mongrel cluster didn't provide any features for dynamically sizing  
>>> the
>>> number of mongrel_rails instances necessary to keep the application
>>> running smoothly. Also, mongrels nature of continuing to enqueue
>>> connections while processing a long running request was a problem  
>>> for
>>> our application. In essence we needed a smarter mongrel that would
>>> know which instances of a "cluster" were busy or free; if none were
>>> free then it should spin up a new instance.
>>>
>>> The attached patch does just that. It's definitely tailored for a  
>>> Unix
>>> derivative platform and depends on the functionality of the  
>>> UNIXSocket
>>> class to pass file descriptors between instances. It works in a
>>> master-slave type of server setup. The "master" is the initial
>>> instance of mongrel rails which goes through its normal  
>>> initialization
>>> gyrations before forking mulitple "slave" instances to do the actual
>>> rails processing. The server listens on the defined TCP/IP address  
>>> and
>>> port accepting request (from Apache in our case), determines which
>>> child if any is available, and then hands off the file descriptor  
>>> for
>>> processing.
>>>
>>> The number of children can dynamically grow and shrink depending on
>>> how busy the application is being kept. The patch has fixed our  
>>> issue
>>> with having long running requests hang the web application.
>>>
>>> The patch isn't without any warts. Due to the way that mongrel
>>> preforms initialization I found it difficult to follow the startup
>>> logic. Because of the convoluted
>>> contortions that occur during startup I was unable to
>>> make deeper changes due to time constraints which means that the
>>> following problem exists.
>>>
>>> * If you start the server as root and the configuration causes a
>>> setuid/setgid call to a non-root user then during shutdown mongrel  
>>> may
>>> not be able to remove its pid file.
>>>
>>> I'm not aware of any other lurking problem, but I wouldn't be
>>> surprised if one or more exists.
>>>
>>> If you have any further question just send me a note or post to the
>>> list. I'll stay subscribed for a few weeks at least. Also, for the
>>> maintainers: I've added a copyright notice for Raritan Computer on
>>> those files that had extensive changes. I believe that is the  
>>> correct
>>> way to handle major changes, let me know if I'm incorrect in my
>>> understanding.
>>>
>>> Enjoy
>>>
>>> -- Brian Weaver
>>>
>>> *sigh* here is the rest of the text I'm required to include
>>>
>>> THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
>>> WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
>>> MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
>>>
>>>
>>>
>>> --
>>>
>>> /* insert witty comment here */
>>> _______________________________________________
>>> Mongrel-users mailing list
>>> Mongrel-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>
>>
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
>
>
> -- 
>
> /* insert witty comment here */
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users

- Ezra Zygmuntowicz
-- Founder & Software Architect
-- ezra at engineyard.com
-- EngineYard.com


From zedshaw at zedshaw.com  Tue May  6 01:59:17 2008
From: zedshaw at zedshaw.com (Zed A. Shaw)
Date: Mon, 5 May 2008 22:59:17 -0700
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
Message-ID: <20080506055917.GA17620@zedshaw>

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> Hello All,
> 
> Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
> changes to mongrel that I've written on behalf of
> my employer, Raritan Computer, Inc. Also, these changes are released
> under the existing Ruby license used for Mongrel.

Interesting Brian, glad you finally got the patch into the queue. :-)

> The attached patch does just that. It's definitely tailored for a Unix
> derivative platform and depends on the functionality of the UNIXSocket
> class to pass file descriptors between instances. It works in a
> master-slave type of server setup. The "master" is the initial
> instance of mongrel rails which goes through its normal initialization
> gyrations before forking mulitple "slave" instances to do the actual
> rails processing. The server listens on the defined TCP/IP address and
> port accepting request (from Apache in our case), determines which
> child if any is available, and then hands off the file descriptor for
> processing.

So, I've just read through this patch, and correct me if I'm wrong, but
your patch removes the ability for Mongrel to run as a plain web server
right?  There also seems to be no tests, so does it just use the
existing tests but within the confines of the new forking setup?
 
> The patch isn't without any warts. Due to the way that mongrel
> preforms initialization I found it difficult to follow the startup
> logic. Because of the convoluted
> contortions that occur during startup I was unable to
> make deeper changes due to time constraints which means that the
> following problem exists.

Can you describe what was confusing?  The code's not too large so maybe
it just wasn't documented really clearly.

Also, had you tried just subclassing HttpServer and implementing the
alternative mechanism for both the master and slaves?
 
> * If you start the server as root and the configuration causes a
> setuid/setgid call to a non-root user then during shutdown mongrel may
> not be able to remove its pid file.

The pid file management has always been a mess because of ruby's very
poor process management, and just the variablility in POSIX anyway.
Eventually it'll just have to be a C api that does it right I think.

> I'm not aware of any other lurking problem, but I wouldn't be
> surprised if one or more exists.
> 
> If you have any further question just send me a note or post to the
> list. I'll stay subscribed for a few weeks at least. Also, for the
> maintainers: I've added a copyright notice for Raritan Computer on
> those files that had extensive changes. I believe that is the correct
> way to handle major changes, let me know if I'm incorrect in my
> understanding.

Actually, and this is a big misnomer among open source folks, the
copyright isn't on a file, it's on the project.  It's kind of like
saying if you edit a page in my book with a marker you get to put your
copyright on that page.  Additionally there isn't enough change to that
file to warrant an entire copyright change as it's still primarily
written by myself and other project contributors.

What you can do is either officially donate it back to the project under
the original copyright with a small notice giving credit to yourself
and/or Raritan that you did the master/slave/unixsocket hack, or pull
this out into a separate, completely and wholely written by you/raritan
file that has none of my code and then release under your own license
however you like.

Talk it over with your corporate masters, but considering you have an
entire product based on other people's generously donated free work it
might be a really great nice thing to give back.

Makes people all warm and fuzzy and like your products in exchange.

---
Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/


From zedshaw at zedshaw.com  Tue May  6 02:01:19 2008
From: zedshaw at zedshaw.com (Zed A. Shaw)
Date: Mon, 5 May 2008 23:01:19 -0700
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052236x56ac4af0s5cb375897b441f2d@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<2148D024-45B1-487A-8FAA-3D78A45DD813@gmail.com>
	<7f4149bb0805052236x56ac4af0s5cb375897b441f2d@mail.gmail.com>
Message-ID: <20080506060119.GB17620@zedshaw>

On Tue, May 06, 2008 at 01:36:32AM -0400, Brian Weaver wrote:
> Thanks....
> 
> I was just reviewing the patch and I've notice several spelling errors
> (and grammar problems) in my log messages that need to be fixed. Also,
> there isn't any logic handling any kind of IO error on the TCP Server
> socket for the UnixDispatchServer. That could be a problem, but I
> don't really expect any IO errors on the actual TCP socket for
> accepting incoming request from Apache (mod_proxy).

Well, experience on the project has shown that the second you toss this
thing on FreeBSD you get a whole new cluster of IOError you never knew
existed.  Then put it on OSX and you get 10 more.  Each all an exception
of wonderful clarity.

I'm pretty sure as people try this out you'll find them all though.
 
> I'll try to review the code changes again after I get some sleep (I
> really should have been in bed hours ago) and submit a new patch with
> any fixes that I make; cosmetic or otherwise.

No! Work now! Sleep later! :-)
 
---
Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/


From evan at cloudbur.st  Tue May  6 10:01:40 2008
From: evan at cloudbur.st (Evan Weaver)
Date: Tue, 6 May 2008 10:01:40 -0400
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <20080506055917.GA17620@zedshaw>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<20080506055917.GA17620@zedshaw>
Message-ID: <b6f68fc60805060701m60362846r91926735aa40bba@mail.gmail.com>

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
>  Actually, and this is a big misnomer among open source folks, the
>  copyright isn't on a file, it's on the project.  It's kind of like
>  saying if you edit a page in my book with a marker you get to put your
>  copyright on that page.  Additionally there isn't enough change to that
>  file to warrant an entire copyright change as it's still primarily
>  written by myself and other project contributors.
>
>  What you can do is either officially donate it back to the project under
>  the original copyright with a small notice giving credit to yourself
>  and/or Raritan that you did the master/slave/unixsocket hack, or pull
>  this out into a separate, completely and wholely written by you/raritan
>  file that has none of my code and then release under your own license
>  however you like.

We haven't required copyright assignment from any of the other
contributors, so there's no way we could relicense anyway without
stripping out their code. So I think it is fine for Raritan to keep
the original copyright, since it's under the same license.

I agree that there's no point in the additional copyright notice,
though; Raritan is just one of the many (some anonymous now)
contributors.

>
>  Talk it over with your corporate masters, but considering you have an
>  entire product based on other people's generously donated free work it
>  might be a really great nice thing to give back.
>
>  Makes people all warm and fuzzy and like your products in exchange.
>
>  ---
>  Evil: http://www.zedshaw.com/
>  Utu : http://savingtheinternetwithhate.com/
>
>
>
>  _______________________________________________
>  Mongrel-users mailing list
>  Mongrel-users at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/mongrel-users
>



-- 
Evan Weaver

From cmdrclueless at gmail.com  Tue May  6 10:44:12 2008
From: cmdrclueless at gmail.com (Brian Weaver)
Date: Tue, 6 May 2008 10:44:12 -0400
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <20080506055917.GA17620@zedshaw>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<20080506055917.GA17620@zedshaw>
Message-ID: <7f4149bb0805060744v7742855dh4c75791cc9cd59d8@mail.gmail.com>

See the comments embedded.....

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
> On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
>  > Hello All,
>  >
>  > Ticket #30 (http://mongrel.rubyforge.org/ticket/30)  has a set of
>  > changes to mongrel that I've written on behalf of
>  > my employer, Raritan Computer, Inc. Also, these changes are released
>  > under the existing Ruby license used for Mongrel.
>
>  Interesting Brian, glad you finally got the patch into the queue. :-)
>
>
>  > The attached patch does just that. It's definitely tailored for a Unix
>  > derivative platform and depends on the functionality of the UNIXSocket
>  > class to pass file descriptors between instances. It works in a
>  > master-slave type of server setup. The "master" is the initial
>  > instance of mongrel rails which goes through its normal initialization
>  > gyrations before forking mulitple "slave" instances to do the actual
>  > rails processing. The server listens on the defined TCP/IP address and
>  > port accepting request (from Apache in our case), determines which
>  > child if any is available, and then hands off the file descriptor for
>  > processing.
>
>  So, I've just read through this patch, and correct me if I'm wrong, but
>  your patch removes the ability for Mongrel to run as a plain web server
>  right?  There also seems to be no tests, so does it just use the
>  existing tests but within the confines of the new forking setup?

No, the patch should not remove the ability for mongel to run a a
plain web server. I basically split the HttpServer class into two
classes: Server and HttpServer. I attempted to move most of what would
be common code between the existing HttpServer class and the new
UnixDispatchServer class into the base Server class. If you specify
'--unixmaster' or put 'unixmaster = true' in the mongrel yaml
configuration file then it runs an instance of the UnixDispatchServer;
otherwise it should run the default HttpServer as it was originally
written to do,

Your right there are currently no test. Yes you can flame me all you
want for that; I'll get to it when I have time. I figured in the
spirit of most open source projects I would go ahead and "release
early" to let people play with it so that I could get feedback. The
primary development platform was Linux (CentOS 4 actually) and I
tested it heavily with our application before I posted it. It's far
from perfect, but it is a start.

Besides other people might feel like contributing, I wouldn't want to
deprive them of a decent opportunity by writing all the test myself.
That would seem kind of greedy. ;-)

>
>
>  > The patch isn't without any warts. Due to the way that mongrel
>  > preforms initialization I found it difficult to follow the startup
>  > logic. Because of the convoluted
>  > contortions that occur during startup I was unable to
>  > make deeper changes due to time constraints which means that the
>  > following problem exists.
>
>  Can you describe what was confusing?  The code's not too large so maybe
>  it just wasn't documented really clearly.
>
>  Also, had you tried just subclassing HttpServer and implementing the
>  alternative mechanism for both the master and slaves?
>
>
>  > * If you start the server as root and the configuration causes a
>  > setuid/setgid call to a non-root user then during shutdown mongrel may
>  > not be able to remove its pid file.
>
>  The pid file management has always been a mess because of ruby's very
>  poor process management, and just the variablility in POSIX anyway.
>  Eventually it'll just have to be a C api that does it right I think.
>

Almost all systems support the 'unlink' command. At issues is when you
drop privilege levels from root after creating a pid file then it's
almost impossible to remove. I've gotten around this issue by using a
service script (/etc/init.d/mongrel_rails) on our system to check for
the existence of a pid file, ensure that the process really isn't
running, and then removing the pid before starting up mongrel. It's
not an insurmountable problem, I just figured I would state it up
front to save other developers and users some time.

One could always change the code to save the privilege level allowing
for a new call to setuid/setgid, but this could come back to bite if
the process has a vulnerability. It's a small problem to do the checks
before starting or after stopping mongrel for the added security of
running as a non-privileged user.

>
>  > I'm not aware of any other lurking problem, but I wouldn't be
>  > surprised if one or more exists.
>  >
>  > If you have any further question just send me a note or post to the
>  > list. I'll stay subscribed for a few weeks at least. Also, for the
>  > maintainers: I've added a copyright notice for Raritan Computer on
>  > those files that had extensive changes. I believe that is the correct
>  > way to handle major changes, let me know if I'm incorrect in my
>  > understanding.
>
>  Actually, and this is a big misnomer among open source folks, the
>  copyright isn't on a file, it's on the project.  It's kind of like
>  saying if you edit a page in my book with a marker you get to put your
>  copyright on that page.  Additionally there isn't enough change to that
>  file to warrant an entire copyright change as it's still primarily
>  written by myself and other project contributors.

I'm not a copyright lawyer, but I bet if you talk to a few you would
get some variance on how it really should be done. The point is the
work I've done has been for Raritan. If the changes are such that
copyright law would assign the ownership, for all or part of the
changes, to Raritan then that's who owns it. Raritan is simply trying
to follow the Ruby license that Mongrel is using....

Specifically section 2a of the license at
http://www.ruby-lang.org/en/LICENSE.txt

>
>  What you can do is either officially donate it back to the project under
>  the original copyright with a small notice giving credit to yourself
>  and/or Raritan that you did the master/slave/unixsocket hack, or pull
>  this out into a separate, completely and wholely written by you/raritan
>  file that has none of my code and then release under your own license
>  however you like.
>
>  Talk it over with your corporate masters, but considering you have an
>  entire product based on other people's generously donated free work it
>  might be a really great nice thing to give back.
>
>  Makes people all warm and fuzzy and like your products in exchange.

I know all about warm and fuzzy, giving to the community, paying for
the privilege, and being out of work with no income at the end of the
day. Luckily the work lives own and my previous company was able to
find someone to keep the open source fork going while we tried our
commercial endeavor. The commercial endeavor was a bust, but that's
life. If you have any interest in network management then you should
hop on over to OpenNMS (http://www.opennms.org/) and you can see my
first dabbling with open source.

-- Weave

-- 

/* insert witty comment here */

From lists at ruby-forum.com  Tue May  6 16:55:33 2008
From: lists at ruby-forum.com (Scott Derrick)
Date: Tue, 6 May 2008 22:55:33 +0200
Subject: [Mongrel] expires header for .css
Message-ID: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>

I'm using mongrel as my http server.

I was using yslow to evaluate the performance of my web site and noticed
I was downloading the .css file on every request???  I use one .css file
for the whole site and I though I would be cached?

Yslow indicates that there is no expires date/time set for the css file.

How can I tell mongrel to send an expires header with my static content?

thanks,
Scott
-- 
Posted via http://www.ruby-forum.com/.

From stephanwehner at gmail.com  Tue May  6 17:13:52 2008
From: stephanwehner at gmail.com (Stephan Wehner)
Date: Tue, 6 May 2008 14:13:52 -0700
Subject: [Mongrel] expires header for .css
In-Reply-To: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>
References: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>
Message-ID: <e66d1efb0805061413v72116698k8720c9f6e1d3e149@mail.gmail.com>

On Tue, May 6, 2008 at 1:55 PM, Scott Derrick <lists at ruby-forum.com> wrote:
>  How can I tell mongrel to send an expires header with my static content?

Seems Ben Schwarz ran into the same:

http://germanforblack.com/2007/extreme-expires-headers-for-nginx-and-mongrel


Would this come into play?

$ tail -3 config/environments/production.rb
# Don't want asset-ids
# Want browser caching
ENV['RAILS_ASSET_ID'] = '' # blank.


Stephan
-- 
Stephan Wehner

-> http://stephan.sugarmotor.org
-> http://www.thrackle.org
-> http://www.buckmaster.ca
-> http://www.trafficlife.com
-> http://stephansmap.org

From swindsor at gmail.com  Tue May  6 17:18:23 2008
From: swindsor at gmail.com (Scott Windsor)
Date: Tue, 6 May 2008 14:18:23 -0700
Subject: [Mongrel] expires header for .css
In-Reply-To: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>
References: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>
Message-ID: <aa2e18ec0805061418l36d8fe2emf808c6dcb9af1109@mail.gmail.com>

Is your webserver directly talking to users or are you using a proxy?

You can do this in mongrel, but generally you can configure your proxy to
serve these files directly.  It's much faster to serve them statically.

For nginx, you can use the following...
http://wiki.codemongers.com/NginxRubyonRailsMongrel

in your server block, add the following (it will statically serve and set
the expiration date to the maximum allowed for images, javascript and css).

location ~* \.(js|css|jpg|jpeg|gif|png)$ {
  if (-f $request_filename) {
    expires      max;
    break;
  }
}

Note that you'll have to make sure that when you change
css/javascript/images to either create a new file or add query parameters to
defeat the caching (this is done for you in rails by default).
Additionally, if you're dynamically rendering CSS or JS, then this will
break as well.

Also make sure that you've configured gzip compression for text files
(css/javascript) in nginx as well.  This will also give you a big speed
improvement for most of your clients.

- scott

On Tue, May 6, 2008 at 1:55 PM, Scott Derrick <lists at ruby-forum.com> wrote:

> I'm using mongrel as my http server.
>
> I was using yslow to evaluate the performance of my web site and noticed
> I was downloading the .css file on every request???  I use one .css file
> for the whole site and I though I would be cached?
>
> Yslow indicates that there is no expires date/time set for the css file.
>
> How can I tell mongrel to send an expires header with my static content?
>
> thanks,
> Scott
> --
> Posted via http://www.ruby-forum.com/.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mongrel-users/attachments/20080506/c50a453d/attachment.html>

From lists at ruby-forum.com  Tue May  6 17:50:51 2008
From: lists at ruby-forum.com (Scott Derrick)
Date: Tue, 6 May 2008 23:50:51 +0200
Subject: [Mongrel] expires header for .css
In-Reply-To: <aa2e18ec0805061418l36d8fe2emf808c6dcb9af1109@mail.gmail.com>
References: <f00650ba1ad3b98e40ab47c49e627f94@ruby-forum.com>
	<aa2e18ec0805061418l36d8fe2emf808c6dcb9af1109@mail.gmail.com>
Message-ID: <835addbecd09bff361f43d23bef44939@ruby-forum.com>

Scott Windsor wrote:
> Is your webserver directly talking to users or are you using a proxy?

At this point mongrel is talking directly to the users.   I've been 
looking at nginx but since I'm in a critical phase of development right 
now I didn't want to enter any new services at this time.

> 
> You can do this in mongrel, but generally you can configure your proxy 

Yes I would like to do this in mongrel. Pre recomendation I added this 
to my environmnet.rb

  if defined? Mongrel::DirHandler
    module Mongrel
      class DirHandler
        def send_file_with_expires(req_path, request, response, 
header_only=false)
          response.header['Cache-Control'] = 'max-age=315360000'
          response.header['Expires'] = (Time.now + 10.years).rfc2822
          send_file_without_expires(req_path, request, response, 
header_only)
        end
        alias_method :send_file_without_expires, :send_file
        alias_method :send_file, :send_file_with_expires
      end
    end
  end
 but it doesn't add the expires header to the css file.

thanks for your help in this matter,

Scott



-- 
Posted via http://www.ruby-forum.com/.

From lists at ruby-forum.com  Tue May  6 18:14:27 2008
From: lists at ruby-forum.com (Roger Pack)
Date: Wed, 7 May 2008 00:14:27 +0200
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
Message-ID: <71be3106dcdfd446462ab878112c847c@ruby-forum.com>

> Mongrel cluster didn't provide any features for dynamically sizing the
> number of mongrel_rails instances necessary to keep the application
> running smoothly. Also, mongrels nature of continuing to enqueue
> connections while processing a long running request was a problem for
> our application. In essence we needed a smarter mongrel that would
> know which instances of a "cluster" were busy or free; if none were
> free then it should spin up a new instance.
Reminds me of swiftiply vaguely.
-- 
Posted via http://www.ruby-forum.com/.

From jeremy at hinegardner.org  Thu May  8 02:11:10 2008
From: jeremy at hinegardner.org (Jeremy Hinegardner)
Date: Thu, 8 May 2008 00:11:10 -0600
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
Message-ID: <20080508061110.GJ30749@hinegardner.org>

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> Unfortunately a long running request could cause our entire web
> application to "hang".  As these requests are being processing one or
> more mongrel instances continued to accept and enqueue new connections
> it can't immediately service.

Another solution: 

You could stick HAproxy in front of your mongrel cluster with a the
configuration somewhat like:

  listen application *:80
      balance roundrobin
      server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
      server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
      server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
      server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
      server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check

The 'minconn 1 maxconn 1' will force haproxy to queue the results within
itself instead of mongrel, and the 'check' will take the mongrel out of
rotation if it goes down, and add it back into the rotation as soon as
it comes back up.  As soon as one mongrel is finished with a request,
haproxy will give it one of the ones it has queued.

This is a simple version of what I'm doing to load balance clusters of
mongrels these days.

enjoy,

-jeremy

-- 
========================================================================
 Jeremy Hinegardner                              jeremy at hinegardner.org 


From normalperson at yhbt.net  Thu May  8 03:50:25 2008
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 8 May 2008 00:50:25 -0700
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <20080508061110.GJ30749@hinegardner.org>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<20080508061110.GJ30749@hinegardner.org>
Message-ID: <20080508075025.GB20616@hand.yhbt.net>

Jeremy Hinegardner <jeremy at hinegardner.org> wrote:
> On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> > Unfortunately a long running request could cause our entire web
> > application to "hang".  As these requests are being processing one or
> > more mongrel instances continued to accept and enqueue new connections
> > it can't immediately service.

Heh, this is also the exact problem I designed qrp around, too.

> Another solution: 
> 
> You could stick HAproxy in front of your mongrel cluster with a the
> configuration somewhat like:
> 
>   listen application *:80
>       balance roundrobin
>       server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
>       server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
>       server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
>       server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
>       server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check
> 
> The 'minconn 1 maxconn 1' will force haproxy to queue the results within
> itself instead of mongrel, and the 'check' will take the mongrel out of
> rotation if it goes down, and add it back into the rotation as soon as
> it comes back up.  As soon as one mongrel is finished with a request,
> haproxy will give it one of the ones it has queued.
> 
> This is a simple version of what I'm doing to load balance clusters of
> mongrels these days.

I initially tried doing this with haproxy before I made qrp, but it made
monitoring/testing something against an individual backend Mongrel
process far more difficult as I require Mongrel to accept connections
haproxy didn't forward; so I disabled concurrency on the Mongrel
side instead.

-- 
Eric Wong

From jeremy at hinegardner.org  Thu May  8 12:17:35 2008
From: jeremy at hinegardner.org (Jeremy Hinegardner)
Date: Thu, 8 May 2008 10:17:35 -0600
Subject: [Mongrel] Unix Domain Sockets + Fork for improved scalability
In-Reply-To: <20080508075025.GB20616@hand.yhbt.net>
References: <7f4149bb0805052037s458302f0n8c0111f36d79bab9@mail.gmail.com>
	<20080508061110.GJ30749@hinegardner.org>
	<20080508075025.GB20616@hand.yhbt.net>
Message-ID: <20080508161735.GK30749@hinegardner.org>

On Thu, May 08, 2008 at 12:50:25AM -0700, Eric Wong wrote:
> Jeremy Hinegardner <jeremy at hinegardner.org> wrote:
> > On Mon, May 05, 2008 at 11:37:56PM -0400, Brian Weaver wrote:
> > > Unfortunately a long running request could cause our entire web
> > > application to "hang".  As these requests are being processing one or
> > > more mongrel instances continued to accept and enqueue new connections
> > > it can't immediately service.
> 
> Heh, this is also the exact problem I designed qrp around, too.
> 
> > Another solution: 
> > 
> > You could stick HAproxy in front of your mongrel cluster with a the
> > configuration somewhat like:
> > 
> >   listen application *:80
> >       balance roundrobin
> >       server  mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
> >       server  mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
> >       server  mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
> >       server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
> >       server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check
> > 
> > The 'minconn 1 maxconn 1' will force haproxy to queue the results within
> > itself instead of mongrel, and the 'check' will take the mongrel out of
> > rotation if it goes down, and add it back into the rotation as soon as
> > it comes back up.  As soon as one mongrel is finished with a request,
> > haproxy will give it one of the ones it has queued.
> > 
> > This is a simple version of what I'm doing to load balance clusters of
> > mongrels these days.
> 
> I initially tried doing this with haproxy before I made qrp, but it made
> monitoring/testing something against an individual backend Mongrel
> process far more difficult as I require Mongrel to accept connections
> haproxy didn't forward; so I disabled concurrency on the Mongrel
> side instead.
> 

And we can't forget about ngx_http_upstream_fair_module.c either.

Thanks Grzegorz and Ezra and anyone else who worked on that.

enjoy,

-jeremy


-- 
========================================================================
 Jeremy Hinegardner                              jeremy at hinegardner.org 


From lists at ruby-forum.com  Tue May 13 03:12:16 2008
From: lists at ruby-forum.com (Sankar Ganesh)
Date: Tue, 13 May 2008 09:12:16 +0200
Subject: [Mongrel] mongrel cluster not working
Message-ID: <441c6bda8257927fd5d1cd0077cd7d24@ruby-forum.com>

Hi

am a newbie to mongrel.. and configuring now mongrel cluster with
apache.

when i start mongrel as a stand alone instance my app is working fine,
by issuing the follwoing command.

mongrel_rails mongrel::start

when i start the mongrel cluster it is giving the following error.

mongrel_rails cluster::start


Status: 500 Internal Server Error Content-Type: text/html
500 Internal Server Error

help me please
-- 
Posted via http://www.ruby-forum.com/.

From public1 at dlwi.com  Thu May 15 22:42:28 2008
From: public1 at dlwi.com (Dimitri D.)
Date: Thu, 15 May 2008 22:42:28 -0400
Subject: [Mongrel] mongrel.conf redirects
Message-ID: <482CF494.6030303@dlwi.com>

So regex is apparently supported in mongrel.conf redirects, but I can't 
get it to work.  Anyone have an example of a regex redirect I can see?

From public at misuse.org  Fri May 16 16:34:13 2008
From: public at misuse.org (public at misuse.org)
Date: Fri, 16 May 2008 13:34:13 -0700
Subject: [Mongrel] Some architecture questions for my mongrelian friends
Message-ID: <20080516204117.D2DC118585CF@rubyforge.org>

Hey,

I'm working on a project, and mongrel may be part of the stack, but 
I've got some more general questions and ideas I'm hoping to run by 
this list. The people on this list have a broader knowledgebase and 
more experience than any place else I know - plus a general 
friendliness and willingness to help!

I'm working with a company who has a really antique application stack. 
Literally from 1998. IIS + ASP + MS SQL server. They want me to help 
"modernize" things. In the abstract I'd say, "get a really good .NET 
team and go that route." But they want me to help. All I work in these 
days is Ruby. And that's all I want to work in. :)

So my questions are like this:

1) Can I in good conscience start migrating this company from IIS/ASP 
to Mongrel/Ruby/Merb/ORM (or something like that)? They have roughly 
2-3M page views per month.

1.a) No matter how good they think I am, wouldn't it be smarter to move 
forward with M$ since that's what they've got already? I don't want to 
be the guy who screws them deeper into the hole by really confusing 
their stack.

I hate it when new dudes come in with their "stack" and bias 
development based on their preferences withou considering what's 
already there. I'd rather walk away from this if Microsoft is really 
their odds-on smart choice (i.e. I don't need the money - I have some 
personal relations that led me here). All I want is the company to be 
successful.

2) Their MS SQL setup is relatively fine. Lots of wacky stored procs 
which bug me but mostly it's fine. Am I crazy to try to run MS SQL 
against Ruby/ORM? Seems like there are some people doing it?

3) If I do this, I'd plan to segment this site into two separate boxes 
and run the Ruby on a Linux box (and maybe outsource that management to 
a group like EngineYard). Then have the LB's split traffic between the 
boxes based on url patterns. Again: crazy? unwise? Currently they're at 
rackspace which knows poodle about Ruby/Mongrel afaict.

Context: The front-end site is not impossibly complex. But there is 
"deep" integration with some backend admin processes which run a large 
part of the business: some crm, PPC, finance/accounting, email and 
billing: all partially implemented and built in hand coded ASP. It's a 
real tangle and it breaks all the time right now. I want to get most of 
these processes out into third party systems with much narrower points 
of contact between the production DB's and the specific admin services. 
This can only happen incrementally over time. This is in addition to 
the front-end websystem migration.

Budgets for this work are not tiny but not enormous. Ditto timeframe. 
Maybe $250k over 6-8 months.

Any tips or advice on taking on large migration projects such as this 
would be appreciated. Advice such as "run!" is welcome also. I realize 
there are no definite answers - I'm just looking for experience or 
advice on how to reach conclusions here.

I realize this is horribly off-topic and impossibly vague. And I 
wouldn't ask for this input, except that I highly admire and regard the 
capabilities and experience of many people who are on this list. I 
can't think of a smarter mail list who could help advise on this. Any 
assistance at all will be greatly appreciated.

Thanks!

Steve

p.s. Anyone who has possible interest in this project professionally 
can also contact me directly off-list.


From jftucker at gmail.com  Fri May 16 17:05:38 2008
From: jftucker at gmail.com (James Tucker)
Date: Fri, 16 May 2008 22:05:38 +0100
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <20080516204117.D2DC118585CF@rubyforge.org>
References: <20080516204117.D2DC118585CF@rubyforge.org>
Message-ID: <8CB3FEBF-9983-4F45-BD87-C4CBAD19BB94@gmail.com>


On 16 May 2008, at 21:34, public at misuse.org wrote:

> Hey,
>
> I'm working on a project, and mongrel may be part of the stack, but  
> I've got some more general questions and ideas I'm hoping to run by  
> this list. The people on this list have a broader knowledgebase and  
> more experience than any place else I know - plus a general  
> friendliness and willingness to help!
>
> I'm working with a company who has a really antique application  
> stack. Literally from 1998. IIS + ASP + MS SQL server. They want me  
> to help "modernize" things. In the abstract I'd say, "get a really  
> good .NET team and go that route." But they want me to help. All I  
> work in these days is Ruby. And that's all I want to work in. :)
>
> So my questions are like this:
>
> 1) Can I in good conscience start migrating this company from IIS/ 
> ASP to Mongrel/Ruby/Merb/ORM (or something like that)? They have  
> roughly 2-3M page views per month.

It can be done. Others can better tell you on the hosting costs, but  
there are sites doing this and apparently making money still :)

> 1.a) No matter how good they think I am, wouldn't it be smarter to  
> move forward with M$ since that's what they've got already? I don't  
> want to be the guy who screws them deeper into the hole by really  
> confusing their stack.

The biggest problems you'll hit with packing Ruby into a production  
windows stack are:
  * No decent web proxy, at least, not like nginx and friends.
  * Apache I've benched up to 8000/s on a quad core on Windows with a  
hello world app and the one click installer. If I remember correctly  
the app was a camping one, so not too bad.
  * The mingw builds of ruby are about 30% faster than the OCI build.
  * You'll have to wait for the release of a decent service manager  
for windows, or roll your own monitor for your services.
  * Logging integration is faster to do unintegrated - analogger has  
been working great for us, but you'll need the trunk build from the  
mercurial repo that has my patches.
  * Windows ruby socket implementations (and ruby herself) have low  
limits for the number of file descriptors. JRuby can help I think, we  
proxy out our services where appropriate.

> I hate it when new dudes come in with their "stack" and bias  
> development based on their preferences withou considering what's  
> already there. I'd rather walk away from this if Microsoft is really  
> their odds-on smart choice (i.e. I don't need the money - I have  
> some personal relations that led me here). All I want is the company  
> to be successful.

JRuby could be a good option for you due to the double-stack style of  
it. You could switch to a Java stack underneath if you need speed /  
less magic / java programmers / whatever.

> 2) Their MS SQL setup is relatively fine. Lots of wacky stored procs  
> which bug me but mostly it's fine. Am I crazy to try to run MS SQL  
> against Ruby/ORM? Seems like there are some people doing it?

Personally I'd go PostgreSQL, if they really like their stored  
procedures, that's not going to play nice with most ruby ORMs. We have  
a product that seals the entire DB behind stored procedures for  
security reasons (extreme case). The front end for that app is rails,  
and we have a single point of control that basically replaces  
activerecord (acts as our virtual-ORM) with about 200 lines of ruby,  
and circa 10k lines of (largely generated) plpgsql.

> 3) If I do this, I'd plan to segment this site into two separate  
> boxes and run the Ruby on a Linux box (and maybe outsource that  
> management to a group like EngineYard). Then have the LB's split  
> traffic between the boxes based on url patterns. Again: crazy?  
> unwise? Currently they're at rackspace which knows poodle about Ruby/ 
> Mongrel afaict.

If you want managed services, then yeah, go for a ruby company. There  
are a lot of differences to deploying ruby fast than some places are  
used to or can envisage. As far as splitting traffic goes, eugh, well  
just think about it carefully.

> Context: The front-end site is not impossibly complex. But there is  
> "deep" integration with some backend admin processes which run a  
> large part of the business: some crm, PPC, finance/accounting, email  
> and billing: all partially implemented and built in hand coded ASP.  
> It's a real tangle and it breaks all the time right now. I want to  
> get most of these processes out into third party systems with much  
> narrower points of contact between the production DB's and the  
> specific admin services. This can only happen incrementally over  
> time. This is in addition to the front-end websystem migration.

Do one piece at a time. It sounds like the kind of scenario where  
you'll be needing to refactor the ASP in order to keep things stable  
whilst migrating pieces of a large system.

> Budgets for this work are not tiny but not enormous. Ditto  
> timeframe. Maybe $250k over 6-8 months.

That's not a lot of money to train up people on ruby. Sure ruby is  
simple from the outset, but some ideas don't map so well. Expect 3 to  
6 months developer learning time before they become seriously  
productive in the language, unless you're in a house full of rock  
stars or polygots. Bad ruby code can sometimes be a lot worse than bad  
code in other languages, but it's really going to depend on the team  
and how they take to the language.

> Any tips or advice on taking on large migration projects such as  
> this would be appreciated. Advice such as "run!" is welcome also. I  
> realize there are no definite answers - I'm just looking for  
> experience or advice on how to reach conclusions here.

Do small chunks at a time. Don't be afraid to refactor the current  
system in order to be sure of stability. Seriously write your tests,  
and be careful around the interface boundaries particularly. The  
problem with tests across interface boundaries is that they're either  
coupled (and slow), or functional (and slow), or mocked (and therefore  
one-sided - it'll only test one side of an interface).

> I realize this is horribly off-topic and impossibly vague. And I  
> wouldn't ask for this input, except that I highly admire and regard  
> the capabilities and experience of many people who are on this list.  
> I can't think of a smarter mail list who could help advise on this.  
> Any assistance at all will be greatly appreciated.
>
> Thanks!
>
> Steve
>
> p.s. Anyone who has possible interest in this project professionally  
> can also contact me directly off-list.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users


From zedshaw at zedshaw.com  Fri May 16 17:12:42 2008
From: zedshaw at zedshaw.com (Zed A. Shaw)
Date: Fri, 16 May 2008 17:12:42 -0400
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <20080516204117.D2DC118585CF@rubyforge.org>
References: <20080516204117.D2DC118585CF@rubyforge.org>
Message-ID: <20080516171242.8c309e83.zedshaw@zedshaw.com>

On Fri, 16 May 2008 13:34:13 -0700
public at misuse.org wrote:

> Hey,
> 
> I'm working on a project, and mongrel may be part of the stack, but 
> I've got some more general questions and ideas I'm hoping to run by 
> this list. The people on this list have a broader knowledgebase and 
> more experience than any place else I know - plus a general 
> friendliness and willingness to help!

I'm glad I don't post here that often.  I'd ruin the friendliness. :-)

> So my questions are like this:
> 
> 1) Can I in good conscience start migrating this company from IIS/ASP 
> to Mongrel/Ruby/Merb/ORM (or something like that)? They have roughly 
> 2-3M page views per month.

Ultimately it'd depend on what the application does and what they need
from a rewrite.  If they need maintainability, then I'd say no, you
should work to move them on to .NET so they don't have to retool
everyone.  From my experience, you'll run into insane amounts of
politics unless all the developers from inside the company are the ones
coming forward asking for a transition to Ruby.

If they just want to modernize and are willing to relearn how
deployment works, management, and coding practices, then I'd say it's
worth a shot.  Any language will be better than what they have, but
honesly they should evaluate all the potential options using a rubric
to make sure they cover everything.

Now, if your question is, "Can Ruby handle this kind of load?" then you
didn't ask the question right.  Let's break down your 3M pv/month
metric into requests/second (the actual measurement you need).  That
comes to effectively 34 requests/second on average.  Now, I'm betting
you have a peak time like everyone else, so if your peak is about 4x
this mean then that's 120 request/second, which a decent Rails
application can handle.

Still, you should go look at your web server logs and get a better
understanding of the traffic patterns.

> 1.a) No matter how good they think I am, wouldn't it be smarter to move 
> forward with M$ since that's what they've got already? I don't want to 
> be the guy who screws them deeper into the hole by really confusing 
> their stack.

Yes, that's what I recommend.  Switching to Rails for them would be a
nightmare.  They'd have to ditch all of their existing knowledge,
technology, and platforms to go with the new stuff.  It'd be a whole
rewrite with no prior knowledge.  And I know the existing devs will
revolt, system admins as well.  Actually, the system administrators
will just have to be replaced.  It's rare that an MCSE windows admin
has the chops to change over to unix system admin.

> I hate it when new dudes come in with their "stack" and bias 
> development based on their preferences withou considering what's 
> already there. I'd rather walk away from this if Microsoft is really 
> their odds-on smart choice (i.e. I don't need the money - I have some 
> personal relations that led me here). All I want is the company to be 
> successful.

That's why I recommend the rubric with a shoot out.  Get the devs
involved in coming up with what's important and finding it.  It's a
pretty simple process and should only take a week.  You might be
surprised to find that there's a tech out there that makes the business
just melt.

> 2) Their MS SQL setup is relatively fine. Lots of wacky stored procs 
> which bug me but mostly it's fine. Am I crazy to try to run MS SQL 
> against Ruby/ORM? Seems like there are some people doing it?

Yeah, you're screwed.  The dual problem of logic in the DB (which Rails
HATES) combined with the fact that this means the DBAs control the show
and they're always morons.  Sorry to say it that way, but I have yet to
meet a DBA who is worth two shits and can't be replaced by a good
script or two.

Hell, I've replaced entire departments with a good script or two.

> 3) If I do this, I'd plan to segment this site into two separate boxes 
> and run the Ruby on a Linux box (and maybe outsource that management to 
> a group like EngineYard). Then have the LB's split traffic between the 
> boxes based on url patterns. Again: crazy? unwise? Currently they're at 
> rackspace which knows poodle about Ruby/Mongrel afaict.

Yep, that's possible but probably not the smartest way to do it.  In
fact, if you have to do this splitting of requests then it's a bad
decision.  Be honest with yourself and ask if this is the simplest
thing they could do.  It's not, so go find a simpler migration path for
them and help them deal with that.

> Any tips or advice on taking on large migration projects such as this 
> would be appreciated. Advice such as "run!" is welcome also. I realize 
> there are no definite answers - I'm just looking for experience or 
> advice on how to reach conclusions here.

I'd say run but help them realize why you're not the right guy and who
to get with for the next steps.  Honestly if you can help them spend a
week or two picking the right technology that fits their business and
existing setup then you'd be saving them so much money and could at
least make a bit of extra.

As for actually doing migration projects, I'd say my experience is they
do NOT work unless you have the following:

1) A project champion who fires half the staff as an example of what
will happen if anyone fucks up his project.
2) A project lead who focuses on the quality of the code and can have
anyone above or below him fired by #1 for fucking with his project.
3) Developers who have knowledge of the past, and another bunch who
don't and aren't afraid to call bullshit on the first batch.  If you
don't have people going around saying "WTF!" every two minutes for at
least a month then you have the wrong team entirely.  If you don't have
existing people going "WTF!" at the new guy's dumbfuck ideas as well
then you don't have the right team.
4) NOT ANYONE WITH AN 'A' IN THEIR ACRONYM ON THE PROJECT.  No
Database Admins, System Admins, Software Architects, Business Analysts,
Or MBAs should be near the fucker until it is ready to deploy.
5) Embed the champion from #1 or someone with just as much on the line
into the team every day for the whole work day and make them
participate.
6) Don't move until you've done a full UI design on paper and had a
grahpic artist work it all out.  If #1 doesn't have this before even
talking to you he's fucked.  If this requirement is satisfied by the
existing site then that rocks.

Other than that, these projects always suck, are always full of
politics, are always budget sucks, and you don't really make as much
money as you think you do.

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

From kevwil at gmail.com  Fri May 16 18:01:40 2008
From: kevwil at gmail.com (Kevin Williams)
Date: Fri, 16 May 2008 16:01:40 -0600
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <20080516204117.D2DC118585CF@rubyforge.org>
References: <20080516204117.D2DC118585CF@rubyforge.org>
Message-ID: <683a886f0805161501v7db9456bw3b6a02fa28cf19e9@mail.gmail.com>

I love Ruby, open source, and all that. However, I work with C# and
Java in a Microsoft-only shop. Well, MS-only until I wrote a
distributed cache in Java on Linux - the revolution has begun!

Anyway, we have discussed this scenario at work, many times. How do we
replace the old VB6 COM and ASP code that hides in the corners with
something ... better? The long and the short of it is, if they insist
on using Windows and/or SQL Server, especially if they insist on
stored procedures, you're screwed. Short of replacing the OS with
Linux, the database with PostgreSQL or MySQL, and running with all the
good Ruby/UNIX tools you've mentioned, you'll never get the app where
you want it to be or where they want it to be.

If you must use Windows, you might go with .NET 3.5 and an MVC
framework like Monorail or the not-yet-final ASP.NET MVC (there are
others, too). There are just enough Ruby-esque features in .NET 3.5,
like lambda expressions and extension methods, to keep me from poking
my eyes out with a spork.

If you get them to agree to Linux servers and a more Ruby-friendly DB,
I think you could do them a favor in the long run with lower costs and
easier maintenance. The OS and DB are the keys, in my opinion.

Good luck!

On Fri, May 16, 2008 at 2:34 PM,  <public at misuse.org> wrote:
> Hey,
>
> I'm working on a project, and mongrel may be part of the stack, but I've got
> some more general questions and ideas I'm hoping to run by this list. The
> people on this list have a broader knowledgebase and more experience than
> any place else I know - plus a general friendliness and willingness to help!
>
> I'm working with a company who has a really antique application stack.
> Literally from 1998. IIS + ASP + MS SQL server. They want me to help
> "modernize" things. In the abstract I'd say, "get a really good .NET team
> and go that route." But they want me to help. All I work in these days is
> Ruby. And that's all I want to work in. :)
>
> So my questions are like this:
>
> 1) Can I in good conscience start migrating this company from IIS/ASP to
> Mongrel/Ruby/Merb/ORM (or something like that)? They have roughly 2-3M page
> views per month.
>
> 1.a) No matter how good they think I am, wouldn't it be smarter to move
> forward with M$ since that's what they've got already? I don't want to be
> the guy who screws them deeper into the hole by really confusing their
> stack.
>
> I hate it when new dudes come in with their "stack" and bias development
> based on their preferences withou considering what's already there. I'd
> rather walk away from this if Microsoft is really their odds-on smart choice
> (i.e. I don't need the money - I have some personal relations that led me
> here). All I want is the company to be successful.
>
> 2) Their MS SQL setup is relatively fine. Lots of wacky stored procs which
> bug me but mostly it's fine. Am I crazy to try to run MS SQL against
> Ruby/ORM? Seems like there are some people doing it?
>
> 3) If I do this, I'd plan to segment this site into two separate boxes and
> run the Ruby on a Linux box (and maybe outsource that management to a group
> like EngineYard). Then have the LB's split traffic between the boxes based
> on url patterns. Again: crazy? unwise? Currently they're at rackspace which
> knows poodle about Ruby/Mongrel afaict.
>
> Context: The front-end site is not impossibly complex. But there is "deep"
> integration with some backend admin processes which run a large part of the
> business: some crm, PPC, finance/accounting, email and billing: all
> partially implemented and built in hand coded ASP. It's a real tangle and it
> breaks all the time right now. I want to get most of these processes out
> into third party systems with much narrower points of contact between the
> production DB's and the specific admin services. This can only happen
> incrementally over time. This is in addition to the front-end websystem
> migration.
>
> Budgets for this work are not tiny but not enormous. Ditto timeframe. Maybe
> $250k over 6-8 months.
>
> Any tips or advice on taking on large migration projects such as this would
> be appreciated. Advice such as "run!" is welcome also. I realize there are
> no definite answers - I'm just looking for experience or advice on how to
> reach conclusions here.
>
> I realize this is horribly off-topic and impossibly vague. And I wouldn't
> ask for this input, except that I highly admire and regard the capabilities
> and experience of many people who are on this list. I can't think of a
> smarter mail list who could help advise on this. Any assistance at all will
> be greatly appreciated.
>
> Thanks!
>
> Steve
>
> p.s. Anyone who has possible interest in this project professionally can
> also contact me directly off-list.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>



-- 
Cheers,

Kevin Williams
http://bantamtech.com/
http://almostserio.us/
http://kevwil.com/

From filipe at icewall.org  Fri May 16 19:36:04 2008
From: filipe at icewall.org (Filipe Lautert)
Date: Fri, 16 May 2008 20:36:04 -0300 (BRT)
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <20080516204117.D2DC118585CF@rubyforge.org>
References: <20080516204117.D2DC118585CF@rubyforge.org>
Message-ID: <alpine.DEB.1.10.0805162024490.24650@sophie.icewall.org>


The problem I see in your project is: if you use ruby + mysql, changing 
things they are used to, your projects needs to be perfect (no delays, no 
bugs, no nothing).
Now, if you go and use M$ things, your project don't need to be perfect, 
cause they are used to it and it's problems.
So it's up to you and your trust in your team. I'll tell you that ruby + 
mongrel can handle your load - it handles my geo application running with 
mapserver + postgis + oracle, so just rails + mysql are piece of cake for 
it :)

Cheers,

filipe

On Fri, 16 May 2008, public at misuse.org wrote:

> Hey,
>
> I'm working on a project, and mongrel may be part of the stack, but I've got 
> some more general questions and ideas I'm hoping to run by this list. The 
> people on this list have a broader knowledgebase and more experience than any 
> place else I know - plus a general friendliness and willingness to help!
>
> I'm working with a company who has a really antique application stack. 
> Literally from 1998. IIS + ASP + MS SQL server. They want me to help 
> "modernize" things. In the abstract I'd say, "get a really good .NET team and 
> go that route." But they want me to help. All I work in these days is Ruby. 
> And that's all I want to work in. :)
>
> So my questions are like this:
>
> 1) Can I in good conscience start migrating this company from IIS/ASP to 
> Mongrel/Ruby/Merb/ORM (or something like that)? They have roughly 2-3M page 
> views per month.
>
> 1.a) No matter how good they think I am, wouldn't it be smarter to move 
> forward with M$ since that's what they've got already? I don't want to be the 
> guy who screws them deeper into the hole by really confusing their stack.
>
> I hate it when new dudes come in with their "stack" and bias development 
> based on their preferences withou considering what's already there. I'd 
> rather walk away from this if Microsoft is really their odds-on smart choice 
> (i.e. I don't need the money - I have some personal relations that led me 
> here). All I want is the company to be successful.
>
> 2) Their MS SQL setup is relatively fine. Lots of wacky stored procs which 
> bug me but mostly it's fine. Am I crazy to try to run MS SQL against 
> Ruby/ORM? Seems like there are some people doing it?
>
> 3) If I do this, I'd plan to segment this site into two separate boxes and 
> run the Ruby on a Linux box (and maybe outsource that management to a group 
> like EngineYard). Then have the LB's split traffic between the boxes based on 
> url patterns. Again: crazy? unwise? Currently they're at rackspace which 
> knows poodle about Ruby/Mongrel afaict.
>
> Context: The front-end site is not impossibly complex. But there is "deep" 
> integration with some backend admin processes which run a large part of the 
> business: some crm, PPC, finance/accounting, email and billing: all partially 
> implemented and built in hand coded ASP. It's a real tangle and it breaks all 
> the time right now. I want to get most of these processes out into third 
> party systems with much narrower points of contact between the production 
> DB's and the specific admin services. This can only happen incrementally over 
> time. This is in addition to the front-end websystem migration.
>
> Budgets for this work are not tiny but not enormous. Ditto timeframe. Maybe 
> $250k over 6-8 months.
>
> Any tips or advice on taking on large migration projects such as this would 
> be appreciated. Advice such as "run!" is welcome also. I realize there are no 
> definite answers - I'm just looking for experience or advice on how to reach 
> conclusions here.
>
> I realize this is horribly off-topic and impossibly vague. And I wouldn't ask 
> for this input, except that I highly admire and regard the capabilities and 
> experience of many people who are on this list. I can't think of a smarter 
> mail list who could help advise on this. Any assistance at all will be 
> greatly appreciated.
>
> Thanks!
>
> Steve
>
> p.s. Anyone who has possible interest in this project professionally can also 
> contact me directly off-list.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>

filipe {
  @              icewall.org
  GPG         1024D/A6BA423E
  http://filipe.icewall.org/
}


From baldmountain at gmail.com  Fri May 16 22:48:52 2008
From: baldmountain at gmail.com (Geoffrey Clements)
Date: Fri, 16 May 2008 22:48:52 -0400
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <20080516204117.D2DC118585CF@rubyforge.org>
References: <20080516204117.D2DC118585CF@rubyforge.org>
Message-ID: <472ed2750805161948t7e744bf7x7c56d2b9e9b31018@mail.gmail.com>

There are some interesting issues beyond just the engineering here.
The main ones have to do with how happy the company is with their site
and the technology it uses. If they are happy then maybe they will be
completely satisfied with moving to a .NET type site.

But then you need to look at Microsoft and what their priorities are,
Let's face it, Vista hasn't been particularly successful. You almost
get the feeling that Windows is a cash cow that M$ will milk for as
long as they can, but they are going to move onto other things...

You also need to talk to the engineering staff and see how they feel.
Maybe give a presentation on Rails and see how it goes over. If the
team is intrigued, or better yet, excited, then A Rails application
may be in their future. If it is met with cold stares then it would be
best to move on.

On Fri, May 16, 2008 at 4:34 PM,  <public at misuse.org> wrote:
> Hey,
>
> I'm working on a project, and mongrel may be part of the stack, but I've got
> some more general questions and ideas I'm hoping to run by this list. The
> people on this list have a broader knowledgebase and more experience than
> any place else I know - plus a general friendliness and willingness to help!
>
> I'm working with a company who has a really antique application stack.
> Literally from 1998. IIS + ASP + MS SQL server. They want me to help
> "modernize" things. In the abstract I'd say, "get a really good .NET team
> and go that route." But they want me to help. All I work in these days is
> Ruby. And that's all I want to work in. :)
>
> So my questions are like this:
>
> 1) Can I in good conscience start migrating this company from IIS/ASP to
> Mongrel/Ruby/Merb/ORM (or something like that)? They have roughly 2-3M page
> views per month.

I would consider a nginx/Mongrel/Rails app. The learning curve is a
bit lower and Rails will handle a lot of extras that Merb will not.
(Yet.)

Also consider using Rubber and hosting on Amazon EC2. Rubber makes
using EC2 a piece of cake and it is easy to add instances when your
traffic increases. Plus it pushes the responsibility of managing
server farms and the connection to the internet to Amazon.

>
> 1.a) No matter how good they think I am, wouldn't it be smarter to move
> forward with M$ since that's what they've got already? I don't want to be
> the guy who screws them deeper into the hole by really confusing their
> stack.
>
> I hate it when new dudes come in with their "stack" and bias development
> based on their preferences withou considering what's already there. I'd
> rather walk away from this if Microsoft is really their odds-on smart choice
> (i.e. I don't need the money - I have some personal relations that led me
> here). All I want is the company to be successful.

See above...

> 2) Their MS SQL setup is relatively fine. Lots of wacky stored procs which
> bug me but mostly it's fine. Am I crazy to try to run MS SQL against
> Ruby/ORM? Seems like there are some people doing it?

They may need "wacky" stored procs because IIS and ASP was not up to
the task. You need to evaluate what they were trying to accomplish
with those "wacky" stored procs and see if they aren't standard
features in Rails.

> 3) If I do this, I'd plan to segment this site into two separate boxes and
> run the Ruby on a Linux box (and maybe outsource that management to a group
> like EngineYard). Then have the LB's split traffic between the boxes based
> on url patterns. Again: crazy? unwise? Currently they're at rackspace which
> knows poodle about Ruby/Mongrel afaict.

No idea.

> Context: The front-end site is not impossibly complex. But there is "deep"
> integration with some backend admin processes which run a large part of the
> business: some crm, PPC, finance/accounting, email and billing: all
> partially implemented and built in hand coded ASP. It's a real tangle and it
> breaks all the time right now. I want to get most of these processes out
> into third party systems with much narrower points of contact between the
> production DB's and the specific admin services. This can only happen
> incrementally over time. This is in addition to the front-end websystem
> migration.
>
> Budgets for this work are not tiny but not enormous. Ditto timeframe. Maybe
> $250k over 6-8 months.

Rails, especially if you drink the Koolaid and follow the patterns, is
a RAPID web development platform. Since we don't know what you are
trying to accomplish we can't judge whether you'll be able to meet
this budget and time frame. But with Rails you are more likely to
succeed than with any other platform.

> Any tips or advice on taking on large migration projects such as this would
> be appreciated. Advice such as "run!" is welcome also. I realize there are
> no definite answers - I'm just looking for experience or advice on how to
> reach conclusions here.

Sorry. I've been lucky to start new projects so I have no idea about
migrating an old app to Rails.

> I realize this is horribly off-topic and impossibly vague. And I wouldn't
> ask for this input, except that I highly admire and regard the capabilities
> and experience of many people who are on this list. I can't think of a
> smarter mail list who could help advise on this. Any assistance at all will
> be greatly appreciated.
>
> Thanks!
>
> Steve
>
> p.s. Anyone who has possible interest in this project professionally can
> also contact me directly off-list.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>



-- 
geoff

From public at misuse.org  Sat May 17 17:11:39 2008
From: public at misuse.org (public at misuse.org)
Date: Sat, 17 May 2008 14:11:39 -0700
Subject: [Mongrel] Some architecture questions for my mongrelian friends
In-Reply-To: <mailman.493.1210983502.5981.mongrel-users@rubyforge.org>
References: <mailman.493.1210983502.5981.mongrel-users@rubyforge.org>
Message-ID: <20080517220338.A441318585EE@rubyforge.org>

Thanks James, Zed, Kevin and Filipe. This is really valuable input and 
I hope it helps others besides just me from the archives someday!

I talked with the CEO yesterday more about this yesterday. I think we 
were able to "level" his expectations about what's possible vs. what's 
reasonable, affordable and not too risky.

Surprisingly in this company, the DBA/IT and programmer guys would love 
to abandon their current systems: stored procs, IIS and the whole MS 
house of cards. I guess that shows you how bad the rig is there. The 
system is so old that no one at the company understands how it really 
works and would all just as soon bury it if they could.

But from all your input (which jibes with my concerns), I think it 
would prohibitively expensive to try to shut down the existing 
infrastructure and write a new white box system, especially outside 
their current MS systems.

They can probably milk their existing infrastructure for some good 
profits for a few more years, if they can get people who will actually 
work and learn on it. (Any ideas on how to find folks who are competent 
AND willing to work on old systems? Outsource maybe?!)

Thanks again for all the insights.

Steve

p.s. Maybe I'll see some of you at Railsconf? I'm giving a talk on 
Search strategies in Rails on Friday 5/30 - afternoon sessions. Stop by 
and say hi!


From lists at ruby-forum.com  Mon May 19 17:47:22 2008
From: lists at ruby-forum.com (Arfon Smith)
Date: Mon, 19 May 2008 23:47:22 +0200
Subject: [Mongrel] god - start command exited with non-zero code = 1
Message-ID: <55ffd71f91044fd9085e0c5845fa5db6@ruby-forum.com>

Hi, I'm trying to use god to monitor my mongrels.  When they are running
everything looks OK but when a mongrel process dies and god tries to
restart it I get the following error:

command exited with non-zero code = 1

The weird thing is that if I try use the exact same command as god is
using but on the command line then the mongrel starts back up.

I've seen this error reported a couple of times but no-one ever seems to
get an answer!

Does anyone have any ideas as to what might be going on?!

Cheers

arfon
-- 
Posted via http://www.ruby-forum.com/.

From cnk at caltech.edu  Mon May 19 21:26:10 2008
From: cnk at caltech.edu (Cynthia Kiser)
Date: Mon, 19 May 2008 18:26:10 -0700
Subject: [Mongrel] god - start command exited with non-zero code = 1
In-Reply-To: <55ffd71f91044fd9085e0c5845fa5db6@ruby-forum.com>
References: <55ffd71f91044fd9085e0c5845fa5db6@ruby-forum.com>
Message-ID: <20080520012610.GG12196@blinky.caltech.edu>

Quoting Arfon Smith <lists at ruby-forum.com>:
> The weird thing is that if I try use the exact same command as god is
> using but on the command line then the mongrel starts back up.

That smells like a difference in environment. Are you using full paths
in your god script? Are there any other things that might need
tweaking? For example, I have a Rails mongrel server that use Oracle
as the database. To start it, I need to set $ORACLE_HOME, add some
stuff to $PATH, and to $LD_LIBRARY_PATH. 

-- 
Cynthia Kiser


From lists at ruby-forum.com  Tue May 20 04:41:02 2008
From: lists at ruby-forum.com (Arfon Smith)
Date: Tue, 20 May 2008 10:41:02 +0200
Subject: [Mongrel] god - start command exited with non-zero code = 1
In-Reply-To: <20080520012610.GG12196@blinky.caltech.edu>
References: <55ffd71f91044fd9085e0c5845fa5db6@ruby-forum.com>
	<20080520012610.GG12196@blinky.caltech.edu>
Message-ID: <2ae08d4ed614e576948f11fd12949341@ruby-forum.com>

Hi, thanks for the help.  I was already giving the full path to the 
mongrel_rails and god but the problem turned out to be that I wasn't 
specifying the absolute path to my RAILS_ROOT so it couldn't find the 
application!

Cheers

Arfon

Cynthia Kiser wrote:
> Quoting Arfon Smith <lists at ruby-forum.com>:
>> The weird thing is that if I try use the exact same command as god is
>> using but on the command line then the mongrel starts back up.
> 
> That smells like a difference in environment. Are you using full paths
> in your god script? Are there any other things that might need
> tweaking? For example, I have a Rails mongrel server that use Oracle
> as the database. To start it, I need to set $ORACLE_HOME, add some
> stuff to $PATH, and to $LD_LIBRARY_PATH.
> 
> --
> Cynthia Kiser

-- 
Posted via http://www.ruby-forum.com/.

From lists at ruby-forum.com  Wed May 21 08:41:40 2008
From: lists at ruby-forum.com (Till Mossakowski)
Date: Wed, 21 May 2008 14:41:40 +0200
Subject: [Mongrel] Mongrel Crashes in Production
In-Reply-To: <71166b3b0803180745r2a10fe0ax1c00904c772327de@mail.gmail.com>
References: <8b1b96220823aa131e1d804604795907@ruby-forum.com>
	<71166b3b0803180745r2a10fe0ax1c00904c772327de@mail.gmail.com>
Message-ID: <4b6b41a94cc698e8d12bfbfad0686799@ruby-forum.com>

The problem is that the method reset_application! is no longer available 
in rails 2.0, but Mongrel 1.1.4 is still using it in 
lib/mongrel/rails.rb.

I think you should be able to fix this problem by commenting out the 
line

trap("HUP") { log "HUP signal received."; reload!          }

in /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.4/lib/mongrel/rails.rb.

Till
-- 
Posted via http://www.ruby-forum.com/.

From zedshaw at zedshaw.com  Wed May 21 09:58:41 2008
From: zedshaw at zedshaw.com (Zed A. Shaw)
Date: Wed, 21 May 2008 09:58:41 -0400
Subject: [Mongrel] Mongrel Crashes in Production
In-Reply-To: <4b6b41a94cc698e8d12bfbfad0686799@ruby-forum.com>
References: <8b1b96220823aa131e1d804604795907@ruby-forum.com>
	<71166b3b0803180745r2a10fe0ax1c00904c772327de@mail.gmail.com>
	<4b6b41a94cc698e8d12bfbfad0686799@ruby-forum.com>
Message-ID: <20080521095841.b9da0cd0.zedshaw@zedshaw.com>

On Wed, 21 May 2008 14:41:40 +0200
Till Mossakowski <lists at ruby-forum.com> wrote:

> The problem is that the method reset_application! is no longer available 
> in rails 2.0, but Mongrel 1.1.4 is still using it in 
> lib/mongrel/rails.rb.

Do we know if that needs a fix?

-- 
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

From lists at ruby-forum.com  Wed May 21 10:14:52 2008
From: lists at ruby-forum.com (Till Mossakowski)
Date: Wed, 21 May 2008 16:14:52 +0200
Subject: [Mongrel] Mongrel Crashes in Production
In-Reply-To: <20080521095841.b9da0cd0.zedshaw@zedshaw.com>
References: <8b1b96220823aa131e1d804604795907@ruby-forum.com>
	<71166b3b0803180745r2a10fe0ax1c00904c772327de@mail.gmail.com>
	<4b6b41a94cc698e8d12bfbfad0686799@ruby-forum.com>
	<20080521095841.b9da0cd0.zedshaw@zedshaw.com>
Message-ID: <e304c7b686b00db1c8b70f438a5ebe70@ruby-forum.com>

>> The problem is that the method reset_application! is no longer available 
>> in rails 2.0, but Mongrel 1.1.4 is still using it in 
>> lib/mongrel/rails.rb.
> 
> Do we know if that needs a fix?

Surely it needs. My fix only is a hack. Probably reset_application 
should be replaced with reload_application, but I am not sure.

Till
-- 
Posted via http://www.ruby-forum.com/.

From stephen.bannasch at deanbrook.org  Thu May 22 15:39:22 2008
From: stephen.bannasch at deanbrook.org (Stephen Bannasch)
Date: Thu, 22 May 2008 15:39:22 -0400
Subject: [Mongrel] gem install of mongrel v 1.1.5 broken in jruby
Message-ID: <p06240803c45b70e3772f@[63.138.152.125]>

Can't install the latest mongrel (1.1.5) in JRuby (jruby revisions 
ranging from 6600 to 6750[trunk] checked) -- the platform is not 
being identified and it's trying to do native compilation.

I'm not sure it's actually a problem with Mongrel or JRuby however -- 
perhaps there isn't a mongrel-1.1.5-java.gem on rubyforge?

Some investigation below:

$ jruby -S gem install mongrel
JRuby limited openssl loaded. gem install jruby-openssl for full support.
http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
Bulk updating Gem source index for: http://gems.rubyforge.org/
Building native extensions.  This could take a while...
extconf.rb:1:in `require': no such file to load -- mkmf (LoadError)
	from extconf.rb:1
ERROR:  Error installing mongrel:
	ERROR: Failed to build gem native extension.

Installing 1.1.4 works fine.

$ jruby -S gem install mongrel -v 1.1.4
JRuby limited openssl loaded. gem install jruby-openssl for full support.
http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
Successfully installed mongrel-1.1.4-java
1 gem installed
Installing ri documentation for mongrel-1.1.4-java...
Installing RDoc documentation for mongrel-1.1.4-java...

Specifying the platform in the gem command does not help:

   $ jruby -S gem install mongrel --platform java
   $ jruby -S gem install mongrel --platform jruby

same error.

I don't see a tag for 1.1.5:

   http://mongrel.rubyforge.org/browser/tags

mongrel 1.1.4 is r985 and this is the only change in the Rakefile since then:

   http://mongrel.rubyforge.org/changeset/997/trunk/Rakefile

Don't see how that would be a problem.


From filipe at icewall.org  Thu May 22 15:59:51 2008
From: filipe at icewall.org (Filipe Lautert)
Date: Thu, 22 May 2008 16:59:51 -0300 (BRT)
Subject: [Mongrel] gem install of mongrel v 1.1.5 broken in jruby
In-Reply-To: <p06240803c45b70e3772f@[63.138.152.125]>
References: <p06240803c45b70e3772f@[63.138.152.125]>
Message-ID: <alpine.DEB.1.10.0805221659070.24650@sophie.icewall.org>


I'm not sure, but I think Evan is realising this now.

Cheers,

filipe
On Thu, 22 May 2008, Stephen Bannasch wrote:

> Can't install the latest mongrel (1.1.5) in JRuby (jruby revisions ranging 
> from 6600 to 6750[trunk] checked) -- the platform is not being identified and 
> it's trying to do native compilation.
>
> I'm not sure it's actually a problem with Mongrel or JRuby however -- perhaps 
> there isn't a mongrel-1.1.5-java.gem on rubyforge?
>
> Some investigation below:
>
> $ jruby -S gem install mongrel
> JRuby limited openssl loaded. gem install jruby-openssl for full support.
> http://wiki.jruby.org/wiki/JRuby_Builtin_OpenSSL
> Bulk updating Gem source index for: http://gems.rubyforge.org/
> Building native extensions.  This could take a while...
> extconf.rb:1:in `require': no such file to load -- mkmf (LoadError)
> 	from extconf.rb:1
> ERROR:  Error installing mongrel:
> 	ERROR: Failed to build gem native extension.
>
> Installing 1.1.4 works fine.
>
> $ jruby -S gem install mongrel -v 1.1.4
> JRuby limited openssl loaded. gem install jruby-openssl for full support