[Mongrel] Mongrel 0.3.13.3 processes occasionally lock up - critical issue

kigsteronline at mac.com kigsteronline at mac.com
Thu Aug 31 01:39:43 EDT 2006


Zed and friends,

Turns out our locking up issue was related to the corrupt ruby stack  
in our application.

We were able to reproduce this bug in Webrick and LightHTTPD, so this  
is not Mongrel specific.

Thanks Zed for all the help,
Konstantin


On Aug 30, 2006, at 5:42 PM, Zed Shaw wrote:

> On Wed, 2006-08-30 at 15:10 -0700, kigsteronline at mac.com wrote:
>> Hello Mongrel gurus!
>>
>>
>> We are about to deploy our Rails application to production, and even
>> though the last couple of weeks of testing went really well, today we
>> are seeing a new critical issue - the browser just hangs sometimes
>> trying to load a page. When this happens, it seems that one or  
>> more of
>> the mongrel_rails processes is hanging and not responding, while
>> others are (so depending on which process the request is routed to,
>> the page loads or not).  The only remedy from this point onward
>> appears to be brutal kill and restart of mongrel processes.
>>
>
> Ah, you might be hitting the CLOSE_WAIT bug.  This makes me think more
> and more that Apache is doing something funky.  This bug and a 99% CPU
> bug is driving down right insane since I can't replicate it to save my
> life.  I've got mongrel serving ISO images to an RFuzz script that
> randomly closes sockets and does nasty requests while thrashing  
> mongrel
> with httperf with a short timeout.
>
> Nothing.  Not even one CLOSE_WAIT socket, no memory leaks  
> (anymore), and
> no CPU thrashing.  It's really driving me nuts.
>
> What I've done is created a special "debug version" of the mongrel
> pre-release that I'd like you to run.  Here's what you do:
>
> 1) download http://www.zedshaw.com/mongrel-0.3.13.4.gem
> 2) install:  gem install mongrel-0.3.13.4.gem
> 3) start your app just like it is.
> 4) Hit mongrels with USR1:  killall -USR1 mongrel_rails
> 5) Let mongrel run until it dies horribly.
> 6) Just before you kill -9 mongrel, run:  lsof -i -P | grep  
> CLOSE_WAIT >
> close_wait.log
> 7) Stop mongrel and send me log/mongrel.log and close_wait.log to
> zedshaw at zedshaw.com.
>
> Hopefully I can nail this for the next release.  The CLOSE_WAIT  
> used to
> be really bad but I cut it down quite a bit, so it should take  
> longer to
> detroy it.
>
> Also make sure to tell me if the CPU usage for mongrel is above 90%
> (it's pegged).
>
> Thanks for the help.
>
>
>
> -- 
> Zed A. Shaw
> http://www.zedshaw.com/
> http://mongrel.rubyforge.org/
> http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users



More information about the Mongrel-users mailing list