[Mongrel] Mongrel processes occasionally lock up - critical issue

Zed Shaw zedshaw at zedshaw.com
Wed Aug 30 20:42:20 EDT 2006

On Wed, 2006-08-30 at 15:10 -0700, kigsteronline at mac.com wrote:
> Hello Mongrel gurus!
> We are about to deploy our Rails application to production, and even
> though the last couple of weeks of testing went really well, today we
> are seeing a new critical issue - the browser just hangs sometimes
> trying to load a page. When this happens, it seems that one or more of
> the mongrel_rails processes is hanging and not responding, while
> others are (so depending on which process the request is routed to,
> the page loads or not).  The only remedy from this point onward
> appears to be brutal kill and restart of mongrel processes.

Ah, you might be hitting the CLOSE_WAIT bug.  This makes me think more
and more that Apache is doing something funky.  This bug and a 99% CPU
bug is driving down right insane since I can't replicate it to save my
life.  I've got mongrel serving ISO images to an RFuzz script that
randomly closes sockets and does nasty requests while thrashing mongrel
with httperf with a short timeout.

Nothing.  Not even one CLOSE_WAIT socket, no memory leaks (anymore), and
no CPU thrashing.  It's really driving me nuts.

What I've done is created a special "debug version" of the mongrel
pre-release that I'd like you to run.  Here's what you do:

1) download http://www.zedshaw.com/mongrel-
2) install:  gem install mongrel-
3) start your app just like it is.
4) Hit mongrels with USR1:  killall -USR1 mongrel_rails
5) Let mongrel run until it dies horribly.
6) Just before you kill -9 mongrel, run:  lsof -i -P | grep CLOSE_WAIT >
7) Stop mongrel and send me log/mongrel.log and close_wait.log to
zedshaw at zedshaw.com.

Hopefully I can nail this for the next release.  The CLOSE_WAIT used to
be really bad but I cut it down quite a bit, so it should take longer to
detroy it.

Also make sure to tell me if the CPU usage for mongrel is above 90%
(it's pegged).

Thanks for the help.

Zed A. Shaw
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

More information about the Mongrel-users mailing list