[Mongrel] Rails App on Mongrel+Apache Stops Listening

Zed A. Shaw zedshaw at zedshaw.com
Wed Jan 24 04:07:04 EST 2007


On Tue, 23 Jan 2007 21:53:39 -0800
Hunter Hillegas <lists at lastonepicked.com> wrote:

> Howdy,
> 
> I'm trying to debug a situation where a Rails app running on Mongrel 
> +Apache stops listening to requests. Restarting Mongrel and Apache  
> brings it back to life.
> 
> First off, here's some info on versions, etc...:

Here's the shortlist of stuff so far that can cause this:

1) Using the stock mysql.rb that comes with rails rather than using the compiled mysql gem.
2) Use a pstore for sessions, or actually trying to negotiate file locking in general.
3) Trying to resolve dns queries against a slow DNS server or broken one.
4) Using memcached and putting keys in it with a space or \0 char in them (memcached don't like that and will block the world).
5) An interesting new one is having Monit check the mongrel port on sites with long running requests.  What happens is since rails blocks further requests during these requests, and monit is typically told to restart when they're slow, you'll get frequent "slow" requests so monit will frequently restart mongrels for no apparent reason.

Ways to debug this include:

1) Using lsof -i -P to see what's open.
2) Attaching to the stuck process with gdb and using some of the Ruby gdb scripts out there to force an exception to be thrown.  Then look in the mongrel.log to see what is logged.
3) Running strace or truss on the stuck process to see what system call it's doing.  If it's repeatedly running select() then that means it is most likely trying to write to a socket but getting nothing back.  This happens in the memcached case especially.
4) Praying to some ancient god for redemption.

I hear #4 works really well if you're bathed in the blood of 7 orphaned virgins, but I'm not that hardcore.

-- 
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.


More information about the Mongrel-users mailing list