[Mongrel] Mongrel stops responding after period of inactivity

Piyush Ranjan piyush.pr at gmail.com
Wed Aug 1 03:22:37 EDT 2007


Hi
Thank you for your suggestions.
I have a few sites running on mongrel. Only one of them has this "sleep
disorder". I  later realized that due to constant load (throughout the day)
on other sites they were working fine whereas this forum had load only
during the day and hence did die overnight. I have inspected the PIDs and
found that mongrel remain in the sleeping mode even after repeated
connection request. For now I have a nagios looking after it(pinging it
every 15 mins).
I must point out here that the DB here is Oracle and mysql is not in the
picture at all


On 8/1/07, Zed A. Shaw <zedshaw at zedshaw.com> wrote:
>
> On Sun, 29 Jul 2007 22:57:29 +0100
> "Olly Lylo" <list at lylo.co.uk> wrote:
>
> > Hi
> > I posted this to the Ruby on Rails Talk group but I thought I'd post it
> here
> > too as it's probably a more appropriate group.  Hope this is ok.
>
>
> Alright, I've got a few minutes to write this down since it seems to be
> biting a few people.
>
> Here's the rules for resolving a "RAILS in Mongrel dies" problem (notice
> the RAILS part?):
>
> 1) 1000's of sites run mongrel without this problem so look to what you
> have installed first.
> 2) Make sure that every single bit of software you are running is the most
> recent version and everything is installed and used correctly.  Common
> culprits are:
>   a) MySQL -- install the gem manually and make sure that the most recent
> is the only one.  DO NOT USE THE RAILS DEFAULT.
>   b) Memcached -- Use the very latest from Eric Hodel's project and do NOT
> put any keys in that have spaces or null (\0) chars.  Yes, your keys cannot
> have spaces.  YES YOUR KEYS CANNOT HAVE SPACES.
>   c) net::http to some web site.  This can cycle forever.
> 3) Next, once you've made sure all the above is isolated then you can
> proceed to stage 2.
>
> STAGE 2
>
> 1) You must let your application run in the best configuration and be
> ready to pounce on it the second a mongrel dies.
> 2) Log in to your server and find out the PID file of the mongrel that
> isn't responding.  You do this by hitting the process on it's actual port
> (like 8000, 8001, 8002, NOT apache/nginx's port of 80).
> 3) Once you know what port it is, then use: sudo lsof -i -P | grep PORT to
> find what process PID is on that port.
> 4) Next, attach to this process with: sudo strace -p PID.  What you'll see
> in a healthy mongrel is lots of variation.  What you'll see in a dead
> mongrel is probably either a bunch of calls to select/poll for the exact
> same filedescriptors, or nothing.
> 5) If you see it doing a select on the file descriptors, then you need to
> find out what is on that FD that is causing it to wait.  Again, use: lsof |
> grep FILEDESCR
>   a) This will potentially tell you where it's connected, etc.  Once you
> do this you know what is being read/written and can go find out what in your
> rails app is using it, and apply even more debugging tools.
>   b) You can also just force it closed externally.  There's a few mentions
> of this but I don't remember the exact procedure.
>   c) The reason this typically happens is that you have a socket that ruby
> has written to and not read from yet, but that socket is closed.  Another
> cause is that you are simply waiting for data (which is what happens with
> memcached and putting a space in your keys).
> 6) If your mongrel is completely dead then move on to stage 3 and also
> you're kind of fucked.
>
> STAGE 3
>
> 1) Get your good shoes on because you're now in GDB land and C.
> 2) http://eigenclass.org/hiki.rb?ruby+live+process+introspection  Explains
> attaching to your mongrel process (you've identified above) using GDB,
> loading some GDB scripts, and then stopping it, inspecting it, and forcing
> an exception.
> 3) Do those thigns.  Attach.  Pause it. Inspect variables.  Get stack
> traces. Force an exception.  See where it's stopped.  Look for where OUTSIDE
> of mongrel it's coming from.
> 4) That's all I can say right now.  Let's hope other people can expand on
> this.
>
> --
> Zed A. Shaw
> - Hate: http://savingtheinternetwithhate.com/
> - Good: http://www.zedshaw.com/
> - Evil: http://yearofevil.com/
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20070801/6b73793c/attachment-0001.html 


More information about the Mongrel-users mailing list