[Mongrel] Mongrel spewing backtraces and nanosleeping

Jacob Atzen jacob at jacobatzen.dk
Thu Sep 14 05:27:01 EDT 2006

On Wed, Sep 13, 2006 at 03:24:25PM -0700, Zed Shaw wrote:
> On Wed, 2006-09-13 at 16:00 +0200, Jacob Atzen wrote:
> > Hi list,
> > 
> > I'm seeing a couple of issues with Mongrel. I'm running FreeBSD 6.1 and
> > have previously been told that there are known conflicts between this
> > and Mongrel, yet I hope these issues will be resolved with time.
> > 
> > I'm overloading Mongrel with httperf on my local workstation. Mongrel is
> > started directly with the mongrel_rails command and there is only one
> > mongrel running. Soon I will start seeing:
> > 
> This guy walks into the doctors office and says, "Doc! It hurts when I
> do this."  He then sticks his finger in his eye and gouges out his
> cornea and eats it.  The doctor yells, "What the hell did you do that
> for?!"  Guy says, "So I can test how much damage my eye can take.  See,
> it sucks, I need a stronger eye."

I will try to explain my motivation for the tests performed. I was
seeing Mongrel processes freezing in the sense that they stopped
handling incoming requests on a production system. As part of the
debugging of this I tried to create a setup where I minimized the
external factors (Apache, Rails to a certain extent) and putting load on
this to see if I could reproduce this behaviour. Now if anyone has
suggestions on better ways of doing this I'm all ears.

> In other words, congratulations you've overloaded Mongrel with httperf.
> That's not too hard since Ruby only has 1024 files and you're probably
> hitting a rails action that opens loads and loads of files or just plain
> maxing it out.

I'm seeing the same behaviour with 256 threads and lsof shows a stable
amount of files opened by Ruby (around 290) throughout the running of
the test. Additionally sockstat shows 255 sockets opened by Ruby

My rails action is trivial, this is the implementation:

  def test
    render :text => "Just testing"

> The way mongrel defends itself is to try closing things off, but there's
> not much it actually can do.  It's at the mercy of Ruby and that's that.
> End of story.
> So, there's no "issue" other than you need:
> 1) To tune your rails actions to be faster.
> 2) More mongrel processes.
> 3) To not do this to mongrel.
> 4) Maybe more machines if this is the load you have to handle.
> You should also get the pre-release and use the USR1 logging
> (mentioned on this list many times) to debug which rails action is
> taking the longest and holding up the other threads.  Then tune that
> rails action up to service the requests faster.
> Hope that helps.

Of course I should have mentioned the following in my previous post, I'm
sorry for the omission:

- I am running the pre-release fetched this tuesday.

- The Rails action is minimized as shown above.

- I am using USR1 debugging and seeing output resembling:

  Wed Sep 13 00:34:36 CEST 2006: 156 threads sync_waiting for /pages/test, 1000 still active in mongrel.

  Now Rails is telling me it can complete 1000+ requests a second, so I
  interpret this number as it's Ruby/Mongrel/Whatever that's using the
  majority of the time processing the requests. Of course my
  interpretation may be wrong.

I'm not really concerned about performance.  The problem is that Mongrel
from time to time will end up in what appears to be an endless loop
where it stops responding to anything besides a kill -9. 

I've noticed that both the open socket count and the open file count
stays constant when this happens, which to me is a further indication
that something is stuck somewhere.

Finally the problem occurs after very different amounts of requests,
sometimes it will show after only 30-40000 hits from httperf, sometimes
I can get 10 times that. I will not start guessing as to why this is.

I don't mind Mongrel trying to defend itself and closing things off. And
if there's something inherent in Ruby or whatever which means that if
Mongrel gets loaded it may suddenly stop responding I will just go back
to mod_fcgid.

In conclusion I believe there is an issue, namely that Mongrel from time
to time freezes totally and never recovers. I hope the above explanation
clarifies my position. If you do not think this is an issue that's fine
and I will not pursue the matter further.

- Jacob Atzen

More information about the Mongrel-users mailing list