[Mongrel] Mongrel server file descriptor leak?

benjamin grant bgrant01101 at gmail.com
Tue Feb 10 16:06:29 EST 2009


This is what Mongrel invokes when that limit is hit:

    def reap_dead_workers(reason='unknown')
      if @workers.list.length > 0
        STDERR.puts "#{Time.now}: Reaping #{@workers.list.length} threads
for slow workers because of '#{reason}'"
        error_msg = "Mongrel timed out this thread: #{reason}"
        mark = Time.now
        @workers.list.each do |worker|
          worker[:started_on] = Time.now if not worker[:started_on]

          if mark - worker[:started_on] > @timeout + @throttle
            STDERR.puts "Thread #{worker.inspect} is too old, killing."
            worker.raise(TimeoutError.new(error_msg))
          end
        end
      end


So check your mongrel logs and see if mongrel is attempting (and possibly
failing) to reap workers.


You could also slap this into application.rb:

    def timeout
        Timeout.timeout(APPCONTROLLER_TIMEOUT) do
            yield
        end
    end

Where APPCONTROLLER_TIMEOUT is the number of seconds you want any given
controller action to run for.  This will effecively force timeouts to occur
independnt of host/os limits and mongrel's handling of exceptions thrown
when reaching them.

I've observed good results with that in some cases.


--> But, see also: http://ph7spot.com/articles/system_timer for more on
timeouts and how effective (or ineffective) they can be.


Meanwhile, raising that open files limit might give you some time to
bug-hunt with.  Depends on how frequent the problem is being triggered.
And have a look at HAProxy and/or NGINX if you're not already running them
as a proxy tier / combo. in front of mongrel.

--

On Tue, Feb 10, 2009 at 12:06 PM, John Nestoriak <lists at ruby-forum.com>wrote:

> Benjamin Grant wrote:
> > Can you post the content of your mongrel_cluster.yml  (or equivalent
> > command
> > line options in use) & mongrel version...?
> >
> > Also the output of "ulimit -a" for the mongrel process owner.
> >
> > --
>
> The mongrels are started under CPanel and we're not using clusters.
>
> >From ps it looks like they're just started with:
> /usr/bin/ruby /usr/bin/mongrel_rails start -p 12005 -d -e production -P
> log/mongrel.pid
>
> It's running mongrel 1.1.5
> ruby 1.8.7 (2008-06-20 patchlevel 22) [i686-linux]
>
> I'm not sure exactly how many file descriptors causes it to die - though
> I've seen 400+ and it running fine and I'm guessing it dies at 512.
>
> Ulimit doesn't seem like it's changed at all.
>
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 1024
> max locked memory       (kbytes, -l) 32
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 73728
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> --
> Posted via http://www.ruby-forum.com/.
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/mongrel-users/attachments/20090210/d6ee078b/attachment.html>


More information about the Mongrel-users mailing list