[Mongrel] Memory leaks in my site

Rob Sanheim rsanheim at gmail.com
Tue Mar 6 16:23:20 EST 2007


On 3/6/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
> On Mar 6, 2007, at 12:34 PM, Joey Geiger wrote:
>
> >> flex_image uses rmagick... which folks have lots of issues with...
> >
> > I re-wrote the bits of flex_image that I use to use mini-magick and
> > it's working quite nicely.
> > I did that when the whole discussion of "RMagick bad" came up a couple
> > of months ago.
> >
> > On 3/6/07, Alexey Verkhovsky <alexey.verkhovsky at gmail.com> wrote:
> >> So, memory leak is the problem, and monit is just highlighting it.
> >> Is that
> >> correct?
> >
> > For me, yes. I'm not exactly sure what is happening, but on my current
> > application, the only controller that was hit overnight was the monit
> > checker, which was enough to cause the leak.
> >
> > Here's a copy of the controller, which I've tried to strip down as
> > much as possible. (On my current production app, it's doing 10k
> > requests per second :)
> >
> > class MonitController < ActionController::Base
> >   session :off
> > ##   this is used by the monitoring scripts to see if the mongrel is
> > up and running
> >   def index
> >   end
> > end
>
>
>         I have seen weird behavior from having monit check with a http
> request. It woudl cause weird issues and mem leaks. I have since
> removed any http checks from all my monit recipes and used an
> external siteuptime monitoring for http status check. Monit just
> watches memory and cpu usage and restarts dead or bloated mongrels.
>
>         It's just a fact of life with large rails applications, some of them
> just like to keep using memory as much as possible. I have found that
> restarting mongrels that go over 110Mb for a few cycles will keep
> things in check pretty well. This is for 64bit systems where memory
> usage is higher then on 32 bit systems.
>
>         Here is a monit recipe I am using on a large number of servers that
> seems to work very well.
>
> set httpd port 9111
>      allow localhost
>
> set daemon 60
> set logfile /data/username/shared/log/monit.log
> set mail-format {from:info at engineyard.com}
> set mailserver smtp.engineyard.com
> set alert eymonit at gmail.com
>
> check process mongrel_username_0
>    with pidfile /data/username/shared/log/mongrel.5000.pid
>    start program = "/data/username/shared/bin/start_mongrel.sh 5000
> username"
>    stop program = "/data/username/shared/bin/stop_mongrel.sh 5000
> username"
>    if totalmem is greater than 110.0 MB for 4 cycles then
> restart       # eating up memory?
>    if cpu is greater than 50% for 2 cycles then
> alert                  # send an email to admin
>    if cpu is greater than 80% for 3 cycles then
> restart                # hung process?
>    if loadavg(5min) greater than 10 for 8 cycles then
> restart          # bad, bad, bad
>    if 10 restarts within 10 cycles then
> timeout                         # something is wrong, call the sys-admin
>    group mongrel
>
> check process mongrel_username_1
>    with pidfile /data/username/shared/log/mongrel.5001.pid
>    start program = "/data/username/shared/bin/start_mongrel.sh 5001
> username"
>    stop program = "/data/username/shared/bin/stop_mongrel.sh 5001
> username"
>    if totalmem is greater than 110.0 MB for 4 cycles then
> restart       # eating up memory?
>    if cpu is greater than 50% for 2 cycles then
> alert                  # send an email to admin
>    if cpu is greater than 80% for 3 cycles then
> restart                # hung process?
>    if loadavg(5min) greater than 10 for 8 cycles then
> restart          # bad, bad, bad
>    if 10 restarts within 10 cycles then
> timeout                         # something is wrong, call the sys-admin
>    group mongrel
>
> check process mongrel_username_2
>    with pidfile /data/username/shared/log/mongrel.5002.pid
>    start program = "/data/username/shared/bin/start_mongrel.sh 5002
> username"
>    stop program = "/data/username/shared/bin/stop_mongrel.sh 5002
> username"
>    if totalmem is greater than 110.0 MB for 4 cycles then
> restart       # eating up memory?
>    if cpu is greater than 50% for 2 cycles then
> alert                  # send an email to admin
>    if cpu is greater than 80% for 3 cycles then
> restart                # hung process?
>    if loadavg(5min) greater than 10 for 8 cycles then
> restart          # bad, bad, bad
>    if 10 restarts within 10 cycles then
> timeout                         # something is wrong, call the sys-admin
>    group mongrel

Ezra,

What do you use for uptime monitoring?  I haven't found anything out
of the box that can check often enough (ie at least every minute).

- Rob


More information about the Mongrel-users mailing list