[Mongrel] problems with apache 2.2 proxying to mongrel cluster

Michael Kovacs kovacs at gmail.com
Tue Jan 2 21:04:50 EST 2007


Yep log rolling was the problem. I setup my staging box and  
reproduced it just fine by rolling the date.
Taking out the log roller command from my environment.rb setup worked  
just fine when I rolled days.

-Michael
http://javathehutt.blogspot.com

On Jan 2, 2007, at 2:55 PM, Ezra Zygmuntowicz wrote:

> Hey Michael-
>
> 	You should definitely now use the ruby logger to do rotation. That
> has caused tons of issues for me and others. Use syslog or logrotate
> to rotate the logs as the Logger rotator is borken trust me.
>
> 	As for why you are having opne process stay alive and the others
> die. I think what causes that is that all four of your mongrels will
> try to rotate the logs. And its a free for all. So the one who does
> it last 'wins' and stays alive. Or if it is a mysql timeout issue
> then what is probably happening is that only one of your mongrels is
> getting 'exercised' . By this I mean the load balancer is only
> sending requests to one mongrel because there is not enough traffic
> to make it drive all four mongrels. This means that the one mongrel
> is getting requests which keep its db connection alive. While the
> others are not getting requests so their db connectiosn die off.
>
> 	I think it's one of these two things. Sop don't use Logger to rotate
> logs for starters and see if that fixes it for you.
>
> Cheers
> -Ezra
>
> On Jan 2, 2007, at 2:32 PM, Michael Kovacs wrote:
>
>> Hmm... thanks for the suggestion I'll give that a look as well as
>> search the archives here.
>>
>> I do have the logger rotating daily right now:
>> config.logger = Logger.new(config.log_path, 'daily')
>>
>> I think I'll remove that and see if this occurs again. The thing that
>> still bothers me though
>> is that if one mongrel is broken from this shouldn't they all be? I
>> can always count on 1 of 4 mongrels
>> working so that every 3rd request is successful :-) In order to
>> monitor that I'd have to hit the site twice to
>> ensure that I'm not just getting the good mongrel on the first hit.
>> Not very fresh.
>>
>> -Michael
>> http://javathehutt.blogspot.com
>>
>> On Jan 2, 2007, at 1:36 PM, Joey Geiger wrote:
>>
>>> This may have to do with log rotation. There was a thread about
>>> similar issue posted in I believe December. You might want to try
>>> searching the mailing list archives.
>>>
>>> You're probably rotating your logs based on a specific size,  
>>> which is
>>> why it's happening every couple hours, instead of nightly.
>>>
>>> On 1/2/07, Michael Kovacs <kovacs at gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> I've been having problems with the apache 2.2-mod_proxy_balancer-
>>>> mongrel
>>>> setup.
>>>>
>>>> My setup is:
>>>>
>>>> CentOS 4.3
>>>> apache 2.2.3 (compiled from source) with mod_proxy_balancer
>>>> mysql 4.1
>>>> ruby 1.8.4
>>>> mongrel 0.3.14 (I know I need to update but I think this problem is
>>>> independent of the mongrel version)
>>>> mongrel_cluster 0.2.0
>>>> rails_machine 0.1.1
>>>>
>>>> I have apache setup as per Coda's configuration on his blog
>>>> posting from
>>>> several months back.
>>>> http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-
>>>> rails-mongrel-apache-capistrano-and-you/
>>>>
>>>> I have 4 mongrels in my cluster.
>>>>
>>>> Things work fine for periods of time but after several hours of
>>>> inactivity
>>>> (I think 8 hours or so) I experience oddness where only 1 of the 4
>>>> mongrels
>>>> is properly
>>>> responding. I end up getting a "500 internal server error" 3 out
>>>> of 4
>>>> requests as they round robin from mongrel to mongrel. There is
>>>> nothing in
>>>> the production
>>>> log file nor in the mongrel log. I've reproduced this problem on
>>>> my staging
>>>> box as well as my production box.
>>>>
>>>> The last time I reproduced the problem I decided to run "top" and
>>>> see what's
>>>> going on when I hit the server.
>>>> Mongrel does receive every request but mysql is only active on the
>>>> 1 request
>>>> that works. In the other mongrels it never spikes up in CPU usage.
>>>>
>>>> Looking at the mysql process list revealed that all of the
>>>> processes had
>>>> received the "sleep" command but one of the processes is still
>>>> working properly. I've played with connection timeouts other than
>>>> to set the
>>>> timeout in my application's environment
>>>> (ActiveRecord::Base.verification_timeout = 14400) as well as the
>>>> mysql
>>>> interactive_timeout variable but it seems that all the mongrels
>>>> should work
>>>> or they shouldn't. The fact that 1 out of 4 always works is rather
>>>> puzzling
>>>> to me.
>>>>
>>>> Trying a 'killall -USR1 mongrel_rails" to turn debug on simply
>>>> killed the 4
>>>> threads running mongrel. So now I'm running the cluster in debug
>>>> mode and am
>>>> going to just let it sit there for several hours until it happens
>>>> again and
>>>> hopefully get some idea of where the breakdown is happening. I
>>>> still think
>>>> it has to be a mysql connection timeout but again, the fact that 1
>>>> of the 4
>>>> always works doesn't lend credence to the timeout theory.
>>>>
>>>> Has anyone experienced this phenomenon themselves?
>>>>
>>>> Thanks for any tips/pointers and thanks Zed for all your hard work
>>>> with
>>>> mongrel.
>>>>
>>>>
>>>>
>>>> -Michaelhttp://javathehutt.blogspot.com
>>>>
>>>>
>>>> _______________________________________________
>>>> Mongrel-users mailing list
>>>> Mongrel-users at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>>>
>>>>
>>> _______________________________________________
>>> Mongrel-users mailing list
>>> Mongrel-users at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>> _______________________________________________
>> Mongrel-users mailing list
>> Mongrel-users at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-users
>>
>
> -- Ezra Zygmuntowicz 
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users



More information about the Mongrel-users mailing list