[Mongrel] Mongrel Hanging (Apache, mod_proxy, Mongrel Cluster, Rails)

Justin Erickson justin at magpieti.com
Mon Jun 29 11:28:19 EDT 2009


We have been having a problem with our mongrel servers hanging at
seemingly random times.

The problem can be most often reproduced by accessing a page, leaving
the browser open for some time (60+ min) then making another request.
The browser will receive a "Proxy Error" from Apache, If we look at
mongrel via "ps aux" it says

"mongrel_rails [8000/1/253]: handling 127.0.0.1: GET /"

Using strace we pulled this:

14:04:15 poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0
14:04:15 write(12, "\1\0\0\0\t", 5)     = 5
14:04:15 read(12,

SEVER IS LOCKED HERE...

Then 15 min later...

14:04:15 read(12, 0x8eab758, 16384)     = -1 ETIMEDOUT (Connection timed out)
14:19:40 shutdown(12, 2 /* send and receive */) = -1 ENOTCONN
(Transport endpoint is not connected)
14:19:40 close(12)                      = 0
14:19:40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
14:19:40 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 12
14:19:40 fcntl64(12, F_SETFL, O_RDONLY) = 0
14:19:40 fcntl64(12, F_GETFL)           = 0x2 (flags O_RDWR)
14:19:40 connect(12, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr("XXX.XXX.XXX.XXX")}, 16) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0", 8) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0", 8) = 0
14:19:40 setsockopt(12, SOL_IP, IP_TOS, [8], 4) = 0
14:19:40 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
14:19:40 read(12, "F\0\0\0\n5.0.60sp1-enterprise-gpl\0\3679"..., 16384) = 74

We have tried all the results we can find on the internet regarding
"Mongrel Hanging"

For Example:

We are NOT using ruby log rolling
We ARE using the c compiled mysql Adapter
We have tried setting "ActiveRecord::Base.verification_timeout = 14400"
The timeout setting on MySQL are default (interactive_timout,
wait_timeout) = 28800

We have tried the TCP keepalive setting found in here:
http://wooga.drbacchus.com/rails-and-mysql-timeouts

We have tried setting an apache setting:
#Fix for Apache bug 39499
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

We have even tried many mysql_adapter.rb patches that "correctly"
reconnect the connection on an error.

None of these have made any difference.

We are currently using:
MySQL 5.0.60sp1
Rails 2.0.2
Mongrel 1.1.5

Thanks you very much for any help!

Justin


More information about the Mongrel-users mailing list