Page request roundtrip time increases substantially after a bit of use
normalperson at yhbt.net
Mon Jan 24 16:54:40 EST 2011
chris mckenzie <kristopolous at yahoo.com> wrote:
> Every now and then, however, it will be about 2.5 seconds. This will then be
> followed by a bunch of the snappy millisecond level transaction times. This is
That is probably GC, but...
> Everything changes about 5-10 minutes into things.
> Then every transaction takes about 2-4 seconds. Static files that are 10 bytes
> in size, 2-4 seconds. Ruby code to emit "Hello World"? 2-4 seconds. Every
> request. Still using just 1 browser.
> After I exit all browsers and then do a netstat on the client machine to see
> that the connections have closed, I can then do a curl command for a static
> file; again 2-4 seconds.
> On the machine running rainbows if I do a netstat, I get this:
> tcp 1 0 10.10.192.12:7788 10.10.131.165:17443 CLOSE_WAIT
> tcp 1 0 10.10.192.12:7788 10.10.131.165:17352 CLOSE_WAIT
> tcp 1 196 10.10.192.12:7788 10.10.131.165:17317 CLOSE_WAIT
> tcp 1 196 10.10.192.12:7788 10.10.131.165:17310 CLOSE_WAIT
^ Strange that Send-Q is 1 across all those connections..
Did you see the machine/connection that ran curl in there? How does
hitting Rainbows! from localhost work?
Are you dropping packets?
> I think that somewhere in the ruby stack, the connections are not
I would do an lsof on some of the worker processes to see if they
still think a connection is open. Do you see the connection from
curl in netstat?
> increase my worker_process count and prolong the long poll, then yes, I'll
> survive for 15 minutes instead of 5; but the problem will still eventually occur
> and I will hit the wall.
> I have yet to try to test unicorn or zbatery for this style of
> solution because I need the keep-alive; and although I know that
> unicorn put in the keep-alive support for rainbows, I haven't really
> taken the time necessary to know how to invoke it. If you think this
> would be instructive, I'd be happy to do so.
The Unicorn parser supports keepalive for Rainbows!, but Unicorn itself
does not. Rainbows! "use :Base" (the default) is basically the same
thing with Unicorn+keepalive. Zbatery doesn't do anything differently
for managing client connections than Rainbows!
> As far as my ruby set-up, I'm using ruby1.8 and I have the ThreadSpawn
> model. We are running on a modern version of Ubuntu without any
> serious customization.
What's your Rainbows! keepalive_timeout set to? The default is 5s.
What's your worker_connections setting?
> If you think that another configuration would do the trick or if you
> know how to squash this bug, it would be very helpful. This problem
> has become of great concern for us. We love rainbows and all that it
> is. Thanks for the project and keep up the good work.
Do you notice your Rainbows! worker processes growing in memory usage
over time? Which Ruby 1.8 patch level? Are you running any custom
GC? If your app supports it, try Ruby 1.9.2-p136, too.
Do you always set Content-Length or "Transfer-Encoding: chunked" in your
app responses? Missing/inaccurate values will throw off clients if you
have keepalive, otherwise I've never seen anything like the problem you
More information about the rainbows-talk