Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU when idle?
normalperson at yhbt.net
Tue Jul 24 00:21:40 UTC 2012
Ilya Maykov <ivmaykov at gmail.com> wrote:
> Hi Eric,
> Sorry for the delayed response. I've added inline answers to your
> questions. We've since resolved this issue by disabling keepalive in
> our rainbows config. So, this probably had to do with the keepalive
> implementation either in Rainbows itself or in the base Unicorn code.
> Answers to your other questions are inlined below.
Thank you very much for the follow-up (I wish more folks would do this
If Rainbows! is using EventMachine, it'll use the
EM.set_comm_inactivity_timeout method in EventMachine.
> On Tue, Jun 19, 2012 at 10:54 AM, Eric Wong <normalperson at yhbt.net> wrote:
> > Ilya Maykov <ivmaykov at gmail.com> wrote:
> > Can you also try "keepalive_timeout 0" to disable keepalive? (EM
> > handles it internally, but I'm not sure how well)
> This turned out to be the problem. Disabling keepalive got rid of the
> CPU pegging. Surprisingly, it also made our average latency drop from
> about 50 ms to about 20 ms per request, even though every request now
> has to negotiate a connection handshake. So, we're just going to keep
> it disabled for now. The bug may be inside the keepalive code in
> Rainbows or Unicorn (not familiar with the codebase so not sure where
> that code lives).
Are you setting Content-Length or "Transfer-Encoding: chunked" in
responses? Rack::ContentLength or Rack::Chunked middleware might need
to be loaded if your framework doesn't already include it.
Lack of these headers may confuse clients, even... Where did you
measure the 50 -> 20ms latency drop, from the client?
About the latency drop:
Was :tcp_cork enabled in your listen directive? I wonder if there's
some bad interaction with :tcp_cork + EM which might explain the
latency (but not the CPU usage). Disabling keepalive would force data
out immediately and avoid any bad effects of :tcp_cork.j
Can you also try listen() with :tcp_defer_accept => 0? That might
help if you're accept()-ing many connections at once.
Anyways, the TCP connection handshake is very fast on LANs/localhost,
but hurts on high-latency connections (I seem to remember mainstream web
browsers double simultaneous requests to compensate for lack of
Which version of EM are you using? The keepalive implementation for
EM+Rainbows! is entirely handled by EM.
More information about the rainbows-talk