Rainbows! + EventMachine + Sinatra::Synchrony == pegged CPU when idle?

[Eric Wong]

Ilya Maykov <ivmaykov at gmail.com> wrote:
> Hi Eric,
> 
> Sorry for the delayed response. I've added inline answers to your
> questions. We've since resolved this issue by disabling keepalive in
> our rainbows config. So, this probably had to do with the keepalive
> implementation either in Rainbows itself or in the base Unicorn code.
> Answers to your other questions are inlined below.

Thank you very much for the follow-up (I wish more folks would do this
:)

If Rainbows! is using EventMachine, it'll use the
EM.set_comm_inactivity_timeout method in EventMachine.

> On Tue, Jun 19, 2012 at 10:54 AM, Eric Wong <normalperson at yhbt.net> wrote:
> > Ilya Maykov <ivmaykov at gmail.com> wrote:
> > Can you also try "keepalive_timeout 0" to disable keepalive?  (EM
> > handles it internally, but I'm not sure how well)
> >
> 
> This turned out to be the problem. Disabling keepalive got rid of the
> CPU pegging. Surprisingly, it also made our average latency drop from
> about 50 ms to about 20 ms per request, even though every request now
> has to negotiate a connection handshake. So, we're just going to keep
> it disabled for now. The bug may be inside the keepalive code in
> Rainbows or Unicorn (not familiar with the codebase so not sure where
> that code lives).

Are you setting Content-Length or "Transfer-Encoding: chunked" in
responses?  Rack::ContentLength or Rack::Chunked middleware might need
to be loaded if your framework doesn't already include it.

Lack of these headers may confuse clients, even...  Where did you
measure the 50 -> 20ms latency drop, from the client?

About the latency drop:

  Was :tcp_cork enabled in your listen directive?  I wonder if there's
  some bad interaction with :tcp_cork + EM which might explain the
  latency (but not the CPU usage).  Disabling keepalive would force data
  out immediately and avoid any bad effects of :tcp_cork.j

  Can you also try listen() with :tcp_defer_accept => 0?  That might
  help if you're accept()-ing many connections at once.


Anyways, the TCP connection handshake is very fast on LANs/localhost,
but hurts on high-latency connections (I seem to remember mainstream web
browsers double simultaneous requests to compensate for lack of
keepalive).

Which version of EM are you using?  The keepalive implementation for
EM+Rainbows! is entirely handled by EM.