Unicorn and HAProxy, 500 Internal errors after checks

Pierre oct at fotopedia.com
Thu Dec 2 03:38:38 EST 2010


On Wed, Dec 1, 2010 at 8:58 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Pierre <oct at fotopedia.com> wrote:
>> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> > Hi Pierre, HAProxy should be configured to send proper HTTP checks and
>> > not just TCP connection checks, the problem will go away then.
>> I understood this could be fixed this way and we will probably do that
>> soon. However, I think this is also the responsibility of Unicorn not
>> to reply anything when there's no request or at least log the error
>> somewhere :)
> I'm not sure how Unicorn is actually replying to anything, does HAProxy
> write *anything* to the socket?

It does. To reproduce the behavior, start a unicorn, start a sniffing
tool (ngrep for example), connect to the unicorn and disconnect

Connecting endpoint:

[08:22][virtual] root at infrabox:~# telnet localhost 2002
Connected to infrabox.virtual.ftnz.net.
Escape character is '^]'.
telnet> quit
Connection closed.

Sniffing Console:

[08:22][virtual] root at infrabox:~# ngrep "" port 2002 -d lo
interface: lo (
filter: (ip or ip6) and ( port 2002 )
T -> [AFP]
  HTTP/1.1 500 Internal Server Error....

Further packet inspection reveals that the 500 Internal Server Error
is sent after the telnet has sent a FIN packet which means it will
just be discarded. I understand logging it could be a DoS vector, but
you must admit that seing 500's from the server without any obvious
reason can be frightening at first.

Re HAProxy, and to complete Laurence reply from my point of view,

> If you serve large responses that can't fit in kernel socket buffers,
> then Unicorn will get stuck writing out to a client that isn't reading
> fast enough.

That can definitely be a problem in that case. To work around that we
use a cache in the stack above the lower level of HAProxy (we have two
other levels of HAProxy in the stack).

Pierre <oct at fotopedia.com>
Server Shepherd at http://www.fotopedia.com/

More information about the mongrel-unicorn mailing list