502s with Nginx, Unicorn, and Unix Domain Sockets

Eric Wong normalperson at yhbt.net
Sat Sep 19 18:08:56 EDT 2009


Tom Preston-Werner <tom at github.com> wrote:
> On Thu, Sep 17, 2009 at 11:48 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > At what request rates were you running into this? ??Also how large are
> > your responses? ??It could be the listen() backlog overflowing if Unicorn
> > isn't logging anything.
> 
> I was hitting the 502s at about 1300 req/sec and 80% CPU utilization.
> Response size was only a few bytes + headers. I was just testing a
> very simple string response from our Rails app to make sure our setup
> could tolerate very high request rates.

Yup, as I suspected: your UNIX socket setup was maxing out right around
where your TCP setup was maxing out.  TCP is just better at
handling/recovering from errors.

> > Does increasing the listen :backlog parameter work? ??Default is 1024
> > (which is pretty high already), maybe try a higher number along with the
> > net.core.netdev_max_backlog sysctl.
> 
> This was the first thing I tried after getting your response, and it
> seems that upping the :backlog to 2048 solves the 502 problem! I'm now
> able to get 1500 req/sec out of Unicorn/UNIX (as opposed to 1350
> req/sec with the TCP/HAProxy setup). I'm quite satisfied with this
> result, and I think this is how we'll end up deploying the app.

Good to know it worked!

However, I do hesitate to recommend a large listen() backlog for
production.  It can impede with monitoring/failover/load-balancing in
multi-server setups even if it looks good on benchmarks.

I'll make a separate call-for-testing mailing list related to
this subject in a bit...

> Thanks for your help, and I'll try to keep you updated on how our
> installation performs and if I see any strange behavior under normal
> traffic.

No problem, thanks for the feedback!  It's great to know people
actually use it.

-- 
Eric Wong


More information about the mongrel-unicorn mailing list