502s with Nginx, Unicorn, and Unix Domain Sockets
normalperson at yhbt.net
Sat Sep 19 18:08:56 EDT 2009
Tom Preston-Werner <tom at github.com> wrote:
> On Thu, Sep 17, 2009 at 11:48 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > At what request rates were you running into this? ??Also how large are
> > your responses? ??It could be the listen() backlog overflowing if Unicorn
> > isn't logging anything.
> I was hitting the 502s at about 1300 req/sec and 80% CPU utilization.
> Response size was only a few bytes + headers. I was just testing a
> very simple string response from our Rails app to make sure our setup
> could tolerate very high request rates.
Yup, as I suspected: your UNIX socket setup was maxing out right around
where your TCP setup was maxing out. TCP is just better at
handling/recovering from errors.
> > Does increasing the listen :backlog parameter work? ??Default is 1024
> > (which is pretty high already), maybe try a higher number along with the
> > net.core.netdev_max_backlog sysctl.
> This was the first thing I tried after getting your response, and it
> seems that upping the :backlog to 2048 solves the 502 problem! I'm now
> able to get 1500 req/sec out of Unicorn/UNIX (as opposed to 1350
> req/sec with the TCP/HAProxy setup). I'm quite satisfied with this
> result, and I think this is how we'll end up deploying the app.
Good to know it worked!
However, I do hesitate to recommend a large listen() backlog for
production. It can impede with monitoring/failover/load-balancing in
multi-server setups even if it looks good on benchmarks.
I'll make a separate call-for-testing mailing list related to
this subject in a bit...
> Thanks for your help, and I'll try to keep you updated on how our
> installation performs and if I see any strange behavior under normal
No problem, thanks for the feedback! It's great to know people
actually use it.
More information about the mongrel-unicorn