502 bad gateway on nginx with recv() failed

Eric Wong normalperson at yhbt.net
Fri Oct 22 17:14:46 EDT 2010

Naresh V <nareshov at gmail.com> wrote:
> Hi,
> I'm serving the puppetmaster application with its config.ru through
> unicorn - proxied by nginx.
> I'm using unix sockets, 4 workers, and 2048 backlog.
> The clients - after their typical "puppet run" - send back a report to
> the master in YAML.
> Some clients whose reports tend to be large (close to 2mb) get a 502
> bad gateway error and error out.
> nginx log:
> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
> Connection reset by peer) while reading response header from upstream,
> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
> HTTP/1.1", upstream:
> "http://unix:/tmp/.sock:/production/report/nagios", host:
> "puppet:8140"

Hi Naresh, do you see anything in the Unicorn stderr log file?

Is the 2mb report part of the response or request?  Unicorn should
have no problems accepting large requests (Rainbows! defaults the
client_max_body_size to 1mb, just like nginx).

It could be Unicorn's internal (default 60s) timeout kicking
in because puppet is slowly reading/generating the 2mb body.

Also, which version of Unicorn and nginx is this?

> I was getting the same thing earlier when I had unicorn listening on
> TCP sockets instead of UNIX sockets. And I had a lot of connections in
> tcp        0      0    
>      TIME_WAIT   -
> tcp        0      0    
>      TIME_WAIT   -
> Fluctuating all the way from 20 to 800. A quick restart of nginx
> tended to bring the number down.

Having many TIME_WAIT sockets is normal and expected when you're
starting/stopping lots of TCP connections.  It's nothing to worry about
unless you get several thousands of requests/second, and then you should
apply tcp_tw_reuse/tcp_tw_recycle as mentioned in
http://unicorn.bogomips.org/TUNING.html (or switch to UNIX domain
sockets and use nginx for keepalive).

Eric Wong

More information about the mongrel-unicorn mailing list