Socket errors and nginx 502 gateway errors

Eric Wong normalperson at yhbt.net
Sat Apr 28 21:59:04 UTC 2012


Pico Aeterna <flippedootninja at gmail.com> wrote:
> Hi,
> 
> We've been getting reports that our users have been getting random 502
> Errors.  After investigating I noticed 'upstream prematurely closed
> connection while reading response header from upstream' in my nginx
> error logs.  From using strace it appears that there's numerous socket
> SO_ERROR..  I temporarily switched out the domain sockets with TCP,
> however am seeing the same result/issue.
> 
> Strace and Nginx error log below:

Can you show us an strace of a unicorn worker, too?  (use
"worker_processes 1" when debugging this)

Also, the unicorn stderr log can be helpful.

> 7366  epoll_ctl(16, EPOLL_CTL_ADD, 89, {EPOLLIN|EPOLLOUT|EPOLLET,
> {u32=2473356496, u64=46963645770960}}) = 0
> 7366  connect(89, {sa_family=AF_FILE,
> path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0
> 7366  getsockopt(89, SOL_SOCKET, SO_ERROR, [1422340301959200768], [4]) = 0
> 7366  writev(89, [{"GET /natalieliao HTTP/1.0\r\nX-Forwarded-For:
> 94.224.228.239, 94.224.228.239\r\nHost"..., 1084}], 1) = 1084

Can you pass "-s 2048" (or larger value) when stracing?  It'll be
helpful to know what else is in the HTTP header.

> Load balancer (HaProxy) => nginx => unicorn

Does hitting unicorn directly with curl (TCP) give an expected result?
(You can use the strace output above to have curl mimic what nginx
sends, or even use printf | socat which works with Unix sockets).

> Relevant sysctl.conf changes

Probably not relevant since you switched to Unix sockets, but
consider default sysctl values while debugging (especially tcp_tw_*)

> net.ipv4.ip_local_port_range = 1024 65535
> net.ipv4.tcp_fin_timeout = 15
> #timewait sockets
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_tw_recycle = 1


More information about the mongrel-unicorn mailing list