Socket errors and nginx 502 gateway errors

Eric Wong normalperson at
Sun Apr 29 19:18:25 UTC 2012

Pico Aeterna <flippedootninja at> wrote:
> reaped #<Process::Status: pid=9103,signaled(SIGIOT=6)> worker=9
> /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:172:
> [BUG] Segmentation fault

OK, something (most likely your app) is causing Ruby to segfault.
This is completely unrelated to nginx, so it'll be easier to hit
unicorn directly to diagnose this.

Since you're new to debugging, I'll try to walk you through dealing
with core dumps:

* Enable core dumps in the same shell process before starting unicorn
  (ulimit -c unlimited && unicorn -c /path/to/config ...)

* On Linux, /proc/sys/kernel/core_* sysctls control where your core dumps
  are created.

And when you get a core dump (a "core" or core.$PID file), you can run
gdb on it:

   gdb /path/to/ruby /path/to/core

Make sure you're using the same ruby binary that unicorn is using.
Ruby should be compiled with debugging symbols ('-g' or '-ggdb') by
default, but if not, you'll probably want to recompile.

> 7369  writev(31, [{"GET
> /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06
> HTTP/1.0\r\nX-Forwarded
> -For:,\r\nHost:\r\nConnection:
> close\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64)
> AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5
> Safari/534.55.3\r\nAccept: */*\r\nReferer:
>\r\nAccept-Language: pl-PL\r\nAccept-Encoding:
> gzip, deflate\r\nCookie:
> __utma=78941577.1641012066.1319789648.1325094074.1325949868.3;
> __gads=ID=269c5cea8f09aea0:T=1319789648:S=ALNI_MaqwdMaIFS5I3FeqLnv86oivdVDuA;
> __qca=P0-1390452860-1319789648200\r\n\r\n", 745}], 1) = 7457369

> Using the following I was able to get a 200 response
> req='GET / HTTP/1.1\r\nHost:\r\n\r\n'
> printf "$req" | socat - UNIX:/var/www/domain/shared/sockets/unicorn.sock

OK, I think your printf request needs to more closely match what nginx
is sending to trigger the segfault.  I would add/remove
headers/parameters you see above until things start segfaulting.

With socat, you may also want to wait a bit for the response for
slower endpoints:

  (printf "$req"; sleep $SECONDS) | socat - ...

Otherwise socat will disconnect immediately after its written to
the socket without waiting for a response.

(a better, but complicated way would be to setup a FIFO as the
 integration tests in unicorn do).

If you're familiar with Rack::Mock*, you may also want to isolate this
to just a Rack application.  It's likely the segfault can be triggered
for your app without unicorn, too.

> Sorry but i'm new to debugging Unicorn.  Hopefully some of this is helpful.

It is helpful.  Debugging unicorn isn't different than debugging
anything else.  The most important is to understand the problem, and
that's done by isolating variables to get a small, reproducible test

More information about the mongrel-unicorn mailing list