unicorn 1.1.x never-ending listen loop IOError exceptions

Eric Wong normalperson at yhbt.net
Tue Mar 22 17:39:05 EDT 2011


David Nghiem <david at zendesk.com> wrote:
> We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
> 2011.02.  We're using UNIX sockets instead of TCP sockets.
> 
> Once in a while, one worker will throw exceptions in a loop, taking up
> lots of CPU and log space.  Killing the worker removes the problem.
> Other workers are not affected.
> 
> Here's the exception:
> 
> E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
> loop exception #<IOError: closed stream>.
> E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
> /data/zendesk/releases/alpha/vendor/bundle/ruby/1.8/gems/unicorn-1.1.5/lib/unicorn.rb:732:in
> `select'
> 
> I don't get any other exceptions.  I've found other people have this
> "IOError: closed stream" error coupled with "Errno::EMFILE" but we
> have plenty of file-descriptors left.

No Errno::EBADF errors at all?

> Here's an lsof dump of the worker:
> 
> COMMAND   PID    USER   FD   TYPE             DEVICE        SIZE    NODE NAME

<snip>  It looks like log reopening failed, below:

> ruby    15209 zendesk    1w   REG              252,0           0
> 3015360 /data/zendesk/log/unicorn.stdout.log
> ruby    15209 zendesk    2w   REG              252,0       24585
> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> ruby    15209 zendesk    3w   REG              252,0 61617590073
> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)

Permissions issue during log rotation?  Are you doing "killall -USR1
unicorn" by any chance or sending a bunch of USR1 signals faster than
unicorn can process them?

Try just sending one USR1 to the master process and letting the master
signal the workers.

-- 
Eric Wong


More information about the mongrel-unicorn mailing list