unicorn 1.1.x never-ending listen loop IOError exceptions
normalperson at yhbt.net
Tue Mar 22 17:39:05 EDT 2011
David Nghiem <david at zendesk.com> wrote:
> We run unicorn 1.1.5 behind nginx under Ruby Enterprise Edition
> 2011.02. We're using UNIX sockets instead of TCP sockets.
> Once in a while, one worker will throw exceptions in a loop, taking up
> lots of CPU and log space. Killing the worker removes the problem.
> Other workers are not affected.
> Here's the exception:
> E, [2011-01-10T10:13:13.518795 #15209] ERROR -- : Unhandled listen
> loop exception #<IOError: closed stream>.
> E, [2011-01-10T10:13:13.518881 #15209] ERROR -- :
> I don't get any other exceptions. I've found other people have this
> "IOError: closed stream" error coupled with "Errno::EMFILE" but we
> have plenty of file-descriptors left.
No Errno::EBADF errors at all?
> Here's an lsof dump of the worker:
> COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
<snip> It looks like log reopening failed, below:
> ruby 15209 zendesk 1w REG 252,0 0
> 3015360 /data/zendesk/log/unicorn.stdout.log
> ruby 15209 zendesk 2w REG 252,0 24585
> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> ruby 15209 zendesk 3w REG 252,0 61617590073
> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
Permissions issue during log rotation? Are you doing "killall -USR1
unicorn" by any chance or sending a bunch of USR1 signals faster than
unicorn can process them?
Try just sending one USR1 to the master process and letting the master
signal the workers.
More information about the mongrel-unicorn