select(): Interrupted system call from curb when stopping unicorn

Graham Bleach graham at darkskills.org.uk
Wed Nov 7 16:42:37 UTC 2012


Hi,

We've just migrated one of our rails applications from nginx/passenger
running on REE 1.8.7 Ubuntu 8.04 to unicorn running on MRI 1.9.3 on
Ubuntu 10.04. The app makes a number of calls to internal services
using curb.

Our deployment script stops unicorn by sending SIGQUIT to the unicorn
master, sleeps for a few seconds to ensure that HAProxy has taken the
node out of service and then starts unicorn again.

However, since the migration, we've started seeing a few errors on
each deploy, when the workers receive the QUIT from the master:

 ERROR (  XXX) {"exception":{"class":"RuntimeError","message":"select():
Interrupted system
call","backtrace":["/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform'","/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform'","/path/bundle/ruby/1.9.1/gems/songkick-transport-0.1.4/lib/songkick/transport/curb.rb:50:in
...

If we're reading this correctly, curb is inside a select() call, gets
interrupted by the QUIT signal and doesn't retry the select() and the
process terminates, causing an error for the unfortunate end user.

Our options for stopping our users seeing error pages seem to be:

1) Patch curb to retry the select() if errno = EINTR as per
https://github.com/taf2/curb/issues/117

We're not confident enough in our ability to do this properly, but
it's probably the most correct way to solve this.

2) Do something to make sure that requests have all completed before
sending the QUIT.

For example, we could change our deployment to make the health check
url that HAProxy polls start returning a non-200 return code, wait
until all connections have completed and then send the QUIT.

The main issue with this is that short of polling the HAProxy status
page, we're not entirely sure how to verify that all connections have
completed.

Does anyone have any suggested solutions or comments?

Regards,
Graham


More information about the mongrel-unicorn mailing list