funky process tree + stillborn masters
normalperson at yhbt.net
Thu Apr 8 19:55:48 EDT 2010
Jamie Wilkinson <jamie at tramchase.com> wrote:
> Since upgrading bundler (but applying the RUBYOPT path fixes) we've
> started experiencing intermittent, difficult-to-isolate USR2 restart
> After a USR2 signal our process tree winds up looking like this, with
> several master-esque processes listed as children (but without the
> "worker[N]" label):
> app 14402 4.4 0.8 199612 70264 ? S 14:07 0:04 unicorn_rails master -c config/unicorn.rb -E production -D
> app 14433 0.0 0.8 204540 68504 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> app 14435 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> app 14438 0.0 0.8 199748 65840 ? S 14:07 0:00 \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app 14440 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> app 14442 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> app 14445 0.0 0.8 199760 65840 ? S 14:07 0:00 \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app 14447 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> app 14449 0.0 0.8 204780 69272 ? Sl 14:07 0:00 \_ unicorn_rails worker -c config/unicorn.rb -E production -D
> Sending another USR2 signal will bring a new master into the mix as a
> child, spins up a single child worker of its own (which also resembles
> the "/usr/bin/ruby1.8" master-esque processes), and then fails to
Odd, if I had to guess PIDs 14438 and 14445 are actually worker and
worker based on the PIDs relative to other workers. So the new
master died right away, which really should've been logged...
> Further USR2 restarts will obviously do nothing, and we're forced to
> either kill -9 the stillborn master or cold-restart all of the
> unicorns. Nothing out of the ordinary is dumped to stderr or stdout
Anything in your before_fork/after_fork hooks? Since it looks like
you're on a Linux system, can you strace the master while you send
it a USR2 and see if anything strange happens?
Also, can you strace the weird looking processes above and see if
they're doing anything?
> Starting unicorns fresh produces a nice process list where every child
> is listed cleanly as "unicorn_rails worker[N]"
> We only have this issue in one of our applications, on a machine that
> has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've
> also only noticed this since upgrading to bundler from bundler08
I assume you're using regular "unicorn" to run your Sinatra apps and not
"unicorn_rails". I made some largish cleanups to both for the 0.97.0
release and and perhaps some bugs slipped into the "_rails" variant.
Can you try regular "unicorn" with a config.ru for Rails? I've stolen
this from the Rainbows! FAQ (http://rainbows.rubyforge.org/FAQ.html):
For Rails 2.3.x and later, the following config.ru will work for you:
ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
For older versions of Rails, the following config.ru will work:
ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
require 'unicorn/app/old_rails/static' # not needed with Unicorn 0.95+
One thing to watch out for is that RAILS_ENV will not be set in the
environment for you, thus we set it to match RACK_ENV.
> Are the goofy worker processes in the process tree a real problem, or
> just a red herring?
Not sure if it's a problem, but with Bundler I assume Rack itself is a
bundled dependency, but you're starting unicorn_rails out of
/usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
use the bundled Rack). Can you ensure your unbundled Rack is the same
version as the bundled one to be on the safe side?
I've yet to try bundler 0.9 (and have barely used earlier), but you can
also try bundling Unicorn and using the bundled bin/unicorn(_rails)
launchers instead to ensure a consistent environment.
Unicorn currently ends up (auto)loading "rack/utils" before the
application is loaded, maybe it could (auto)load it after the app is
loaded for preload_app users.
More information about the mongrel-unicorn