funky process tree + stillborn masters

Eric Wong normalperson at
Tue Apr 13 01:24:10 EDT 2010

Jamie Wilkinson <jamie at> wrote:
> I've also produced straces of the *original* master during USR2
> restarts, both a success trace and a failure trace.  Here's a tarball
> with both complete traces as well as filtered/grepp'd ones:
> I've also found that kill -9'ing the 1st worker of the new orphaned
> master allows it to continue operation as normal (spinning up workers
> and taking control from the original master) -- suggesting something
> is up with just that first worker (!). I'm going to keep noodling with
> before_/after_fork strategies.

Hi Jamie, your successful straces have this oddity in them:

  poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 1 ([{fd=7, revents=POLLIN|POLLHUP}
  read(7, ""..., 8192)                    = 0
  shutdown(7, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not
  close(7)                                = 0

> Eric Wong <normalperson at> wrote:
> > Anything in your before_fork/after_fork hooks?  Since it looks like
> > you're on a Linux system, can you strace the master while you send
> > it a USR2 and see if anything strange happens?
> The only real contents of our before_hook is a
> send-QUIT-on-first-worker, which I swapped out for the default SIGTTOU
> behavior. No change.

It looks like you should be disconnecting/reconnecting whatever is using
fd=7 in the before_fork/after_fork hooks respectively.

Unicorn (and mainline Ruby) do not use the poll() system call (which is
why that strace raised red flags), so there's another C extension
calling poll() in your master process (not good).  I remember one (or
all of) the Ruby Postgres libraries using poll() internally, and there
are likely other things that use poll() in Ruby as well.

Perhaps something like this in the config (assuming you're using
Rails ActiveRecord) (also see

  before_fork do |server,worker|
    defined?(ActiveRecord::Base) and

  after_fork do |server,worker|
    defined?(ActiveRecord::Base) and

Eric Wong

More information about the mongrel-unicorn mailing list