From will at thesuperformula.com  Mon Apr  5 16:05:15 2010
From: will at thesuperformula.com (William N Prater III)
Date: Mon, 5 Apr 2010 13:05:15 -0700
Subject: Nginx Socket with Multiple Unicorn instances
Message-ID: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>

Hello,

Is it possible to run one Nginx socket that is used by multiple master Unicorn processes for different applications?  Or does one need to setup a new upstream for each?

TIA



From ibc at aliax.net  Mon Apr  5 16:34:20 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Mon, 5 Apr 2010 22:34:20 +0200
Subject: Nginx Socket with Multiple Unicorn instances
In-Reply-To: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>
References: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>
Message-ID: <i2scc1f582e1004051334g3e893c59nb317841e6457b5b1@mail.gmail.com>

2010/4/5 William N Prater III <will at thesuperformula.com>:
> Hello,
>
> Is it possible to run one Nginx socket that is used by multiple master Unicorn processes for different applications? ?Or does one need to setup a new upstream for each?

Do you mean different Unicorn servers listening into the same socket?
it doesn't make sense IMHO.


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From will at thesuperformula.com  Mon Apr  5 16:40:32 2010
From: will at thesuperformula.com (William N Prater III)
Date: Mon, 5 Apr 2010 13:40:32 -0700
Subject: Nginx Socket with Multiple Unicorn instances
In-Reply-To: <i2scc1f582e1004051334g3e893c59nb317841e6457b5b1@mail.gmail.com>
References: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>
	<i2scc1f582e1004051334g3e893c59nb317841e6457b5b1@mail.gmail.com>
Message-ID: <70DB86EC-2230-4C2B-96B7-CB6CF1DA3674@thesuperformula.com>

On Apr 5, 2010, at 1:34 PM, I?aki Baz Castillo wrote:

> 2010/4/5 William N Prater III <will at thesuperformula.com>:
>> Hello,
>> 
>> Is it possible to run one Nginx socket that is used by multiple master Unicorn processes for different applications?  Or does one need to setup a new upstream for each?
> 
> Do you mean different Unicorn servers listening into the same socket?
> it doesn't make sense IMHO.

It may not make any sense.  Im not sure how Nginx handles the vhost and proxies.  I was hoping to find an easier way to deploy new apps.  Otherwise I have to edit a few config file each time. 


From normalperson at yhbt.net  Mon Apr  5 19:29:27 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 5 Apr 2010 16:29:27 -0700
Subject: Nginx Socket with Multiple Unicorn instances
In-Reply-To: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>
References: <590E25C9-7434-4EA8-862E-EBB1C03E9B23@thesuperformula.com>
Message-ID: <20100405232927.GA9966@dcvr.yhbt.net>

William N Prater III <will at thesuperformula.com> wrote:
> Hello,
> 
> Is it possible to run one Nginx socket that is used by multiple master
> Unicorn processes for different applications?  Or does one need to
> setup a new upstream for each?

Hi William,

Nginx socket?  It's ambiguous what you mean by that, but since you
mention upstreams I'll assume you mean UNIX domain sockets bound by
Unicorn that nginx connects to.

Generally, you'll have to create a new upstream for each, but nginx can
listen on the same HTTP socket (usually port 80) and use virtual hosts.

So I assume you'll do something like this to talk to separate Unicorn
masters:

  upstream unicorn_a {
    server unix:/tmp/.a fail_timeout=0;
  }
  upstream unicorn_b {
    server unix:/tmp/.b fail_timeout=0;
  }
  server {
    # catch all, this serves an error to people that connect w/o a host name
    listen 80 default deferred;
    server_name _;
    root /srv/default;
  }
  server {
    server_name a.example.com;
    root /srv/a;
    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      if (!-f $request_filename) {
        proxy_pass http://unicorn_a;
      }
    }
  }
  server {
    server_name b.example.com;
    root /srv/b;
    location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      if (!-f $request_filename) {
        proxy_pass http://unicorn_b;
      }
    }
  }

However, if you _really_ trust your apps and frameworks to work
correctly in the same process, you can run multiple apps within the same
Rack server, you can actually have virtual hosts this way in Rack:

--------------------------- 8< ----------------------------
use Rack::ContentLength
use Rack::ContentType, 'text/plain'
# more global middlewares can go here

# virtual hosts:
map "http://a.example.com/" do
  run lambda { |env| [ 200, {}, [ "Hello A\n" ] ] }
end
map "http://b.example.com/" do
  run lambda { |env| [ 200, {}, [ "Hello B\n" ] ] }
end
--------------------------- 8< ----------------------------

However, if your apps (or dependent libraries/frameworks) share any
global resources in an unintended or unsafe manner, then you won't be
able to do this.  The above example is of course highly trivial.

-- 
Eric Wong

From jamie at tramchase.com  Wed Apr  7 19:28:41 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Wed, 7 Apr 2010 16:28:41 -0700
Subject: bundler's RUBYOPT prefixing has been fixed
Message-ID: <8FC34B23-5994-41CC-B5AF-7198EF06909E@tramchase.com>

I filed a bug yesterday with the bundler crew after running into the (crippling) bundler+unicorn+capistrano RUBYOPT-accumulates-over-time issue discussed on this list previously:

http://www.mail-archive.com/mongrel-unicorn at rubyforge.org/msg00273.html

...and am happy to report it has already been fixed and released in bundler 0.9.17!

http://github.com/carlhuda/bundler/issues/issue/259/#comment_180830 

May your unicorns prance freely,

-jamie
http://jamiedubs.com | http://140proof.com

From jamie at tramchase.com  Thu Apr  8 17:21:48 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Thu, 8 Apr 2010 14:21:48 -0700
Subject: funky process tree + stillborn masters
Message-ID: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>

Since upgrading bundler (but applying the RUBYOPT path fixes) we've started experiencing intermittent, difficult-to-isolate USR2 restart failures.

After a USR2 signal our process tree winds up looking like this, with several master-esque processes listed as children (but without the "worker[N]" label):

app      14402  4.4  0.8 199612 70264 ?        S    14:07   0:04 unicorn_rails master -c config/unicorn.rb -E production -D
app      14433  0.0  0.8 204540 68504 ?        Sl   14:07   0:00  \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D
app      14435  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D
app      14438  0.0  0.8 199748 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
app      14440  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D
app      14442  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D
app      14445  0.0  0.8 199760 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
app      14447  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D
app      14449  0.0  0.8 204780 69272 ?        Sl   14:07   0:00  \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D

Sending another USR2 signal will bring a new master into the mix as a child, spins up a single child worker of its own (which also resembles the "/usr/bin/ruby1.8" master-esque processes), and then fails to continue. 

Further USR2 restarts will obviously do nothing, and we're forced to either kill -9 the stillborn master or cold-restart all of the unicorns. Nothing out of the ordinary is dumped to stderr or stdout

Starting unicorns fresh produces a nice process list where every child is listed cleanly as "unicorn_rails worker[N]"

We only have this issue in one of our applications, on a machine that has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've also only noticed this since upgrading to bundler from bundler08

Are the goofy worker processes in the process tree a real problem, or just a red herring?

-jamie

From normalperson at yhbt.net  Thu Apr  8 19:55:48 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 8 Apr 2010 16:55:48 -0700
Subject: funky process tree + stillborn masters
In-Reply-To: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
Message-ID: <20100408235548.GA11184@dcvr.yhbt.net>

Jamie Wilkinson <jamie at tramchase.com> wrote:
> Since upgrading bundler (but applying the RUBYOPT path fixes) we've
> started experiencing intermittent, difficult-to-isolate USR2 restart
> failures.
> 
> After a USR2 signal our process tree winds up looking like this, with
> several master-esque processes listed as children (but without the
> "worker[N]" label):
> 
> app      14402  4.4  0.8 199612 70264 ?        S    14:07   0:04 unicorn_rails master -c config/unicorn.rb -E production -D
> app      14433  0.0  0.8 204540 68504 ?        Sl   14:07   0:00  \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D
> app      14435  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D
> app      14438  0.0  0.8 199748 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app      14440  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D
> app      14442  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D
> app      14445  0.0  0.8 199760 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
> app      14447  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D
> app      14449  0.0  0.8 204780 69272 ?        Sl   14:07   0:00  \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D

> Sending another USR2 signal will bring a new master into the mix as a
> child, spins up a single child worker of its own (which also resembles
> the "/usr/bin/ruby1.8" master-esque processes), and then fails to
> continue. 

Hi Jamie,

Odd, if I had to guess PIDs 14438 and 14445 are actually worker[2] and
worker[5] based on the PIDs relative to other workers.  So the new
master died right away, which really should've been logged...

> Further USR2 restarts will obviously do nothing, and we're forced to
> either kill -9 the stillborn master or cold-restart all of the
> unicorns. Nothing out of the ordinary is dumped to stderr or stdout

Anything in your before_fork/after_fork hooks?  Since it looks like
you're on a Linux system, can you strace the master while you send
it a USR2 and see if anything strange happens?

Also, can you strace the weird looking processes above and see if
they're doing anything?

> Starting unicorns fresh produces a nice process list where every child
> is listed cleanly as "unicorn_rails worker[N]"
> 
> We only have this issue in one of our applications, on a machine that
> has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've
> also only noticed this since upgrading to bundler from bundler08

I assume you're using regular "unicorn" to run your Sinatra apps and not
"unicorn_rails".  I made some largish cleanups to both for the 0.97.0
release and and perhaps some bugs slipped into the "_rails" variant.

Can you try regular "unicorn" with a config.ru for Rails?  I've stolen
this from the Rainbows! FAQ (http://rainbows.rubyforge.org/FAQ.html):

  For Rails 2.3.x and later, the following config.ru will work for you:

    ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
    require "config/environment"
    use Rails::Rack::Static
    run ActionController::Dispatcher.new

  For older versions of Rails, the following config.ru will work:

    ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
    require 'config/boot'
    require 'config/environment'
    require 'unicorn/app/old_rails'
    require 'unicorn/app/old_rails/static' # not needed with Unicorn 0.95+
    use Unicorn::App::OldRails::Static
    run Unicorn::App::OldRails.new

  One thing to watch out for is that RAILS_ENV will not be set in the
  environment for you, thus we set it to match RACK_ENV.

> Are the goofy worker processes in the process tree a real problem, or
> just a red herring?

Not sure if it's a problem, but with Bundler I assume Rack itself is a
bundled dependency, but you're starting unicorn_rails out of
/usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
use the bundled Rack).  Can you ensure your unbundled Rack is the same
version as the bundled one to be on the safe side?

I've yet to try bundler 0.9 (and have barely used earlier), but you can
also try bundling Unicorn and using the bundled bin/unicorn(_rails)
launchers instead to ensure a consistent environment.

Unicorn currently ends up (auto)loading "rack/utils" before the
application is loaded, maybe it could (auto)load it after the app is
loaded for preload_app users.

-- 
Eric Wong

From normalperson at yhbt.net  Thu Apr  8 21:20:50 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 8 Apr 2010 18:20:50 -0700
Subject: funky process tree + stillborn masters
In-Reply-To: <20100408235548.GA11184@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
Message-ID: <20100409012050.GA31641@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> I assume you're using regular "unicorn" to run your Sinatra apps and not
> "unicorn_rails".  I made some largish cleanups to both for the 0.97.0
> release and and perhaps some bugs slipped into the "_rails" variant.

<snip>

> Jamie Wilkinson <jamie at tramchase.com> wrote:
> > Are the goofy worker processes in the process tree a real problem, or
> > just a red herring?
> 
> Not sure if it's a problem, but with Bundler I assume Rack itself is a
> bundled dependency, but you're starting unicorn_rails out of
> /usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
> use the bundled Rack).  Can you ensure your unbundled Rack is the same
> version as the bundled one to be on the safe side?
> 
> I've yet to try bundler 0.9 (and have barely used earlier), but you can
> also try bundling Unicorn and using the bundled bin/unicorn(_rails)
> launchers instead to ensure a consistent environment.
> 
> Unicorn currently ends up (auto)loading "rack/utils" before the
> application is loaded, maybe it could (auto)load it after the app is
> loaded for preload_app users.

Do the following two patches help?  I've also pushed out a few
cleanups to unicorn.git and also put up a prerelease gem at:

  http://unicorn.bogomips.org/files/gems/unicorn-0.97.0.7.g22e3.gem

Shortlog of changes since 0.97.0:

Eric Wong (7):
      tests: fix to run under Ruby 1.9.2dev
      KNOWN_ISSUES: document Array#shuffle / Array#sample issue under 1.9
      unicorn_rails: use TOPLEVEL_BINDING for eval
      unicorn_rails: respect user's encoding in config.ru in 1.9
      unicorn_rails: rename variable for consistency
      bin/*: remove unnecessary listeners variable
      unicorn: load constants after app has loaded

>From e1a72d58add4260feb6da56d9d588270173da74f Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Thu, 8 Apr 2010 17:10:46 -0700
Subject: [PATCH] unicorn_rails: use TOPLEVEL_BINDING for eval

This is to ensure there are no namespace inconsistencies
---
 bin/unicorn_rails |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 37ee027..de2361e 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -148,7 +148,7 @@ def rails_builder(config, daemonize)
     when /\.ru$/
       raw = File.open(config, "rb") { |fp| fp.sysread(fp.stat.size) }
       raw.sub!(/^__END__\n.*/, '')
-      eval("Rack::Builder.new {(#{raw}\n)}.to_app", nil, config)
+      eval("Rack::Builder.new {(#{raw}\n)}.to_app", TOPLEVEL_BINDING, config)
     else
       require config
       Object.const_get(File.basename(config, '.rb').capitalize)
-- 

>From 22e3ed4de0e89b97dac91c95c796eb8a7f93e5de Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Thu, 8 Apr 2010 18:05:43 -0700
Subject: [PATCH] unicorn: load constants after app has loaded

This will help ensure we use the same version of Rack the
application uses and avoid loading conflicting/incompatible
versions.
---
 lib/unicorn.rb |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b63abeb..75ce09d 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -800,15 +800,15 @@ module Unicorn
 
     def build_app!
       if app.respond_to?(:arity) && app.arity == 0
-        # exploit COW in case of preload_app.  Also avoids race
-        # conditions in Rainbows! since load/require are not thread-safe
-        Unicorn.constants.each { |x| Unicorn.const_get(x) }
-
         if defined?(Gem) && Gem.respond_to?(:refresh)
           logger.info "Refreshing Gem list"
           Gem.refresh
         end
         self.app = app.call
+
+        # exploit COW in case of preload_app.  Also avoids race
+        # conditions in Rainbows! since load/require are not thread-safe
+        Unicorn.constants.each { |x| Unicorn.const_get(x) }
       end
     end

-- 
Eric Wong

From normalperson at yhbt.net  Thu Apr  8 22:37:05 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 9 Apr 2010 02:37:05 +0000
Subject: [ANN] raindrops - real-time stats for preforking Rack servers
Message-ID: <20100409023705.GB5154@dcvr.yhbt.net>

(I announced this yesterday on ruby-talk, but I realized it would
be wise to actually announce it on the list of a server it was
designed for :)

Raindrops is a real time stats package to show statistics for Rack HTTP
servers.  It is designed for preforking servers such as Rainbows! and
Unicorn, but should support any Rack HTTP server under Ruby 1.9, 1.8 and
possibly Rubinius (untested) on platforms supporting POSIX shared memory
and compiled with GCC (for atomic builtins).

Raindrops includes a Struct-like Raindrops::Struct class that may be used
standalone to create atomic counters shared across any number of forked
processes under SMP.

* http://raindrops.bogomips.org/
* raindrops at librelist.com
* git://git.bogomips.org/raindrops.git

== Features

* counters are shared across all forked children and lock-free

* counters are kept on separate cache lines to reduce contention under SMP

* may expose server statistics as a Rack Middleware endpoint
  (default: "/_raindrops")

* middleware displays the number of actively processing and writing
  clients from a single request regardless of which worker process
  it hits.

== Linux-only Extra Features!

* Middleware response includes extra stats for bound TCP and
  Unix domain sockets (configurable, it can include stats from
  other TCP or UNIX domain socket servers).

* TCP socket stats use efficient inet_diag facilities via netlink
  instead of parsing /proc/net/tcp to minimize overhead.
  This was fun to discover and write.

== Install

raindrops requires GCC 4.x (or compatible) or later to support the
atomic builtins (__sync_{add,sub}_and_fetch()).  Atomic operations on
other compilers may be supported if there is demand.

If you're using a packaged Ruby distribution, make sure you have a C
compiler and the matching Ruby development libraries and headers.

If you use RubyGems:

    gem install raindrops

Otherwise grab the latest tarball from:

http://raindrops.bogomips.org/files/

Unpack it, and run "ruby setup.rb"

== Usage (Rainbows!/Unicorn preload_app=false)

If you're using preload_app=false (the default) in your Rainbows!/Unicorn
config file, you'll need to create the global Stats object before
forking.

        require 'raindrops'
        $stats ||= Raindrops::Middleware::Stats.new

In your Rack config.ru:

        use Raindrops::Middleware, :stats => $stats

== Usage (Rainbows!/Unicorn preload_app=true)

If you're using preload_app=true in your Rainbows!/Unicorn
config file, just add the middleware to your stack:

In your Rack config.ru:

        use Raindrops::Middleware

== Usage (Linux-extras)

To get bound listener statistics under Linux, you need to specify the
listener names for your server.  You can even include listen sockets for
*other* servers on the same machine.  This can be handy for monitoring
your nginx proxy as well.

In your Rack config.ru, just pass the :listeners argument as an array of
strings (along with any other arguments).  You can specify any
combination of TCP or Unix domain socket names:

        use Raindrops::Middleware, :listeners => %w(0.0.0.0:80 /tmp/.sock)

See the tests/ and examples/ directory for more examples

== Development

You can get the latest source via git from the following locations:

  git://git.bogomips.org/raindrops.git
  git://repo.or.cz/raindrops.git (mirror)

You may browse the code from the web and download the latest snapshot
tarballs here:

* http://git.bogomips.org/cgit/raindrops.git (cgit)
* http://repo.or.cz/w/raindrops.git (gitweb)

Inline patches (from "git format-patch") to the mailing list are
preferred because they allow code review and comments in the reply to
the patch.

We will adhere to mostly the same conventions for patch submissions as
git itself.  See the Documentation/SubmittingPatches document
distributed with git on on patch submission guidelines to follow.  Just
don't email the git mailing list or maintainer with raindrops patches.

== Contact

All feedback (bug reports, user/development discussion, patches, pull
requests) go to the mailing list: mailto:raindrops at librelist.com

-- 
Eric Wong

From normalperson at yhbt.net  Sun Apr 11 22:52:01 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 11 Apr 2010 19:52:01 -0700
Subject: funky process tree + stillborn masters
In-Reply-To: <20100409012050.GA31641@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
Message-ID: <20100412025200.GA29391@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Do the following two patches help?  I've also pushed out a few
> cleanups to unicorn.git and also put up a prerelease gem at:

Hi Jamie, did you get a chance to try this?

-- 
Eric Wong

From jamie at tramchase.com  Mon Apr 12 23:23:30 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Mon, 12 Apr 2010 20:23:30 -0700
Subject: funky process tree + stillborn masters
In-Reply-To: <20100412025200.GA29391@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
Message-ID: <F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>

Whew! After a long day of debugging, I think I've  gotten closer to isolating this.

I've upgraded us to your pre-release gem but I can still get it working (both before & after a lot of the below fixes)

>> After a USR2 signal our process tree winds up looking like this, with
>> several master-esque processes listed as children (but without the
>> "worker[N]" label):
>> 
>> app      14402  4.4  0.8 199612 70264 ?        S    14:07   0:04 unicorn_rails master -c config/unicorn.rb -E production -D
>> app      14433  0.0  0.8 204540 68504 ?        Sl   14:07   0:00  \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D
>> app      14435  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D
>> app      14438  0.0  0.8 199748 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
>> app      14440  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D
>> app      14442  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D
>> app      14445  0.0  0.8 199760 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
>> app      14447  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D
>> app      14449  0.0  0.8 204780 69272 ?        Sl   14:07   0:00  \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D
>> 
>> Sending another USR2 signal will bring a new master into the mix as a
>> child, spins up a single child worker of its own (which also resembles
>> the "/usr/bin/ruby1.8" master-esque processes), and then fails to
>> continue. 
> 
> Anything in your before_fork/after_fork hooks?  Since it looks like
> you're on a Linux system, can you strace the master while you send
> it a USR2 and see if anything strange happens?

The only real contents of our before_hook is a send-QUIT-on-first-worker, which I swapped out for the default SIGTTOU behavior. No change.


> I assume you're using regular "unicorn" to run your Sinatra apps and not
> "unicorn_rails".  I made some largish cleanups to both for the 0.97.0
> release and and perhaps some bugs slipped into the "_rails" variant.
> 
> Not sure if it's a problem, but with Bundler I assume Rack itself is a
> bundled dependency, but you're starting unicorn_rails out of
> /usr/bin/unicorn_rails which indicates Unicorn is not bundled (and won't
> use the bundled Rack).  Can you ensure your unbundled Rack is the same
> version as the bundled one to be on the safe side?
> 

My system & bundled rack versions match.

Swapped to vanilla "unicorn" instead of "unicorn_rails" -- also no dice

I switched to using "bundle exec unicorn", which uses RAILS_ROOT/vendor/bundler_gems/bin/unicorn instead of /usr/bin/unicorn. Was convinced this would be it, but no dice.

Attaching some relevant stack traces... the new "orphan" master & its 1st child are pretty boring while just hanging out:

Process 20738 attached - interrupt to quit
futex(0x2aaaaafb23c0, FUTEX_WAIT, 2, NULL

Sending a USR2 to the new, orphaned master...

Process 20738 attached - interrupt to quit
select(10, [9], [], [], {23, 661000})   = ? ERESTARTNOHAND (To be restarted)
--- SIGUSR2 (User defined signal 2) @ 0 (0) ---
rt_sigreturn(0xc)                       = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
clock_gettime(CLOCK_MONOTONIC, {2782804, 359708496}) = 0
select(0, [], [], [], {0, 0})           = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
fcntl(5, F_GETFL)                       = 0x801 (flags O_WRONLY|O_NONBLOCK)
write(5, "."..., 1)                     = 1
clock_gettime(CLOCK_MONOTONIC, {2782804, 360046496}) = 0
select(10, [9], [], [], {20, 464341}


I've also produced straces of the *original* master during USR2 restarts, both a success trace and a failure trace. 
Here's a tarball with both complete traces as well as filtered/grepp'd ones:
http://jamiedubs.com/files/unicorn-strace.tgz

I've also found that kill -9'ing the 1st worker of the new orphaned master allows it to continue operation as normal (spinning up workers and taking control from the original master) -- suggesting something is up with just that first worker (!). I'm going to keep noodling with before_/after_fork strategies.

-jamie

From normalperson at yhbt.net  Tue Apr 13 01:24:10 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 13 Apr 2010 05:24:10 +0000
Subject: funky process tree + stillborn masters
In-Reply-To: <F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
	<F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
Message-ID: <20100413052410.GA31794@dcvr.yhbt.net>

Jamie Wilkinson <jamie at tramchase.com> wrote:
> I've also produced straces of the *original* master during USR2
> restarts, both a success trace and a failure trace.  Here's a tarball
> with both complete traces as well as filtered/grepp'd ones:
> http://jamiedubs.com/files/unicorn-strace.tgz
> 
> I've also found that kill -9'ing the 1st worker of the new orphaned
> master allows it to continue operation as normal (spinning up workers
> and taking control from the original master) -- suggesting something
> is up with just that first worker (!). I'm going to keep noodling with
> before_/after_fork strategies.

Hi Jamie, your successful straces have this oddity in them:

  poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 1 ([{fd=7, revents=POLLIN|POLLHUP}
  read(7, ""..., 8192)                    = 0
  shutdown(7, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not
  close(7)                                = 0

> Eric Wong <normalperson at yhbt.net> wrote:
> > Anything in your before_fork/after_fork hooks?  Since it looks like
> > you're on a Linux system, can you strace the master while you send
> > it a USR2 and see if anything strange happens?
> 
> The only real contents of our before_hook is a
> send-QUIT-on-first-worker, which I swapped out for the default SIGTTOU
> behavior. No change.

It looks like you should be disconnecting/reconnecting whatever is using
fd=7 in the before_fork/after_fork hooks respectively.

Unicorn (and mainline Ruby) do not use the poll() system call (which is
why that strace raised red flags), so there's another C extension
calling poll() in your master process (not good).  I remember one (or
all of) the Ruby Postgres libraries using poll() internally, and there
are likely other things that use poll() in Ruby as well.

Perhaps something like this in the config (assuming you're using
Rails ActiveRecord) (also see http://unicorn.bogomips.org/examples/)

  before_fork do |server,worker|
    defined?(ActiveRecord::Base) and
      ActiveRecord::Base.connection.disconnect!
  end

  after_fork do |server,worker|
    defined?(ActiveRecord::Base) and
      ActiveRecord::Base.establish_connection
  end

-- 
Eric Wong

From ibc at aliax.net  Thu Apr 15 13:17:10 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Thu, 15 Apr 2010 19:17:10 +0200
Subject: Explanation of how Unicorn returns error code when daemonized and 
	cannot bind the address
Message-ID: <o2occ1f582e1004151017s98788d2au437724819596dde0@mail.gmail.com>

Hi Eric, as you remember you did an improvement in Unicorn in order it
to return an error code (maybe 1) when running daemonized and trying
to bind in a wrong address (i.e. a non local address).

As far as I remember you implemented a pipe between daemon and master
process, so if the master returns error it sends such result to the
daemon process and it also ends with error. This is more than useful
and required when handling init script with HeartBeat and similar
software in which the return code of the init script must be correct
(it just can return 0 if it has started correctly).

If I'm wrong, could you detail it a bit please? I want to propose the
same improvement in other project in which I participate.

Really thanks a lot.

-- 
I?aki Baz Castillo
<ibc at aliax.net>

From ibc at aliax.net  Thu Apr 15 13:46:08 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Thu, 15 Apr 2010 19:46:08 +0200
Subject: Explanation of how Unicorn returns error code when daemonized and
	cannot bind the address
In-Reply-To: <o2occ1f582e1004151017s98788d2au437724819596dde0@mail.gmail.com>
References: <o2occ1f582e1004151017s98788d2au437724819596dde0@mail.gmail.com>
Message-ID: <x2pcc1f582e1004151046r38ec2443kcac3cca928dff5c2@mail.gmail.com>

2010/4/15 I?aki Baz Castillo <ibc at aliax.net>:
> Hi Eric, as you remember you did an improvement in Unicorn in order it
> to return an error code (maybe 1) when running daemonized and trying
> to bind in a wrong address (i.e. a non local address).
>
> As far as I remember you implemented a pipe between daemon and master
> process, so if the master returns error it sends such result to the
> daemon process and it also ends with error. This is more than useful
> and required when handling init script with HeartBeat and similar
> software in which the return code of the init script must be correct
> (it just can return 0 if it has started correctly).
>
> If I'm wrong, could you detail it a bit please? I want to propose the
> same improvement in other project in which I participate.

Well, I've rechecked the code and it seems simple: the grandparent
process wait listening into the pipe. The master process would write
in the pipe if all goes ok, then grandparent would exit with 0. If the
master process fails it closes the pipe, the grandparent detects it
and exits with 1 (error)  :)


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From ibc at aliax.net  Fri Apr 16 14:49:17 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Fri, 16 Apr 2010 20:49:17 +0200
Subject: Using HTTP/1.1 and permanent connections
Message-ID: <r2hcc1f582e1004161149ja4187a68k7867f794aeed8d7f@mail.gmail.com>

Hi, I know that Unicorn forces TCP disconnection as it's explained at
the top of lib/unicorn/http_response.rb:

  # A design decision was made to force the client to not pipeline or
  # keepalive requests.  HTTP/1.1 pipelining really kills the
  # performance due to how it has to be handled and how unclear the
  # standard is.  To fix this the HttpResponse always gives a
  # "Connection: close" header which forces the client to close right
  # away.  The bonus for this is that it gives a pretty nice speed boost
  # to most clients since they can close their connection immediately.


However I want to try TCP permanent connections from the client (or a
proxy) to Unicorn. I've tryed to modify same file as above by changing
at the end:


        # Remove "Connection: close":
        socket.write("HTTP/1.1 #{status}\r\n" \
                     "Date: #{Time.now.httpdate}\r\n" \
                     "Status: #{status}\r\n" \
                     "#{out.join('')}\r\n")
      end

      body.each { |chunk| socket.write(chunk) }
      # Don't close the socket:
      #socket.close # flushes and uncorks the socket immediately
      ensure
        body.respond_to?(:close) and body.close


But of course this is not enough and it fails. Not sure what exactly
happens, it seems that a Unicorn worker doesn't process requests
anymore until replying the first response.

So I would like to know if it's feasible to make Unicorn work in
persistent mode.
NOTE: I already know that the current design is really good, better
than using persistent connections, but I want to experiment with
persistent connections for other purposes I will explain in a future.


Thanks a lot.


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From normalperson at yhbt.net  Fri Apr 16 16:09:18 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 16 Apr 2010 13:09:18 -0700
Subject: Using HTTP/1.1 and permanent connections
In-Reply-To: <r2hcc1f582e1004161149ja4187a68k7867f794aeed8d7f@mail.gmail.com>
References: <r2hcc1f582e1004161149ja4187a68k7867f794aeed8d7f@mail.gmail.com>
Message-ID: <20100416200918.GA10256@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I know that Unicorn forces TCP disconnection as it's explained at
> the top of lib/unicorn/http_response.rb:
> 
>   # A design decision was made to force the client to not pipeline or
>   # keepalive requests.  HTTP/1.1 pipelining really kills the
>   # performance due to how it has to be handled and how unclear the
>   # standard is.  To fix this the HttpResponse always gives a
>   # "Connection: close" header which forces the client to close right
>   # away.  The bonus for this is that it gives a pretty nice speed boost
>   # to most clients since they can close their connection immediately.
> 
> 
> However I want to try TCP permanent connections from the client (or a
> proxy) to Unicorn. I've tryed to modify same file as above by changing
> at the end:

<snip>

> But of course this is not enough and it fails. Not sure what exactly
> happens, it seems that a Unicorn worker doesn't process requests
> anymore until replying the first response.
> 
> So I would like to know if it's feasible to make Unicorn work in
> persistent mode.
> NOTE: I already know that the current design is really good, better
> than using persistent connections, but I want to experiment with
> persistent connections for other purposes I will explain in a future.

Hi I?aki,

Unicorn won't try to read further requests from the socket (see
process_client() in lib/unicorn.rb).

However, you can use Rainbows! without specifying a concurrency model at
all (or "use :Base").  That way you'll get proper HTTP/1.1 keepalive
support and you can control keepalive_timeout:

# both of these values are the defaults for unconfigured Rainbows!
Rainbows! do
  use :Base
  keepalive_timeout 5
end

-- 
Eric Wong

From ibc at aliax.net  Sat Apr 17 18:53:42 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Sun, 18 Apr 2010 00:53:42 +0200
Subject: Using HTTP/1.1 and permanent connections
In-Reply-To: <20100416200918.GA10256@dcvr.yhbt.net>
References: <r2hcc1f582e1004161149ja4187a68k7867f794aeed8d7f@mail.gmail.com>
	<20100416200918.GA10256@dcvr.yhbt.net>
Message-ID: <k2gcc1f582e1004171553q39803ca6oc23d9d32efc75ac6@mail.gmail.com>

2010/4/16 Eric Wong <normalperson at yhbt.net>:
> Hi I?aki,
>
> Unicorn won't try to read further requests from the socket (see
> process_client() in lib/unicorn.rb).

Ok, understood now.


> However, you can use Rainbows! without specifying a concurrency model at
> all (or "use :Base"). ?That way you'll get proper HTTP/1.1 keepalive
> support and you can control keepalive_timeout:
>
> # both of these values are the defaults for unconfigured Rainbows!
> Rainbows! do
> ?use :Base
> ?keepalive_timeout 5
> end


Great! Thanks a lot.


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From jjames2200 at googlemail.com  Sun Apr 18 21:48:12 2010
From: jjames2200 at googlemail.com (JJames)
Date: Mon, 19 Apr 2010 06:18:12 +0430
Subject: Work From Home!
Message-ID: <mailman.0.1271628402.12526.mongrel-unicorn@rubyforge.org>

Hello,

We are going to show you the 
fastest way to make money at 
home! 

Can you picture yourself 
working from home and and 
filling out short and simple 
online forms for fast and 
easy cash? 

Getting paid for the easiest 
work imaginable? What's more, 
what if you could do this all 
while sitting at home in your 
pajamas!

Interested? Click here:
http://offto.net/nd4orl/

Your friend,
J.James

Note:
If you wish not to receive 
further emails, please reply
to this email with a subject
"REMOVE"

From normalperson at yhbt.net  Mon Apr 19 14:21:47 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 19 Apr 2010 18:21:47 +0000
Subject: funky process tree + stillborn masters
In-Reply-To: <20100413052410.GA31794@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
	<F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
	<20100413052410.GA31794@dcvr.yhbt.net>
Message-ID: <20100419182146.GA22994@dcvr.yhbt.net>

Hi Jamie, any more news on this issue?  I plan on doing a minor release
later today or tomorrow with a few minor bug fixes in unicorn.git

-- 
Eric Wong

From normalperson at yhbt.net  Mon Apr 19 17:31:35 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 19 Apr 2010 21:31:35 +0000
Subject: [ANN] unicorn 0.97.1 - fix HTTP parser for Rainbows!/Zbatery
Message-ID: <20100419213135.GH26722@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

This release fixes a denial-of-service vector for derived
servers exposed directly to untrusted clients.

This bug does not affect most Unicorn deployments as Unicorn is
only supported with trusted clients (such as nginx) on a LAN.
nginx is known to reject clients that send invalid
Content-Length headers, so any deployments on a trusted LAN
and/or behind nginx are safe.

Servers affected by this bug include (but are not limited to)
Rainbows! and Zbatery.  This bug does not affect Thin nor
Mongrel, as neither got the request body filtering treatment
that the Unicorn HTTP parser got in August 2009.

The bug fixed in this release could result in a
denial-of-service as it would trigger a process-wide assertion
instead of raising an exception.  For servers such as
Rainbows!/Zbatery that serve multiple clients per worker
process, this could abort all clients connected to the
particular worker process that hit the assertion.

-- 
Eric Wong

From ibc at aliax.net  Mon Apr 26 04:18:19 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Mon, 26 Apr 2010 10:18:19 +0200
Subject: Shared memory between workers
Message-ID: <w2qcc1f582e1004260118za5609803x3753efd2ac850e4c@mail.gmail.com>

Hi, I plan to build a SIP TCP server (no UDP) based on
Unicorn/Rainbows! HTTP server. The main different between a SIP server
and HTTP server are:

- SIP uses persistent TCP connections, so I should use Rainbows!.
- For a SIP server it's not valid a simple request-response model.
Different workers could handle SIP messages (requests and responses)
belonging to the same SIP session so I need a shared memory between
all the workers.

Another option is using EventMachine, perhaps more suitable for this
purpose by design as it uses a single Ruby process so sharing memory
is not a problem. In the other side using a single process in a
multicore server is a pain.
I would like to use Unicorn/Rainbows as I love its design: by far it's
the more reliable and efficient Ruby HTTP server and it takes
advantages of Unix's features.

I don't want to use a DB server neither MemCache as "shared memory" as
it would be too slow. Is there any way to share RAM memory between
different Unicorn/Rainbows! workers in a *safe* way? I could create a
Hash or Array of SIP sessions into the master process so all the
workers reuse it, but I don't think it would be safe to access/write
into it from different Ruby processes. For that I would also need a
semaphore system (perhaps again a shared variable between all workers
in order to lock the shared Array/Hash when a worker writes into it).

Any tip about it? suggstions?
Thanks a lot.

-- 
I?aki Baz Castillo
<ibc at aliax.net>

From normalperson at yhbt.net  Mon Apr 26 15:03:39 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 26 Apr 2010 12:03:39 -0700
Subject: Shared memory between workers
In-Reply-To: <w2qcc1f582e1004260118za5609803x3753efd2ac850e4c@mail.gmail.com>
References: <w2qcc1f582e1004260118za5609803x3753efd2ac850e4c@mail.gmail.com>
Message-ID: <20100426190338.GA27686@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Hi, I plan to build a SIP TCP server (no UDP) based on
> Unicorn/Rainbows! HTTP server. The main different between a SIP server
> and HTTP server are:
> 
> - SIP uses persistent TCP connections, so I should use Rainbows!.
> - For a SIP server it's not valid a simple request-response model.
> Different workers could handle SIP messages (requests and responses)
> belonging to the same SIP session so I need a shared memory between
> all the workers.
> 
> Another option is using EventMachine, perhaps more suitable for this
> purpose by design as it uses a single Ruby process so sharing memory
> is not a problem. In the other side using a single process in a
> multicore server is a pain.
> I would like to use Unicorn/Rainbows as I love its design: by far it's
> the more reliable and efficient Ruby HTTP server and it takes
> advantages of Unix's features.
> 
> I don't want to use a DB server neither MemCache as "shared memory" as
> it would be too slow. Is there any way to share RAM memory between
> different Unicorn/Rainbows! workers in a *safe* way? I could create a
> Hash or Array of SIP sessions into the master process so all the
> workers reuse it, but I don't think it would be safe to access/write
> into it from different Ruby processes. For that I would also need a
> semaphore system (perhaps again a shared variable between all workers
> in order to lock the shared Array/Hash when a worker writes into it).
> 
> Any tip about it? suggstions?

Hi I?aki,

First off I wouldn't call Memcached slow...  How many requests do you
need out of it and what kind of access patterns are you doing?

If you're on a modern Linux 2.6 box, then anything on the local
filesystem is basically shared memory if you have enough RAM to store
your working set (and it sounds like you do).

If you're willing to be confined to a single box, I've had great
experiences with TokyoCabinet the past few years.  One system I built
shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
(previously Mongrel) workers.  TokyoCabinet uses mmap and pread/pwrite,
so you can safely share DB handles across any number of native
threads/processes.

Writes in TC need filesystem locking and you can only have one active
writer, but that was still plenty fast enough in my experience.

If fsync() becomes a bottleneck, then for non-critical data either:

  1) disable it (TC lets you)
  2) or use libeatmydata if you choose something that doesn't let
     you disable fsync()

<soapbox>
  My personal opinion is that fsync()/O_SYNC are completely overrated.
  All important machines these days have redundant storage, redundant
  power supplies, ECC memory, hardware health monitoring/alerting and
  backup power.  Any remaining hardware/kernel failure possibilities
  are far less likely than application bugs that corrupt data :)

  But, if you really have critical data, /then/ get an SSD or
  a battery-backed write cache.
</soapbox>

-- 
Eric Wong

From ibc at aliax.net  Mon Apr 26 18:46:30 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Tue, 27 Apr 2010 00:46:30 +0200
Subject: Shared memory between workers
In-Reply-To: <20100426190338.GA27686@dcvr.yhbt.net>
References: <w2qcc1f582e1004260118za5609803x3753efd2ac850e4c@mail.gmail.com>
	<20100426190338.GA27686@dcvr.yhbt.net>
Message-ID: <v2gcc1f582e1004261546y874ecdd0yf3931d014ab3a000@mail.gmail.com>

2010/4/26 Eric Wong <normalperson at yhbt.net>:

> Hi I?aki,
>
> First off I wouldn't call Memcached slow... ?How many requests do you
> need out of it and what kind of access patterns are you doing?

Well, I want to do as efficient as possible. This is not the main
purpose (if so I wouldn't use Ruby) but it's important.

For example:
- 8 workers.
- A SIP request arrives and it handled by worker 1.
- The server must store in *shared* memory info about such request
(status, last SIP response sereplied for it....). The request
transaction remains in memory for some seconds after sending the
response.
- Due to a non so unusual SIP issue, a retransmission of the previous
request arrives to the server but now it is handled by worker 2.
- Worker 2 must check the request "branch" id into shared memory, find
the previously created entry, check that a response was already sent
for it, and re-send such response.

Well, in fact this is really common in SIP under UDP. Under TCP
usually there are no retransmissions (but they could be so the worker
must check it).



> If you're on a modern Linux 2.6 box, then anything on the local
> filesystem is basically shared memory if you have enough RAM to store
> your working set (and it sounds like you do).

And what about mounting a filesystem under RAM? :would it be ever faster? :)


> If you're willing to be confined to a single box, I've had great
> experiences with TokyoCabinet the past few years. ?One system I built
> shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn
> (previously Mongrel) workers. ?TokyoCabinet uses mmap and pread/pwrite,
> so you can safely share DB handles across any number of native
> threads/processes.

I didn't know it. I've just read the doc, installed and tested it.
Really impressive!


> Writes in TC need filesystem locking and you can only have one active
> writer, but that was still plenty fast enough in my experience.

If you use a "on memory database" there wouldn't be filesystem locking, right?


> If fsync() becomes a bottleneck, then for non-critical data either:
>
> ?1) disable it (TC lets you)
> ?2) or use libeatmydata if you choose something that doesn't let
> ? ? you disable fsync()
>
> <soapbox>
> ?My personal opinion is that fsync()/O_SYNC are completely overrated.
> ?All important machines these days have redundant storage, redundant
> ?power supplies, ECC memory, hardware health monitoring/alerting and
> ?backup power. ?Any remaining hardware/kernel failure possibilities
> ?are far less likely than application bugs that corrupt data :)
>
> ?But, if you really have critical data, /then/ get an SSD or
> ?a battery-backed write cache.
> </soapbox>

Thanks, but the fact is that I don't need persistent storage. This is,
the data I will need to share between Ruby processes (workers) is
about realtime data, so if the server is stopped and restarted I don't
want to retrieve such information. Well, it could be useful for some
cases however, I need to think about it.
I've seen a method "sync" in the Tokyo Tyrant client API which forces
the server to store/sync the on memory data to the file (if a file is
being used).


So thanks a lot. Definitely I'll learn about TokioCabinet as it seems
a really nice solution for my needs.
Regards.


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From joel at watsonian.net  Mon Apr 26 22:37:08 2010
From: joel at watsonian.net (Joel Watson)
Date: Mon, 26 Apr 2010 19:37:08 -0700
Subject: [PATCH] Add worker interrogation via INFO signals
Message-ID: <FCE9AA21-B75A-4518-8FDE-2654CF9592B4@watsonian.net>

Hey all,

Below is a proposed patch I worked on over the weekend. Just adding a note here to mention that I'm currently not a subscriber to the mailing list, so please CC me on any replies. Let me know what you all think. This change was made on a local topic branch off of the maint branch. If you'd like to view the change on GitHub, you can do so here: http://github.com/watsonian/unicorn/compare/maint...siginfo

-Joel
================================
From a0ccb9efe508d4bd0a2a238305fedcbfc051d202 Mon Sep 17 00:00:00 2001
From: watsonian <watsonian at gmail.com>
Date: Mon, 26 Apr 2010 18:25:02 -0700
Subject: [PATCH] Add worker interrogation via INFO signals

You can now send a worker an INFO signal and it will write the
current request uri it's processing and how long it's been processing
it for to the log. Sending the master process an INFO signal will
send all workers an INFO signal as well.

This addresses cases where it's desirable to know exactly what a
worker is doing at a particular point in time (e.g., if it's hanging
on a particular request such that it isn't writing out to log files).
---
 lib/unicorn.rb |   25 ++++++++++++++++++++-----
 1 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index b63abeb..b11360c 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -163,7 +163,7 @@ module Unicorn
     # releases of Unicorn.  You may need to access it in the
     # before_fork/after_fork hooks.  See the Unicorn::Configurator RDoc
     # for examples.
-    class Worker < Struct.new(:nr, :tmp, :switched)
+    class Worker < Struct.new(:nr, :tmp, :switched, :request_uri, :request_start)
 
       # worker objects may be compared to just plain numbers
       def ==(other_nr)
@@ -422,6 +422,8 @@ module Unicorn
             self.worker_processes += 1
           when :TTOU
             self.worker_processes -= 1 if self.worker_processes > 0
+          when :INFO
+            kill_each_worker(:INFO)
           when :HUP
             respawn = true
             if config.config_file
@@ -462,7 +464,7 @@ module Unicorn
 
     # list of signals we care about and trap in master.
     QUEUE_SIGS = [ :WINCH, :QUIT, :INT, :TERM, :USR1, :USR2, :HUP,
-                   :TTIN, :TTOU ]
+                   :TTIN, :TTOU, :INFO ]
 
     # defer a signal for later processing in #join (master process)
     def trap_deferred(signal)
@@ -629,9 +631,12 @@ module Unicorn
 
     # once a client is accepted, it is processed in its entirety here
     # in 3 easy steps: read request, call app, write app response
-    def process_client(client)
+    def process_client(client, worker)
       client.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
-      response = app.call(env = REQUEST.read(client))
+      env = REQUEST.read(client)
+      worker.request_start = Time.now.to_i
+      worker.request_uri = env["REQUEST_URI"]
+      response = app.call(env)
 
       if 100 == response.first.to_i
         client.write(Const::EXPECT_100_RESPONSE)
@@ -641,6 +646,9 @@ module Unicorn
       HttpResponse.write(client, response, HttpRequest::PARSER.headers?)
     rescue => e
       handle_error(client, e)
+    ensure
+      worker.request_start = Time.now.to_i
+      worker.request_uri = nil
     end
 
     # gets rid of stuff the worker has no business keeping track of
@@ -662,6 +670,7 @@ module Unicorn
       worker.user(*user) if user.kind_of?(Array) && ! worker.switched
       self.timeout /= 2.0 # halve it for select()
       build_app! unless preload_app
+      worker.request_start = Time.now.to_i # keep track of initial idle time
     end
 
     def reopen_worker_logs(worker_nr)
@@ -684,6 +693,7 @@ module Unicorn
       # closing anything we IO.select on will raise EBADF
       trap(:USR1) { nr = -65536; SELF_PIPE.first.close rescue nil }
       trap(:QUIT) { alive = nil; LISTENERS.each { |s| s.close rescue nil } }
+      trap(:INFO) { logger.info worker_info(worker) }
       [:TERM, :INT].each { |sig| trap(sig) { exit!(0) } } # instant shutdown
       logger.info "worker=#{worker.nr} ready"
       m = 0
@@ -704,7 +714,7 @@ module Unicorn
 
         ready.each do |sock|
           begin
-            process_client(sock.accept_nonblock)
+            process_client(sock.accept_nonblock, worker)
             nr += 1
             alive.chmod(m = 0 == m ? 1 : 0)
           rescue Errno::EAGAIN, Errno::ECONNABORTED
@@ -817,6 +827,11 @@ module Unicorn
             ]).concat(START_CTX[:argv]).join(' ')
     end
 
+    # used to see what a worker is doing when it's sent an :INFO signal
+    def worker_info(worker)
+      "worker[#{worker.nr}] - #{worker.request_uri || "idle"} - #{(Time.now.to_i - worker.request_start).to_s if worker.request_start}"
+    end
+
     def redirect_io(io, path)
       File.open(path, 'ab') { |fp| io.reopen(fp) } if path
       io.sync = true
-- 
1.7.0.2



From normalperson at yhbt.net  Tue Apr 27 04:59:22 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 27 Apr 2010 08:59:22 +0000
Subject: [PATCH] Add worker interrogation via INFO signals
In-Reply-To: <FCE9AA21-B75A-4518-8FDE-2654CF9592B4@watsonian.net>
References: <FCE9AA21-B75A-4518-8FDE-2654CF9592B4@watsonian.net>
Message-ID: <20100427085922.GA8080@dcvr.yhbt.net>

Joel Watson <joel at watsonian.net> wrote:
> Hey all,
> 
> Below is a proposed patch I worked on over the weekend. Just adding a
> note here to mention that I'm currently not a subscriber to the
> mailing list, so please CC me on any replies. Let me know what you all
> think. This change was made on a local topic branch off of the maint
> branch. If you'd like to view the change on GitHub, you can do so
> here: http://github.com/watsonian/unicorn/compare/maint...siginfo
> 
> -Joel
> ================================
> >From a0ccb9efe508d4bd0a2a238305fedcbfc051d202 Mon Sep 17 00:00:00 2001
> From: watsonian <watsonian at gmail.com>
> Date: Mon, 26 Apr 2010 18:25:02 -0700
> Subject: [PATCH] Add worker interrogation via INFO signals
> 
> You can now send a worker an INFO signal and it will write the
> current request uri it's processing and how long it's been processing
> it for to the log. Sending the master process an INFO signal will
> send all workers an INFO signal as well.
> 
> This addresses cases where it's desirable to know exactly what a
> worker is doing at a particular point in time (e.g., if it's hanging
> on a particular request such that it isn't writing out to log files).
> ---
>  lib/unicorn.rb |   25 ++++++++++++++++++++-----
>  1 files changed, 20 insertions(+), 5 deletions(-)

Hi Joel,

I'm alright with most of the changes, but I'm not ready to put more
things into every single request that people cannot opt-out of.

Since this data is written in a way that most users will rarely access,
I would avoid the changes to process_client:

>      # once a client is accepted, it is processed in its entirety here
>      # in 3 easy steps: read request, call app, write app response
> -    def process_client(client)
> +    def process_client(client, worker)
>        client.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
> -      response = app.call(env = REQUEST.read(client))
> +      env = REQUEST.read(client)
> +      worker.request_start = Time.now.to_i
> +      worker.request_uri = env["REQUEST_URI"]
> +      response = app.call(env)

And instead do it as middleware which requires opting in:

  # XXX Totally untested code written at ~1:30 AM, may not even compile
  # this middleware is only useful for Unicorn
  module Unicorn
  class Info < Struct.new(:app, :body)
    TIME_KEY = "unicorn.info_time".freeze

    # +env+ passed is always Unicorn::HttpRequest::REQ when used with Unicorn
    def call(env)
      env[TIME_KEY] = Time.now.to_i
      status, headers, self.body = app.call(env)
    end

    # this is a no-op proxy method, we only have this so we can
    # wrap the "close" method later on...
    def each(&block)
      body.each(&block)
    end

    # http_response.rb will call this in an ensure statement,
    # we'll finalize our per-request data here
    def close
      # Unicorn has only a single env hash to pass to Rack that always
      # gets reused across requests, this micro-optimization makes it
      # less likely to require expensive rehashing/resizing every
      # request.  Normally Unicorn will wait until the start of the next
      # request to clear this.
      HttpRequest::REQ.clear[TIME_KEY] = Time.now.to_i

      body.close if body.respond_to?(:close)
    end
  end # class Info
  end # module Unicorn

>        if 100 == response.first.to_i
>          client.write(Const::EXPECT_100_RESPONSE)
> @@ -641,6 +646,9 @@ module Unicorn
>        HttpResponse.write(client, response, HttpRequest::PARSER.headers?)
>      rescue => e
>        handle_error(client, e)
> +    ensure
> +      worker.request_start = Time.now.to_i
> +      worker.request_uri = nil
>      end


> +    # used to see what a worker is doing when it's sent an :INFO signal
> +    def worker_info(worker)
> +      "worker[#{worker.nr}] - #{worker.request_uri || "idle"} - #{(Time.now.to_i - worker.request_start).to_s if worker.request_start}"

No long lines, please, I can only work in 80 column terminals.

Also, the prevalent logger calls use "worker=#{worker.nr}" and not
"worker[#{worker.nr}]"

Since Unicorn::HttpRequest::REQ is global (ZOI rule), we can just read
that hash directly, no need to stash stuff in the worker object.
Based on the proposed middleware, we can have this.

     trap(:INFO) do
       t0 = HttpRequest::REQ[Info::TIME_KEY]
       logger.info "worker=#{worker.nr} - " \
                   "#{HttpRequest::REQ['REQUEST_URI'] || 'idle'} - " \
                   "#{(Time.now.to_i - t0) if t0}"
     end if defined?(Unicorn::Info)

Thanks for digging into the code, let me know what you think of the
proposed middleware solution.

-- 
Eric Wong

From jamie at tramchase.com  Wed Apr 28 00:05:54 2010
From: jamie at tramchase.com (Jamie Wilkinson)
Date: Tue, 27 Apr 2010 21:05:54 -0700
Subject: funky process tree + stillborn masters
In-Reply-To: <20100419182146.GA22994@dcvr.yhbt.net>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
	<F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
	<20100413052410.GA31794@dcvr.yhbt.net>
	<20100419182146.GA22994@dcvr.yhbt.net>
Message-ID: <B26A4E0B-9155-40A1-B53C-D1F6521DA20B@tramchase.com>

Update from the trenches: I've traced this down to the newrelic_rpm agent

Noticed this is not the 1st time this has been brought up, since newrelic spins up the stats collector in its own thread.

Attempted the (old) logger mutex monkeypatch mentioned in the unicorn docs without luck. Noodled with various permutations of NewRelic::Agent.shutdown in before/after_fork without success. NewRelic apparently has some compat issues with bundler, but that didn't affect it, nor did switching to the plugin.

I'm running the latest newrelic_rpm agent (2.11.2) and the latest unicorn (0.97.1). 

I imagine this is contention over its logfile. Is there any low-hanging fruit I should try?

I've also filed a bug with NewRelic:
http://support.newrelic.com/discussions/support/2577-newrelic-agentbundler-causing-stillborn-unicorn-processes?unresolve=true

-jamie

On Apr 19, 2010, at 11:21 AM, Eric Wong wrote:

> Hi Jamie, any more news on this issue?  I plan on doing a minor release
> later today or tomorrow with a few minor bug fixes in unicorn.git
> 
> -- 
> Eric Wong


From normalperson at yhbt.net  Wed Apr 28 03:40:02 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 28 Apr 2010 07:40:02 +0000
Subject: funky process tree + stillborn masters
In-Reply-To: <B26A4E0B-9155-40A1-B53C-D1F6521DA20B@tramchase.com>
References: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com>
	<20100408235548.GA11184@dcvr.yhbt.net>
	<20100409012050.GA31641@dcvr.yhbt.net>
	<20100412025200.GA29391@dcvr.yhbt.net>
	<F04EF571-4CCF-425F-82BC-C2363308C6C4@tramchase.com>
	<20100413052410.GA31794@dcvr.yhbt.net>
	<20100419182146.GA22994@dcvr.yhbt.net>
	<B26A4E0B-9155-40A1-B53C-D1F6521DA20B@tramchase.com>
Message-ID: <20100428074002.GB8080@dcvr.yhbt.net>

Jamie Wilkinson <jamie at tramchase.com> wrote:
> Update from the trenches: I've traced this down to the newrelic_rpm
> agent
> 
> Noticed this is not the 1st time this has been brought up, since
> newrelic spins up the stats collector in its own thread.
> 
> Attempted the (old) logger mutex monkeypatch mentioned in the unicorn
> docs without luck. Noodled with various permutations of
> NewRelic::Agent.shutdown in before/after_fork without success.
> NewRelic apparently has some compat issues with bundler, but that
> didn't affect it, nor did switching to the plugin.
> 
> I'm running the latest newrelic_rpm agent (2.11.2) and the latest
> unicorn (0.97.1). 
> 
> I imagine this is contention over its logfile. Is there any
> low-hanging fruit I should try?

Hi Jamie, thanks for the follow up.

Exactly which version of Ruby + patchlevel are you using?   Some of the
1.8.7 releases had threading bugs in them which you may be hitting:

  http://redmine.ruby-lang.org/issues/show/1993
  http://www.daniel-azuma.com/blog/view/z2ysbx0e4c3it9/ruby_1_8_7_io_select_threading_bug

But ...

> I've also filed a bug with NewRelic:
> http://support.newrelic.com/discussions/support/2577-newrelic-agentbundler-causing-stillborn-unicorn-processes?unresolve=true

> # straces show there's a bad file descriptor read -- presumably logfiles.
> # I've noodled with shutting down the agent in unicorn before/after forks
> # without a lot of luck. Tried newrelic as a plugin with the same issue,
> # as well as some of the bundler fixes mentioned in the FAQ, as well as
> # the

Now that we know it's NewRelic, I suspect it could be reading from the
agent's client socket, not a log file.  You can map fd => files/sockets
with "ls -l /proc/$pid/fd" or "lsof -p $pid"

Perhaps in the before_fork hook you can try closing the TCPSocket (or
similar) that NewRelic is using.  Merely stopping the agent thread
isn't guaranteed to close the client socket properly (in fact, I can
almost guaratee it won't close the socket at the OS level).

Since you're on Linux, try putting the output of "ls -l /proc/#$$/fd" or
"lsof -p #$$" in both the after_fork+before_fork hooks to get an idea of
which descriptors are open across forks.

> # Unicorn doesn't play very nicely with threads, are there are any other
> # manual setup/teardown methods beyond NewRelic::Agent.shutdown I could
> # try to get fd's closed properly between forks?

There's nothing inherent to Unicorn that prevents it from playing nicely
with threads.  It's just not playing nicely with threaded code written
without potential fork() calls in mind.

-- 
Eric Wong

From gleb.lists at pluron.com  Wed Apr 28 20:12:40 2010
From: gleb.lists at pluron.com (Gleb Arshinov)
Date: Wed, 28 Apr 2010 17:12:40 -0700
Subject: Shared memory between workers
In-Reply-To: <v2gcc1f582e1004261546y874ecdd0yf3931d014ab3a000@mail.gmail.com>
References: <w2qcc1f582e1004260118za5609803x3753efd2ac850e4c@mail.gmail.com> 
	<20100426190338.GA27686@dcvr.yhbt.net>
	<v2gcc1f582e1004261546y874ecdd0yf3931d014ab3a000@mail.gmail.com>
Message-ID: <l2k21e991821004281712q1d701226id5e430a5ea17a68e@mail.gmail.com>

> If you use a "on memory database" there wouldn't be filesystem locking, right?

fsync throughput (a.k.a. commit rate) is easy to test:

  sudo aptitude install sysbench
  sysbench --test=fileio --file-fsync-freq=1 --file-num=1 \
   --file-total-size=16384 --file-test-mode=rndwr run \
   | grep "Requests/sec"

You can then see if any additional tuning (RAM drive, etc.) is even
necessary for your requirements.

Unless something is doing write-back caching in between the app and
the drive this rate will be a function of RPM of your drive, e.g.
~120/s for 7200RPM   We did a bunch of benchmarking when setting up a
couple of DB servers recently and found that consumer-grade drives
(SATA) lie and have write-back caching enabled by default, and you get
1-3k/s.  It's not considered safe to run a database on such
configuration as HDD cache lacks battery backup.  For comparison we
get ~10k/s on a 4 disk RAID 10 SAS array (w/ battery backup unit) with
write-back caching turned on in the controller and off in the drives.

So, the point is that consumer hardware lies and you are quite likely
to get good fsync performance.  Since you don't care about data safety
for your setup this should be fine.

Best regards,

Gleb