From normalperson at yhbt.net  Mon Mar  1 13:42:47 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 1 Mar 2010 18:42:47 +0000
Subject: [ANN] unicorn 0.97.0 - polishing and cleaning up
Message-ID: <20100301184247.GA22271@dcvr.yhbt.net>

Unicorn is an HTTP server for Rack applications designed to only serve
fast clients on low-latency, high-bandwidth connections and take
advantage of features in Unix/Unix-like kernels.  Slow clients should
only be served by placing a reverse proxy capable of fully buffering
both the the request and response in between Unicorn and slow clients.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

Changes:

A bunch of small fixes related to startup/configuration and hot
reload issues with HUP:

* Variables in the user-generated config.ru files no longer
  risk clobbering variables used in laucher scripts.

* signal handlers are initialized before the pid file is
  dropped, so over-eager firing of init scripts won't
  mysteriously nuke a process.

* SIGHUP will return app to original state if an updated
  config.ru fails to load due to {Syntax,Load}Error.

* unicorn_rails should be Rails 3 compatible out-of-the-box
  ('unicorn' works as always, and is recommended for Rails 3)

* unicorn_rails is finally "working_directory"-aware when
  generating default temporary paths and pid file

* config.ru encoding is the application's default in 1.9,
  not forced to binary like many parts of Unicorn.

* configurator learned to handle the "user" directive outside
  of after_fork hook (which will always remain supported).

There are also various internal cleanups and possible speedups.

-- 
Eric Wong

From surf at theflow.de  Mon Mar  8 15:31:59 2010
From: surf at theflow.de (Florian Munz)
Date: Mon, 8 Mar 2010 21:31:59 +0100
Subject: Monit configuration
Message-ID: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>

Hi,

is anyone using monit together with Unicorn?

I'd like to monitor the Unicorn child processes for memory growth. But
monit needs a pid file for each process and since Unicorn only writes a
pid file for the master this becomes a bit complicated.

I know bluepill and god can handle this but I was wondering if anyone
has got this working somehow before I start looking into these.


thanks,
Florian


From ibc at aliax.net  Mon Mar  8 16:03:46 2010
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 8 Mar 2010 22:03:46 +0100
Subject: Monit configuration
In-Reply-To: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
References: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
Message-ID: <201003082203.47100.ibc@aliax.net>

El Lunes 08 Marzo 2010, Florian Munz escribi?:
> Hi,
> 
> is anyone using monit together with Unicorn?
> 
> I'd like to monitor the Unicorn child processes for memory growth. 

> But monit needs a pid file for each process

Why do you say that? I use monit to monitorize some forking servers and it 
takes into accoutn of the child processes. The "trick" is telling monit to 
inspect the master process pid (that present unde /var/lib/SERVER.pid).




-- 
I?aki Baz Castillo <ibc at aliax.net>

From surf at theflow.de  Mon Mar  8 17:12:20 2010
From: surf at theflow.de (Florian Munz)
Date: Mon, 8 Mar 2010 23:12:20 +0100
Subject: Monit configuration
References: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
	<201003082203.47100.ibc@aliax.net>
Message-ID: <1jf25i8.pgqtmsdlxuvgN%surf@theflow.de>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> Why do you say that? I use monit to monitorize some forking servers and it
> takes into accoutn of the child processes. The "trick" is telling monit to
> inspect the master process pid (that present unde /var/lib/SERVER.pid).

Interesting. From the documentation I can't figure out how you can
monitor the children individually though.

I can test for the memory of the master process and I can test the
memory size of master plus all children. But want I want to do is
basically: Sent a QUIT to the child process if its memory size is
greater than 300MB.


- Florian


From win at wincent.com  Mon Mar  8 16:30:57 2010
From: win at wincent.com (Wincent Colaiuta)
Date: Mon, 8 Mar 2010 22:30:57 +0100
Subject: Monit configuration
In-Reply-To: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
References: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
Message-ID: <C7F5BD41-ADC6-4CF4-A7A1-67776C2AAD48@wincent.com>

El 08/03/2010, a las 21:31, Florian Munz escribi?:

> Hi,
> 
> is anyone using monit together with Unicorn?
> 
> I'd like to monitor the Unicorn child processes for memory growth. But
> monit needs a pid file for each process and since Unicorn only writes a
> pid file for the master this becomes a bit complicated.

I'm using Monit to monitor Unicorn.

I use the "totalmem" and "totalcpu" keywords to watch the usage of the master process and all child processes.

Previously I used Monit to monitor a bunch of Mongrel servers instead. Those used to stop responding every few days, but I've had no failures nor memory problems since switching to Unicorn.

Wincent




From ibc at aliax.net  Mon Mar  8 17:28:54 2010
From: ibc at aliax.net (=?iso-8859-1?q?I=F1aki_Baz_Castillo?=)
Date: Mon, 8 Mar 2010 23:28:54 +0100
Subject: Monit configuration
In-Reply-To: <1jf25i8.pgqtmsdlxuvgN%surf@theflow.de>
References: <1jf20ty.f9196k1pdqne4N%surf@theflow.de>
	<201003082203.47100.ibc@aliax.net>
	<1jf25i8.pgqtmsdlxuvgN%surf@theflow.de>
Message-ID: <201003082328.55061.ibc@aliax.net>

El Lunes 08 Marzo 2010, Florian Munz escribi?:
> I can test for the memory of the master process and I can test the
> memory size of master plus all children. But want I want to do is
> basically: Sent a QUIT to the child process if its memory size is
> greater than 300MB.

Ok, that's not possible with Monit (AFAIK).

Well, you could create a script that stores in some file the Unicorn's child 
processes pids, so Monit could inspect them.


-- 
I?aki Baz Castillo <ibc at aliax.net>

From sunaku at gmail.com  Mon Mar  8 17:58:10 2010
From: sunaku at gmail.com (Suraj Kurapati)
Date: Mon, 8 Mar 2010 14:58:10 -0800
Subject: feature request - when_ready() hook
In-Reply-To: <cfbcd2f00911301547i3cacbcdco6f2ee808fe9108ad@mail.gmail.com>
References: <cfbcd2f00911252050y3f43f950y77123253e1284d90@mail.gmail.com>
	<20091126060519.GC22762@dcvr.yhbt.net>
	<cfbcd2f00911261105y58d176d7p40f9ad2c81712b21@mail.gmail.com>
	<20091126195348.GB26561@dcvr.yhbt.net>
	<cfbcd2f00911301547i3cacbcdco6f2ee808fe9108ad@mail.gmail.com>
Message-ID: <cfbcd2f01003081458v4bb57b37j7e1233d212a8b1f0@mail.gmail.com>

Hi Eric,

On Mon, Nov 30, 2009 at 3:47 PM, Suraj Kurapati wrote:
> from Capistrano, I send SIGUSR2 to the existing Unicorn master (which
> will become the old Unicorn master), wait 90 seconds, and then send
> SIGQUIT to the old Unicorn master. [...] The only thing I'm worried
> about is that I'll have to keep adjusting this timeout as the
> infrastructure my app depends upon becomes slower/faster.  A
> when_ready() hook would really do wonders for me

Inspired by your solution[1] to a recent question about forking, I
solved this problem by temporarily sneaking into the build_app! method
and killing the old Unicorn master after the Rails app has been built:

  before_fork do |server, worker|
    #
    # the following allows a new master process to incrementally
    # phase out the old master process with SIGTTOU to avoid a
    # thundering herd (especially in the "preload_app false" case)
    # when doing a transparent upgrade.  The last worker spawned
    # will then kill off the old master process with a SIGQUIT.
    #
    old_pid_file = server.config[:pid].to_s + '.oldbin'

    if File.exist? old_pid_file and
      server.pid != old_pid_file and
      worker.nr == server.worker_processes-1
    then
      #
      # wait until Rails app is built and ready to serve (we do this by
      # sneaking into Unicorn's build_app! method) by the last worker
      # process inside the new Unicorn before stopping old Unicorn
      #
      orig_meth_name = :build_app!
      orig_meth_impl = server.method(orig_meth_name)

      server_metaclass = class << server; self; end
      server_metaclass.class_eval do

        # replace Unicorn's method with our own sneaky version
        define_method orig_meth_name do

          # behave like Unicorn's original method
          orig_meth_impl.call

          # do our sneaky business! (kill the old Unicorn)
          begin
            Process.kill :QUIT, File.read(old_pid_file).to_i
          rescue Errno::ENOENT, Errno::ESRCH
            # ignore
          end

          # restore Unicorn's original method
          server_metaclass.class_eval do
            define_method orig_meth_name, orig_meth_impl
          end

        end
      end
    end
  end

Thanks for your help! :)

[1]: http://article.gmane.org/gmane.comp.lang.ruby.unicorn.general/425

From lutine at artiversum.com  Tue Mar  9 13:06:34 2010
From: lutine at artiversum.com (Shop Levitra on www.fh75.com)
Date: Tue, 09 Mar 2010 19:06:34 +0100
Subject: preva ricat ed  scute
Message-ID: <4B968BEC.6060409@combiglas.com>

deton ates  shitt im  plati nizes  adora ble  canal boat  cavia r
hours  rewir es  teete r  explo it  place r  evoca tion  preva ricat ed
 envel ope  cleri c  disse minat ed  immod esty  wised  photo label ed
plow  ratsb ane  pandu rate  athir st  dummi ed


From deepfryed at gmail.com  Sun Mar 14 18:00:27 2010
From: deepfryed at gmail.com (Bharanee Rathna)
Date: Mon, 15 Mar 2010 09:00:27 +1100
Subject: unicorn segfaulting on Array.shuffle under ruby 1.9.1
Message-ID: <70b2ff111003141500r55eebf03kb0d993d9e34c5d96@mail.gmail.com>

Hi Guys,

http://gist.github.com/332266

results in a segfault when run under unicorn.  It seems to be fine
with webrick or thin. Any ideas ?

- bharanee

From normalperson at yhbt.net  Sun Mar 14 19:53:57 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 14 Mar 2010 23:53:57 +0000
Subject: unicorn segfaulting on Array.shuffle under ruby 1.9.1
In-Reply-To: <70b2ff111003141500r55eebf03kb0d993d9e34c5d96@mail.gmail.com>
References: <70b2ff111003141500r55eebf03kb0d993d9e34c5d96@mail.gmail.com>
Message-ID: <20100314235357.GC27911@dcvr.yhbt.net>

Bharanee Rathna <deepfryed at gmail.com> wrote:
> Hi Guys,
> 
> http://gist.github.com/332266
> 
> results in a segfault when run under unicorn.  It seems to be fine
> with webrick or thin. Any ideas ?

Hi Bharanee, it looks like the random number generator needs to be
reinitialized after fork()...

Try putting "rand" anywhere before shuffle, you can reproduce
it below even without Unicorn:

------------------- 8< ----------------
pid = fork do
  rand # removing this will segfault
  (1..10).to_a.shuffle
end
p Process.waitpid2(pid)
------------------- 8< ----------------

I'll see if I can fix it and follow up with a report to ruby-core/redmine

-- 
Eric Wong

From normalperson at yhbt.net  Sun Mar 14 20:44:25 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 14 Mar 2010 17:44:25 -0700
Subject: unicorn segfaulting on Array.shuffle under ruby 1.9.1
In-Reply-To: <20100314235357.GC27911@dcvr.yhbt.net>
References: <70b2ff111003141500r55eebf03kb0d993d9e34c5d96@mail.gmail.com>
	<20100314235357.GC27911@dcvr.yhbt.net>
Message-ID: <20100315004425.GA21171@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> I'll see if I can fix it and follow up with a report to ruby-core/redmine

I managed to report the issue here (and managed to not forget
the attachment, this might be a first)

http://redmine.ruby-lang.org/issues/show/2962

-- 
Eric Wong

From jfieber at adobe.com  Mon Mar 15 10:32:06 2010
From: jfieber at adobe.com (John Fieber)
Date: Mon, 15 Mar 2010 07:32:06 -0700
Subject: TCP reset and socket buffer draining responsibility
Message-ID: <D476A0C6-7D74-464D-B962-96D5EF9E5F92@adobe.com>

I have a bug where some Rack middleware that determines certain PUT/POST error responses without needing to read the request body.  On Linux, this causes a TCP reset when Unicorn closes the socket with unread data in the input buffer.  The reset preempts the delivery written but still buffered response, so client receives a simple RST and no response.   If I inject some read-buffer draining code just before the close, the problem goes away.

The RST behavior is obviously not desired, but my question is what level of the stack is responsible for making sure it doesn't happen?  It strikes me that Unicorn is in the best position to do it efficiently and reliably, but anything higher up can also do it.

Thoughts?

-john


From normalperson at yhbt.net  Mon Mar 15 13:48:00 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 15 Mar 2010 10:48:00 -0700
Subject: TCP reset and socket buffer draining responsibility
In-Reply-To: <D476A0C6-7D74-464D-B962-96D5EF9E5F92@adobe.com>
References: <D476A0C6-7D74-464D-B962-96D5EF9E5F92@adobe.com>
Message-ID: <20100315174800.GB3261@dcvr.yhbt.net>

John Fieber <jfieber at adobe.com> wrote:
> I have a bug where some Rack middleware that determines certain
> PUT/POST error responses without needing to read the request body.  On
> Linux, this causes a TCP reset when Unicorn closes the socket with
> unread data in the input buffer.  The reset preempts the delivery
> written but still buffered response, so client receives a simple RST
> and no response.   If I inject some read-buffer draining code just
> before the close, the problem goes away.
> 
> The RST behavior is obviously not desired, but my question is what
> level of the stack is responsible for making sure it doesn't happen?
> It strikes me that Unicorn is in the best position to do it
> efficiently and reliably, but anything higher up can also do it.

Hi John,

You have to use the read buffer draining code in your app or middleware.

The Unicorn::TeeInput class doesn't read off the socket until it needs
to.  This is because Rack requires the input stream to be rewindable,
and the input stream being rewindable means it must be buffered to a
temporary file which could potentially fill up the disk (especially with
chunked input that doesn't declare a Content-Length).

So Unicorn gives apps/middleware control to cut off uploading clients in
this case.

-- 
Eric Wong

From mjdavies at glam.ac.uk  Tue Mar 16 06:09:51 2010
From: mjdavies at glam.ac.uk (Matt Davies)
Date: Tue, 16 Mar 2010 10:09:51 +0000
Subject: Newbie problems
Message-ID: <8EF8E22C-951C-481E-B539-9226CB4CBA1E@glam.ac.uk>

Hello everyone

I'm trying out Unicorn at the moment, I've used nginx and passenger extensively for some time now but I fancy a change :-). 

Here's the specs

nginx version: nginx/0.7.65
ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin10.2.0]
unicorn (0.97.0)
rails 2.3.5
bundler 0.9.7

When I run unicorn_rails from within the app root it all starts fine, and nginx proxies the requests over to unicorn, all hunky dorey.

But, I can't get unicorn to start properly with a config file, even if the file is empty.

Here's the error I'm getting back when trying to start unicorn_rails

sudo unicorn_rails -c /path/to/unicorn.rb -E production -D -d
{:app=>
  #<Proc:0x000000010118dd80@/usr/local/lib/ruby/gems/1.8/gems/unicorn-0.97.0/bin/unicorn_rails:120>,
 :unicorn_options=>
  {:config_file=>"/path/to/unicorn.rb",
   :listeners=>[]},
 :daemonize=>true}
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/cache
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/pids
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sessions
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sockets
Exception `EOFError' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45 - end of file reached
master failed to start, check stderr log for details

I've switched logging on in the config file like so

logger Logger.new("log/unicorn.log")

all that shows in that log is

I, [2010-03-16T10:01:08.250818 #1824]  INFO -- : listening on addr=0.0.0.0:8080 fd=5

Nginx however is throwing this error

2010/03/16 09:21:42 [error] 1539#0: *1 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: ipaddress, server: machinename, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "machinename"

I'm sure I must be doing something very silly, I've searched the mailing list but can't find anything that is specific to this issue.

If anyone can help I'd be most appreciative.

Regards

Matt







From normalperson at yhbt.net  Tue Mar 16 14:10:28 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 16 Mar 2010 18:10:28 +0000
Subject: Newbie problems
In-Reply-To: <8EF8E22C-951C-481E-B539-9226CB4CBA1E@glam.ac.uk>
References: <8EF8E22C-951C-481E-B539-9226CB4CBA1E@glam.ac.uk>
Message-ID: <20100316181028.GA20824@dcvr.yhbt.net>

Matt Davies <mjdavies at glam.ac.uk> wrote:
> Hello everyone
> 
> I'm trying out Unicorn at the moment, I've used nginx and passenger extensively for some time now but I fancy a change :-). 
> 
> Here's the specs
> 
> nginx version: nginx/0.7.65
> ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin10.2.0]
> unicorn (0.97.0)
> rails 2.3.5
> bundler 0.9.7
> 
> When I run unicorn_rails from within the app root it all starts fine, and nginx proxies the requests over to unicorn, all hunky dorey.
> 
> But, I can't get unicorn to start properly with a config file, even if the file is empty.
> 
> Here's the error I'm getting back when trying to start unicorn_rails
> 
> sudo unicorn_rails -c /path/to/unicorn.rb -E production -D -d
> {:app=>
>   #<Proc:0x000000010118dd80@/usr/local/lib/ruby/gems/1.8/gems/unicorn-0.97.0/bin/unicorn_rails:120>,
>  :unicorn_options=>
>   {:config_file=>"/path/to/unicorn.rb",
>    :listeners=>[]},
>  :daemonize=>true}
> Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/cache
> Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/pids
> Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sessions
> Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sockets
> Exception `EOFError' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45 - end of file reached

> master failed to start, check stderr log for details

Hi Matt,

Was there anything else in the stderr output?  Does starting
unicorn_rails without "-D" work (or show you more info)?  Daemonizing
will redirect stderr to /dev/null unless you explicitly set
"stderr_path" in your config file.

> I've switched logging on in the config file like so
> 
> logger Logger.new("log/unicorn.log")
> 
> all that shows in that log is
> 
> I, [2010-03-16T10:01:08.250818 #1824]  INFO -- : listening on addr=0.0.0.0:8080 fd=5

If your application fails to load, the application probably spews
the error messages to stderr instead of the Unicorn logger, which
is only for Unicorn-only messages.

> Nginx however is throwing this error
> 
> 2010/03/16 09:21:42 [error] 1539#0: *1 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: ipaddress, server: machinename, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "machinename"

Try getting more info out of stderr first, since there appears to
be a problem with the startup.

But after that, what happens when you hit http://127.0.0.1:8080/ with
curl from that host?

> I'm sure I must be doing something very silly, I've searched the mailing list but can't find anything that is specific to this issue.

No worries.  Also, if you leave out the -D switch, you can try
truss/strace/dtrace-ing the process if you still don't get enough
information.

-- 
Eric Wong

From ghazel at gmail.com  Wed Mar 17 05:51:55 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Wed, 17 Mar 2010 02:51:55 -0700
Subject: Unicorn 0.97.0 old master is never dying
Message-ID: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com>

I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
sending a USR2 to the 0.97.0 master launches a new master and all the
workers, but never kills the old master. The new workers retry binding
their admin TCP ports forever. Killing all the Unicorns and
downgrading to returns peace to the forest.

Here are the before_fork and after_fork from my config/unicorn.rb:
http://codepad.org/5LyVtyq7

Did something change with the Unicorn master in 0.97.0 that would
require a change in my config, or is this a bug?

-Greg

From normalperson at yhbt.net  Thu Mar 18 04:40:22 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 18 Mar 2010 01:40:22 -0700
Subject: Unicorn 0.97.0 old master is never dying
In-Reply-To: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com>
References: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com>
Message-ID: <20100318084022.GA11167@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
> sending a USR2 to the 0.97.0 master launches a new master and all the
> workers, but never kills the old master. The new workers retry binding
> their admin TCP ports forever. Killing all the Unicorns and
> downgrading to returns peace to the forest.
> 
> Here are the before_fork and after_fork from my config/unicorn.rb:
> http://codepad.org/5LyVtyq7
> 
> Did something change with the Unicorn master in 0.97.0 that would
> require a change in my config, or is this a bug?
> 
> -Greg

Hi Greg,

Odd, are you passing the pid file path via the command-line?

I don't see the pid directive in the config file you posted there, so
there was nothing to check with File.exists?.  Are you using `unicorn'
or `unicorn_rails'?

There were some startup changes in 0.97.0, but I couldn't reproduce what
you're seeing with 'pid'.  That said, I'm not sure if that method of
killing old masters in the after_fork is very common.

-- 
Eric Wong

From ghazel at gmail.com  Thu Mar 18 12:35:12 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Thu, 18 Mar 2010 09:35:12 -0700
Subject: Unicorn 0.97.0 old master is never dying
In-Reply-To: <20100318084022.GA11167@dcvr.yhbt.net>
References: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com> 
	<20100318084022.GA11167@dcvr.yhbt.net>
Message-ID: <151643b21003180935h159eb627v725797a42083d1ca@mail.gmail.com>

On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
>> sending a USR2 to the 0.97.0 master launches a new master and all the
>> workers, but never kills the old master. The new workers retry binding
>> their admin TCP ports forever. Killing all the Unicorns and
>> downgrading to returns peace to the forest.
>>
>> Here are the before_fork and after_fork from my config/unicorn.rb:
>> http://codepad.org/5LyVtyq7
>>
>> Did something change with the Unicorn master in 0.97.0 that would
>> require a change in my config, or is this a bug?
>>
>> -Greg
>
> Hi Greg,
>
> Odd, are you passing the pid file path via the command-line?
>
> I don't see the pid directive in the config file you posted there, so
> there was nothing to check with File.exists?. ?Are you using `unicorn'
> or `unicorn_rails'?
>
> There were some startup changes in 0.97.0, but I couldn't reproduce what
> you're seeing with 'pid'. ?That said, I'm not sure if that method of
> killing old masters in the after_fork is very common.
>
> --
> Eric Wong
>

Eric,

I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is
the entire config file: http://codepad.org/v6lUsuzD

What is the preferred way to do an nginx-style restart? My system does
not really have the memory to start twice the number of workers.

-Greg

From james at imaj.es  Thu Mar 18 17:12:49 2010
From: james at imaj.es (James Cox)
Date: Thu, 18 Mar 2010 17:12:49 -0400
Subject: Issue with unicorn not starting via bluepill?
Message-ID: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com>

trying to start my unicorn via bluepill, and running into this :

Starting unicorn...\n{:unicorn_options=>
  {:config_file=>"/home/configs/unicorn/marketing.rb", :listeners=>[]},
 :daemonize=>true,
 :app=>
  #<Proc:0x000000001042c718@/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/bin/unicorn_rails:120>}
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/cache
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/pids
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/sessions
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/sockets
Exception `EOFError' at
/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45
- end of file reached
master failed to start, check stderr log for details


it's a rather complicated setup etc, but essentially i've got bluepill
instructing a start script which sets the right version of ruby just
before hand (i.e. the rvm stuff).

I can run it manually by going into the RAILS_ROOT and running it, but
trying to run via bluepill explodes.

any ideas?

Thanks,
James

From normalperson at yhbt.net  Fri Mar 19 04:23:35 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 19 Mar 2010 01:23:35 -0700
Subject: Unicorn 0.97.0 old master is never dying
In-Reply-To: <151643b21003180935h159eb627v725797a42083d1ca@mail.gmail.com>
References: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com>
	<20100318084022.GA11167@dcvr.yhbt.net>
	<151643b21003180935h159eb627v725797a42083d1ca@mail.gmail.com>
Message-ID: <20100319082335.GA3191@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson at yhbt.net> wrote:
> > ghazel at gmail.com wrote:
> >> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
> >> sending a USR2 to the 0.97.0 master launches a new master and all the
> >> workers, but never kills the old master. The new workers retry binding
> >> their admin TCP ports forever. Killing all the Unicorns and
> >> downgrading to returns peace to the forest.
> >>
> >> Here are the before_fork and after_fork from my config/unicorn.rb:
> >> http://codepad.org/5LyVtyq7
> >>
> >> Did something change with the Unicorn master in 0.97.0 that would
> >> require a change in my config, or is this a bug?
> >>
> >> -Greg
> >
> > Hi Greg,
> >
> > Odd, are you passing the pid file path via the command-line?
> >
> > I don't see the pid directive in the config file you posted there, so
> > there was nothing to check with File.exists?. ?Are you using `unicorn'
> > or `unicorn_rails'?
> >
> > There were some startup changes in 0.97.0, but I couldn't reproduce what
> > you're seeing with 'pid'. ?That said, I'm not sure if that method of
> > killing old masters in the after_fork is very common.
> >
> > --
> > Eric Wong
> >
> 
> Eric,
> 
> I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is
> the entire config file: http://codepad.org/v6lUsuzD

Ugh.  I can't even get to that file right now.  I actually prefer
to have config files inline so:

a) I can comment easily on them
b) I don't have to be online when reading them
   (I sync my mail to a local machine and lose connectivity often)


On a side note, maybe an strace/truss can help figure out what's
going on.  James seems to be having a similar problem in the other
thread...

> What is the preferred way to do an nginx-style restart? My system does
> not really have the memory to start twice the number of workers.

If you're doing it during low traffic periods, I would SIGTTOU away
most (maybe all but one) of your workers and then SIGHUP the master

-- 
Eric Wong

From normalperson at yhbt.net  Fri Mar 19 04:26:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 19 Mar 2010 01:26:31 -0700
Subject: Issue with unicorn not starting via bluepill?
In-Reply-To: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com>
References: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com>
Message-ID: <20100319082631.GA5672@dcvr.yhbt.net>

James Cox <james at imaj.es> wrote:
> trying to start my unicorn via bluepill, and running into this :
> 
> Starting unicorn...\n{:unicorn_options=>
>   {:config_file=>"/home/configs/unicorn/marketing.rb", :listeners=>[]},
>  :daemonize=>true,
>  :app=>
>   #<Proc:0x000000001042c718@/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/bin/unicorn_rails:120>}
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/cache
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/pids
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/sessions
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/sockets
> Exception `EOFError' at
> /usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45
> - end of file reached
> master failed to start, check stderr log for details
> 
> 
> it's a rather complicated setup etc, but essentially i've got bluepill
> instructing a start script which sets the right version of ruby just
> before hand (i.e. the rvm stuff).
> 
> I can run it manually by going into the RAILS_ROOT and running it, but
> trying to run via bluepill explodes.
> 
> any ideas?

I'm not familiar with Bluepill, it's a process monitor of some
sort.  Both you and Greg are using unicorn_rails which did get
some overhauling in Unicorn 0.97.0.  Everything seems fine on
my end, but maybe there's a piece of the puzzle I'm overlooking..

I'll dig further.

-- 
Eric Wong

From ghazel at gmail.com  Fri Mar 19 04:55:32 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Fri, 19 Mar 2010 01:55:32 -0700
Subject: Unicorn 0.97.0 old master is never dying
In-Reply-To: <20100319082335.GA3191@dcvr.yhbt.net>
References: <151643b21003170251g3cabd79bq36c9af668db574e0@mail.gmail.com> 
	<20100318084022.GA11167@dcvr.yhbt.net>
	<151643b21003180935h159eb627v725797a42083d1ca@mail.gmail.com> 
	<20100319082335.GA3191@dcvr.yhbt.net>
Message-ID: <151643b21003190155i351360bg7ce0694d80e0f94@mail.gmail.com>

>> Here is the entire config file: http://codepad.org/v6lUsuzD
>
> Ugh. ?I can't even get to that file right now. ?I actually prefer
> to have config files inline so:
>
> a) I can comment easily on them
> b) I don't have to be online when reading them
> ? (I sync my mail to a local machine and lose connectivity often)

We're still working on that internet that works when you're not
online. In the meantime:


# Sample configuration file for Unicorn (not Rack)
#
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.

require '/data/myapp/current/config/unicorn_constants'

# Use at least one worker per core if you're on a dedicated server,
# more will usually help for _short_ waits on databases/caches.
worker_processes UnicornConstants::NUM_WORKERS

# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
working_directory "/data/myapp/current" # available in 0.94.0+

# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen "/tmp/unicorn.sock", :backlog => 64
listen 8080, :tcp_nopush => true, :tries => -1

# nuke workers after X seconds instead of (60 seconds default)
timeout 300

# feel free to point this anywhere accessible on the filesystem
pid "/data/myapp/shared/pids/unicorn.pid"

# some applications/frameworks log to stderr or stdout, so prevent
# them from going to /dev/null when daemonized here:
stderr_path "/data/myapp/shared/log/unicorn.stderr.log"
stdout_path "/data/myapp/shared/log/unicorn.stdout.log"

# combine REE with "preload_app true" for memory savings
# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!

  # The following is only recommended for memory/DB-constrained
  # installations.  It is not needed if your system can house
  # twice as many worker_processes as you have configured.
  #
  # This allows a new master process to incrementally
  # phase out the old master process with SIGTTOU to avoid a
  # thundering herd (especially in the "preload_app false" case)
  # when doing a transparent upgrade.  The last worker spawned
  # will then kill off the old master process with a SIGQUIT.
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  # per-process listener ports for debugging/admin/migrations
  addr = "127.0.0.1:#{9000 + worker.nr}"
  server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)

  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection

  # if preload_app is true, then you may also want to check and
  # restart any other shared sockets/descriptors such as Memcached,
  # and Redis.  TokyoCabinet file handles are safe to reuse
  # between any number of forked children (assuming your kernel
  # correctly implements pread()/pwrite() system calls)
end



-Greg

From normalperson at yhbt.net  Fri Mar 19 04:57:59 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 19 Mar 2010 01:57:59 -0700
Subject: Issue with unicorn not starting via bluepill?
In-Reply-To: <20100319082631.GA5672@dcvr.yhbt.net>
References: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com>
	<20100319082631.GA5672@dcvr.yhbt.net>
Message-ID: <20100319085759.GA3323@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> James Cox <james at imaj.es> wrote:
> > trying to start my unicorn via bluepill, and running into this :

> > master failed to start, check stderr log for details

Both of you:

Was there anything useful in stderr_path?  The daemomized
process can never output to the terminal, so it had to
log somewhere... There's also strace, but looking at the
Unicorn code I haven't been able to find much (I'm very
sleep deprived right now, though).

@Greg: I got through to /v6lUsuzD finally, can you put some
debug statements around your before_fork hook and print out
the pid path name and whether it exists or not?

-- 
Eric Wong

From normalperson at yhbt.net  Tue Mar 23 14:54:18 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 23 Mar 2010 11:54:18 -0700
Subject: Issue with unicorn not starting via bluepill?
In-Reply-To: <20100319085759.GA3323@dcvr.yhbt.net>
References: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com>
	<20100319082631.GA5672@dcvr.yhbt.net>
	<20100319085759.GA3323@dcvr.yhbt.net>
Message-ID: <20100323185418.GA434@dcvr.yhbt.net>

Eric Wong <normalperson at yhbt.net> wrote:
> Eric Wong <normalperson at yhbt.net> wrote:
> > James Cox <james at imaj.es> wrote:
> > > trying to start my unicorn via bluepill, and running into this :
> 
> > > master failed to start, check stderr log for details
> 
> Both of you:

Hi, did you guys ever figure this out?

> Was there anything useful in stderr_path?  The daemomized
> process can never output to the terminal, so it had to
> log somewhere... There's also strace, but looking at the
> Unicorn code I haven't been able to find much (I'm very
> sleep deprived right now, though).
> 
> @Greg: I got through to /v6lUsuzD finally, can you put some
> debug statements around your before_fork hook and print out
> the pid path name and whether it exists or not?

From james at imaj.es  Wed Mar 24 15:14:12 2010
From: james at imaj.es (James Cox)
Date: Wed, 24 Mar 2010 15:14:12 -0400
Subject: Issue with unicorn not starting via bluepill?
In-Reply-To: <20100323185418.GA434@dcvr.yhbt.net>
References: <c3938dec1003181412s77c73bbfyc40450c9b8a08d7e@mail.gmail.com> 
	<20100319082631.GA5672@dcvr.yhbt.net>
	<20100319085759.GA3323@dcvr.yhbt.net> 
	<20100323185418.GA434@dcvr.yhbt.net>
Message-ID: <c3938dec1003241214h13470f7dq9f88a27adca2cdde@mail.gmail.com>

Not yet- i've been fixing other things first. Looking at this again shortly

On Tue, Mar 23, 2010 at 2:54 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Eric Wong <normalperson at yhbt.net> wrote:
>> Eric Wong <normalperson at yhbt.net> wrote:
>> > James Cox <james at imaj.es> wrote:
>> > > trying to start my unicorn via bluepill, and running into this :
>>
>> > > master failed to start, check stderr log for details
>>
>> Both of you:
>
> Hi, did you guys ever figure this out?
>
>> Was there anything useful in stderr_path? ?The daemomized
>> process can never output to the terminal, so it had to
>> log somewhere... There's also strace, but looking at the
>> Unicorn code I haven't been able to find much (I'm very
>> sleep deprived right now, though).
>>
>> @Greg: I got through to /v6lUsuzD finally, can you put some
>> debug statements around your before_fork hook and print out
>> the pid path name and whether it exists or not?
>



-- 
James Cox,
Consultant, Raconteur, Photographer, Entrepreneur
t: 07968 349990  e: james at imaj.es w: http://imaj.es/
talk: http://twitter.com/imajes photos: http://flickr.com/imajes