From ruben.grunge84 at gmail.com  Fri Oct  1 20:57:42 2010
From: ruben.grunge84 at gmail.com (=?UTF-8?B?UnViw6luIETDoXZpbGE=?=)
Date: Fri, 1 Oct 2010 19:57:42 -0500
Subject: Unicorn doesn't reload the app after the HUP signal
Message-ID: <AANLkTinNN9uyYxfotnKa3nUBr18DMF+j=zZoPiyKKVV3@mail.gmail.com>

Hi folks, I've experimented a problem while deploying my app, I've
sent the HUP signal to the master process, I've checked everything is
ok: new master and workers are spawned and the old ones are
killed(I've checked the PIDs), but the new code deployed isn't
reflected in the living site, so I've to stop and start again unicorn
in order to see the new changes, here is my unicorn config:

rails_env = ENV['RAILS_ENV'] || 'production'
worker_processes (rails_env == 'production' ? 3 : 1)
preload_app true
timeout 30
listen File.expand_path(File.join(__FILE__,
"../../tmp/sockets/unicorn.sock")), :backlog => 2048

if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end

before_fork do |server, worker|
  old_pid = RAILS_ROOT + '/tmp/pids/unicorn.pid.oldbin'
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  ActiveRecord::Base.establish_connection
  begin
    worker.user('peruautos', 'www-data') if Process.euid == 0
  rescue => e
    if RAILS_ENV == 'development'
      STDERR.puts "couldn't change user, oh well"
    else
      raise e
    end
  end
end

I'm using unicorn 1.1.2 and rack 1.0.1

Regards.

From normalperson at yhbt.net  Sat Oct  2 04:13:50 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 2 Oct 2010 01:13:50 -0700
Subject: Unicorn doesn't reload the app after the HUP signal
In-Reply-To: <AANLkTinNN9uyYxfotnKa3nUBr18DMF+j=zZoPiyKKVV3@mail.gmail.com>
References: <AANLkTinNN9uyYxfotnKa3nUBr18DMF+j=zZoPiyKKVV3@mail.gmail.com>
Message-ID: <20101002081350.GA13954@dcvr.yhbt.net>

Rub?n D?vila <ruben.grunge84 at gmail.com> wrote:
> Hi folks, I've experimented a problem while deploying my app, I've
> sent the HUP signal to the master process, I've checked everything is
> ok: new master and workers are spawned and the old ones are
> killed(I've checked the PIDs), but the new code deployed isn't
> reflected in the living site, so I've to stop and start again unicorn
> in order to see the new changes, here is my unicorn config:
> 
> rails_env = ENV['RAILS_ENV'] || 'production'
> worker_processes (rails_env == 'production' ? 3 : 1)
> preload_app true
> timeout 30
> listen File.expand_path(File.join(__FILE__,
> "../../tmp/sockets/unicorn.sock")), :backlog => 2048

Hi Rub?n,

Are you using cap or something else that makes symlinks during
deployment?  It could've been started in an old release directory while
it didn't have the "current" symlink pointing to it.  You can work
around this by putting:

  working_directory "/path/to/app/current"

in your config file.

<snip>

> I'm using unicorn 1.1.2 and rack 1.0.1

From jpr5 at darkridge.com  Sat Oct  2 12:38:02 2010
From: jpr5 at darkridge.com (Jordan Ritter)
Date: Sat, 2 Oct 2010 09:38:02 -0700
Subject: Problem with binding UNIX listeners before checking PID 
Message-ID: <8D95A44B-A098-43BE-B532-7D74BD957F31@darkridge.com>

Howdy.

I have lately been frustrated by the following use case:

	1. Run nginx/unicorn in production, listening on a UNIX socket with a defined pid file.  Things run good. 
	2. Someone pushes code, unicorn restarts just fine, workers are all up and running.  
	3. But someone is suspicious, or maybe they forget which box they're logged into, so they invoke unicorn manually.  Same directory, same settings.

	4. It looks like the pid file check kicked in, because unicorn refuses to boot - hey, it's already running, bugger off.  great.
	5. BUT, this happened *after* the listener processing: the manually-invoked unicorn unlinks the real unicorn master's unix listener, so it's left dead in the water and everybody loses.  

unicorn master doesn't know its listener is actually gone (but lsof shows open unix socket fd, netstat shows unix socket still present, so cursory investigation is misleading), but nginx keeps spewing ECONNREFUSEDs because the unix socket it's hitting belongs to that accidental unicorn instance that already decided not to stick around.

I think this is effectively about a behavioral difference in Unicorn::SocketHelper#bind_listen around the handling of UNIX vs. TCP sockets (this doesn't happen with TCP sockets because there's no unlink/disconnect step), and the fact that HttpServer#start evaluates the listener config before the PID path/config.

Now I see comments in and around HttpServer#initialize talking about races wrt binding to the listener and whatnot, and being newish to the codebase I admit I haven't yet fully absorbed all the considerations at play. 

But I think it's fair to say that killing the listener(s) (in the UNIX socket case) before discovering you shouldn't have run in the first place (from the PID file) qualifies as buggy/bad/broken behavior.  

I might suggest simply swapping their processing order in #start, but given the complexity of in-place restarts and other race considerations, I have doubts solving this would be that easy.

Any thoughts/ideas?

cheers,
--jordan



From ruben.grunge84 at gmail.com  Sat Oct  2 20:25:13 2010
From: ruben.grunge84 at gmail.com (=?UTF-8?B?UnViw6luIETDoXZpbGE=?=)
Date: Sat, 2 Oct 2010 19:25:13 -0500
Subject: Unicorn doesn't reload the app after the HUP signal
In-Reply-To: <20101002081350.GA13954@dcvr.yhbt.net>
References: <AANLkTinNN9uyYxfotnKa3nUBr18DMF+j=zZoPiyKKVV3@mail.gmail.com>
	<20101002081350.GA13954@dcvr.yhbt.net>
Message-ID: <AANLkTi=-=BFzVqBAShr+SiGc7y-Y_FkpoU0snRX751gL@mail.gmail.com>

Yes, I was using capistrano, very thanks.

From ben at outright.com  Sun Oct  3 18:03:41 2010
From: ben at outright.com (Ben Curren)
Date: Sun, 3 Oct 2010 15:03:41 -0700
Subject: Unicorn Signal Handling Shared for Other Servers
Message-ID: <AANLkTikMSD=-1BwLe_Dtu6ybCOABcFkVk_ZU9pDP2D8K@mail.gmail.com>

I was curious if anyone has attempted to, or is interested, it ripping
out Unicorn's signal handling and daemon code to make it possible to
share with other Ruby Servers. I really love have 0 down time deploys
and would love to have this in workling, thrift or other custom
servers we have written. I'm more than willing to fork the code and
take stab at it but wanted to check with the group to see if something
has already been attempted. I'm not the type who likes to reinvent the
wheel.

-- 
Ben Curren
CoFounder
Outright.com

Blog: blog.outright.com
Twitter: twitter.com/bcurren

From normalperson at yhbt.net  Mon Oct  4 00:17:13 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 4 Oct 2010 04:17:13 +0000
Subject: Problem with binding UNIX listeners before checking PID
In-Reply-To: <8D95A44B-A098-43BE-B532-7D74BD957F31@darkridge.com>
References: <8D95A44B-A098-43BE-B532-7D74BD957F31@darkridge.com>
Message-ID: <20101004041713.GA28709@dcvr.yhbt.net>

Jordan Ritter <jpr5 at darkridge.com> wrote:
> Howdy.
> 
> I have lately been frustrated by the following use case:
> 
> 	1. Run nginx/unicorn in production, listening on a UNIX socket
> 	with a defined pid file.  Things run good.
> 	2. Someone pushes code, unicorn restarts just fine, workers are
> 	all up and running.
> 	3. But someone is suspicious, or maybe they forget which
> 	box they're logged into, so they invoke unicorn manually.
> 	Same directory, same settings.
> 
> 	4. It looks like the pid file check kicked in, because unicorn
> 	refuses to boot - hey, it's already running, bugger off.  great.
> 	5. BUT, this happened *after* the listener processing: the
> 	manually-invoked unicorn unlinks the real unicorn master's unix
> 	listener, so it's left dead in the water and everybody loses.
> 
> unicorn master doesn't know its listener is actually gone (but lsof shows
> open unix socket fd, netstat shows unix socket still present, so cursory
> investigation is misleading), but nginx keeps spewing ECONNREFUSEDs
> because the unix socket it's hitting belongs to that accidental unicorn
> instance that already decided not to stick around.
> 
> I think this is effectively about a behavioral difference in
> Unicorn::SocketHelper#bind_listen around the handling of UNIX vs. TCP
> sockets (this doesn't happen with TCP sockets because there's no
> unlink/disconnect step), and the fact that HttpServer#start evaluates
> the listener config before the PID path/config.
> 
> Now I see comments in and around HttpServer#initialize talking about races
> wrt binding to the listener and whatnot, and being newish to the codebase
> I admit I haven't yet fully absorbed all the considerations at play.
> 
> But I think it's fair to say that killing the listener(s) (in the UNIX
> socket case) before discovering you shouldn't have run in the first place
> (from the PID file) qualifies as buggy/bad/broken behavior.

Hi Jordan,

Thanks for the detailed bug report.  I knew from experience with other
daemons that lingering UNIX sockets caused troubles for some users, but
I failed to take into account the case where a user mistakenly starting
the process twice.

Yes, getting pid file writing/ordering "right"[1] is very tricky.

> I might suggest simply swapping their processing order in #start, but
> given the complexity of in-place restarts and other race considerations,
> I have doubts solving this would be that easy.

That wouldn't work if pid files weren't in use at all.

> Any thoughts/ideas?

A simpler check would be to use connect(2) (but not make any HTTP request)
to see if the socket is alive.  Patch coming.

[1] - I don't believe there actually is a way to always be right,
      just less bad/broken than the alternatives.
-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct  4 00:22:58 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 4 Oct 2010 04:22:58 +0000
Subject: [PATCH] avoid unlinking actively listening sockets
In-Reply-To: <20101004041713.GA28709@dcvr.yhbt.net>
References: <8D95A44B-A098-43BE-B532-7D74BD957F31@darkridge.com>
	<20101004041713.GA28709@dcvr.yhbt.net>
Message-ID: <20101004042258.GA30932@dcvr.yhbt.net>

While we've always unlinked dead sockets from nuked/leftover
processes, blindly unlinking them can cause unnecessary failures
when an active process is already listening on them.  We now
make a simple connect(2) check to ensure the socket is not in
use before unlinking it.

Thanks to Jordan Ritter for the detailed bug report leading to
this fix.

ref: http://mid.gmane.org/8D95A44B-A098-43BE-B532-7D74BD957F31 at darkridge.com
---

  Eric Wong <normalperson at yhbt.net> wrote:
  > A simpler check would be to use connect(2) (but not make any HTTP request)
  > to see if the socket is alive.  Patch coming.

  s/simpler/better/

  Also pushed out to "master".  I guess a 1.1.4 release with this fix
  only is on the way since there isn't much else to release, yet.

 lib/unicorn/socket_helper.rb    |   10 ++++-
 t/t0011-active-unix-socket.sh   |   79 +++++++++++++++++++++++++++++++++++++++
 test/unit/test_socket_helper.rb |    9 ++++-
 3 files changed, 95 insertions(+), 3 deletions(-)
 create mode 100644 t/t0011-active-unix-socket.sh

diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index 9a155e1..1d03eab 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -111,8 +111,14 @@ module Unicorn
       sock = if address[0] == ?/
         if File.exist?(address)
           if File.socket?(address)
-            logger.info "unlinking existing socket=#{address}"
-            File.unlink(address)
+            begin
+              UNIXSocket.new(address).close
+              # fall through, try to bind(2) and fail with EADDRINUSE
+              # (or succeed from a small race condition we can't sanely avoid).
+            rescue Errno::ECONNREFUSED
+              logger.info "unlinking existing socket=#{address}"
+              File.unlink(address)
+            end
           else
             raise ArgumentError,
                   "socket=#{address} specified but it is not a socket!"
diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh
new file mode 100644
index 0000000..6f9ac53
--- /dev/null
+++ b/t/t0011-active-unix-socket.sh
@@ -0,0 +1,79 @@
+#!/bin/sh
+. ./test-lib.sh
+t_plan 11 "existing UNIX domain socket check"
+
+read_pid_unix () {
+	x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \
+	    socat - UNIX:$unix_socket | \
+	    tail -1)
+	test -n "$x"
+	y="$(expr "$x" : '\([0-9]\+\)')"
+	test x"$x" = x"$y"
+	test -n "$y"
+	echo "$y"
+}
+
+t_begin "setup and start" && {
+	rtmpfiles unix_socket unix_config
+	rm -f $unix_socket
+	unicorn_setup
+	grep -v ^listen < $unicorn_config > $unix_config
+	echo "listen '$unix_socket'" >> $unix_config
+	unicorn -D -c $unix_config pid.ru
+	unicorn_wait_start
+	orig_master_pid=$unicorn_pid
+}
+
+t_begin "get pid of worker" && {
+	worker_pid=$(read_pid_unix)
+	t_info "worker_pid=$worker_pid"
+}
+
+t_begin "fails to start with existing pid file" && {
+	rm -f $ok
+	unicorn -D -c $unix_config pid.ru || echo ok > $ok
+	test x"$(cat $ok)" = xok
+}
+
+t_begin "worker pid unchanged" && {
+	test x"$(read_pid_unix)" = x$worker_pid
+	> $r_err
+}
+
+t_begin "fails to start with listening UNIX domain socket bound" && {
+	rm $ok $pid
+	unicorn -D -c $unix_config pid.ru || echo ok > $ok
+	test x"$(cat $ok)" = xok
+	> $r_err
+}
+
+t_begin "worker pid unchanged (again)" && {
+	test x"$(read_pid_unix)" = x$worker_pid
+}
+
+t_begin "nuking the existing Unicorn succeeds" && {
+	kill -9 $unicorn_pid $worker_pid
+	while kill -0 $unicorn_pid
+	do
+		sleep 1
+	done
+	check_stderr
+}
+
+t_begin "succeeds in starting with leftover UNIX domain socket bound" && {
+	test -S $unix_socket
+	unicorn -D -c $unix_config pid.ru
+	unicorn_wait_start
+}
+
+t_begin "worker pid changed" && {
+	test x"$(read_pid_unix)" != x$worker_pid
+}
+
+t_begin "killing succeeds" && {
+	kill $unicorn_pid
+}
+
+t_begin "no errors" && check_stderr
+
+t_done
diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb
index bbce359..c6d0d42 100644
--- a/test/unit/test_socket_helper.rb
+++ b/test/unit/test_socket_helper.rb
@@ -101,7 +101,14 @@ class TestSocketHelper < Test::Unit::TestCase
 
   def test_bind_listen_unix_rebind
     test_bind_listen_unix
-    new_listener = bind_listen(@unix_listener_path)
+    new_listener = nil
+    assert_raises(Errno::EADDRINUSE) do
+      new_listener = bind_listen(@unix_listener_path)
+    end
+    assert_nothing_raised do
+      File.unlink(@unix_listener_path)
+      new_listener = bind_listen(@unix_listener_path)
+    end
     assert UNIXServer === new_listener
     assert new_listener.fileno != @unix_listener.fileno
     assert_equal sock_name(new_listener), sock_name(@unix_listener)
-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct  4 01:41:24 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 4 Oct 2010 05:41:24 +0000
Subject: Unicorn Signal Handling Shared for Other Servers
In-Reply-To: <AANLkTikMSD=-1BwLe_Dtu6ybCOABcFkVk_ZU9pDP2D8K@mail.gmail.com>
References: <AANLkTikMSD=-1BwLe_Dtu6ybCOABcFkVk_ZU9pDP2D8K@mail.gmail.com>
Message-ID: <20101004054124.GA31068@dcvr.yhbt.net>

Ben Curren <ben at outright.com> wrote:
> I was curious if anyone has attempted to, or is interested, it ripping
> out Unicorn's signal handling and daemon code to make it possible to
> share with other Ruby Servers. I really love have 0 down time deploys
> and would love to have this in workling, thrift or other custom
> servers we have written. I'm more than willing to fork the code and
> take stab at it but wanted to check with the group to see if something
> has already been attempted. I'm not the type who likes to reinvent the
> wheel.

Hi Ben,

I recall some attempts to make libraries around what Unicorn did a year
ago when Unicorn started picking up more press.  I'm not sure how far
any of them got and can't remember the names, but October/November 2009
was the time frame.

Personally, I think the core Ruby methods around Unix APIs are
"just right" and extra wrappers either:

  a) take away needed flexibility
     (no, I don't want umask 0000 with my daemonize())

  b) require as much effort to learn as the Unix APIs
     (Process.daemon() in 1.9, really?)

The Unix APIs themselves are well-documented in manpages, textbooks,
and needless to say, useful outside of Ruby[1].

Ruby already does a great job of hiding the tedious, more painful parts
of the original C APIs from you.  Things like trap(signal, &block),
IO.select, and exceptions (vs writing error check for *everything*)
already make life much easier than before.


For Unicorn (and Rainbows!) development, one huge (maybe the biggest)
factor in trusting it is having integration tests that
start/stop/restart/thrash/nuke the server in various ways a user would.
The test suite hits signal handling logic, directory changes, process
creation/reaping, log file rotation, pid files, etc.  Some parts of the
test suite are ugly (test_exec.rb), but nevertheless effective at
finding bugs.


[1] - I'll admit to being reasonably experienced with Unix programming
      and worked on several daemons before I picked up Ruby.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Oct  4 16:37:41 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 4 Oct 2010 20:37:41 +0000
Subject: [ANN] unicorn 1.1.4 - small bug fix and doc updates
Message-ID: <20101004203741.GA28123@dcvr.yhbt.net>

Changes:

We no longer unlinking actively listening sockets upon startup
(but continue to unlink dead ones).  This bug could trigger
downtime and nginx failures if a user makes an error and
attempts to start Unicorn while it is already running.

Thanks to Jordan Ritter for the detailed bug report leading to
this fix.

ref: http://mid.gmane.org/8D95A44B-A098-43BE-B532-7D74BD957F31 at darkridge.com

There are also minor documentation and test updates pulled in
from master.  This is hopefully the last bugfix release of the
1.1.x series.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong

From normalperson at yhbt.net  Tue Oct  5 21:53:27 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 6 Oct 2010 01:53:27 +0000
Subject: [ANN] unicorn 2.0.0pre1 - a boring "major" release
Message-ID: <20101006015327.GA18868@dcvr.yhbt.net>

Changes:

Mostly internal cleanups for future versions of Rainbows! and
people trying out Rubinius.  There are tiny performance
improvements for Ruby 1.9.2 users which may only be noticeable
with Rainbows!.

There is a new dependency on the "kgio" library for kinder,
gentler I/O :)  Please report any bugs and portability issues
with kgio to the Unicorn mailing list[1].

Unicorn 1.1.x users are NOT required nor even encouraged to
upgrade yet.  Unicorn 1.1.x will be maintained for the
forseeable future.

[1] - mongrel-unicorn at rubyforge.org


I'm holding off on supporting massively multicore machines with
a few hundred processors until a later release.  Expect
Rainbows! to see some changes/improvements in the days to come,
too.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong

From joe at citizenlogistics.com  Thu Oct  7 22:26:15 2010
From: joe at citizenlogistics.com (Joe Edelman)
Date: Thu, 7 Oct 2010 19:26:15 -0700
Subject: ANNC: unicorn_horn gem
Message-ID: <AANLkTim5z55SuB170s-ro3MQvJDgOqHM13Dv6-rmA-yM@mail.gmail.com>

Hi folks,

I've extracted the child process monitoring code from Unicorn, which
is rock solid, into its own gem:

?? http://github.com/jxe/unicorn_horn

You just pass it an enumerable of callables, and it will launch each
in its own child process, restart them quickly if they die, reap them
all on signals, etc.

It will even set $0 in the child processes to meaningful names.

You'll love it.

It's the basis of a new worker process implementation.

--Joe

--
Joe Edelman
Citizen Logistics, Inc.
http://groundcrew.us

From curtis.schofield at gmail.com  Fri Oct  8 00:24:37 2010
From: curtis.schofield at gmail.com (Curtis j Schofield)
Date: Thu, 7 Oct 2010 21:24:37 -0700
Subject: ANNC: unicorn_horn gem
In-Reply-To: <AANLkTim5z55SuB170s-ro3MQvJDgOqHM13Dv6-rmA-yM@mail.gmail.com>
References: <AANLkTim5z55SuB170s-ro3MQvJDgOqHM13Dv6-rmA-yM@mail.gmail.com>
Message-ID: <AANLkTikeR=KGUMTLyQ0m-re_dmd2jpE1OH1tU2X9kuyY@mail.gmail.com>

Great moove!


On Thu, Oct 7, 2010 at 7:26 PM, Joe Edelman <joe at citizenlogistics.com> wrote:
> Hi folks,
>
> I've extracted the child process monitoring code from Unicorn, which
> is rock solid, into its own gem:
>
> ?? http://github.com/jxe/unicorn_horn
>
> You just pass it an enumerable of callables, and it will launch each
> in its own child process, restart them quickly if they die, reap them
> all on signals, etc.
>
> It will even set $0 in the child processes to meaningful names.
>
> You'll love it.
>
> It's the basis of a new worker process implementation.
>
> --Joe
>
> --
> Joe Edelman
> Citizen Logistics, Inc.
> http://groundcrew.us
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>



-- 
make haste slowly \
festina lente ?\

From my.courtesy at att.net  Thu Oct  7 18:01:34 2010
From: my.courtesy at att.net (MICRO HEALTH.)
Date: Fri, 08 Oct 2010 06:01:34 +0800
Subject: The complete  cure for hiv1 ,hiv2 and sickle cell anaemia 
Message-ID: <USER78F3DE97F71086e5036be2456c8c07015c13ab09cd@user78f3de97f7>

Email address.. microhealth at activist.com
Phone number..+60149456790

Hello,

My name is John Shedrack Ikechukwu, im a barchelor of science degree holder in medical microbiology and  im currently in Malaysia for futher studies in the same field . Im a Nigerian by birth.I want to use this opportuinity to tell you that i have a permanent cure/formular for both hiv 1 , hiv 2 and sickle cell anaemia or any kind of blood loss related diseases with 100% efficiency.My formular corrects or makes it possible to transform the sickle  cell anaemia (ss) into a normal active cell.This formular also makes it possible to increase exponentially the blood percentage of  patients sufferering from any effects of  blood loss.For instance if your blood percentage is 30% , my formular will boost your blood to 90% within 48 hours.With my formular there will be no need for blood transfussion since my formular can increase highly the blood percentage or hb of any patient suffering from any form of  blood loss disease  within 48 hours.I want to quickly add to you that  this formular is 100% natural and it doesnt have any side effects.
 
 Secondly i have a permanent cure for hiv1 &2 , my formular is very active and effective it can correct hiv positive status to negative with in a  week with no side effects. All this i said have been tested and proven by my team for 7 years. Im really writting to you to tell you that i will like you to  partner with me  in order to make sure that this cure /formular will be looked into and funded for massive production  and used for  the betterment of the lifes of the present and that of the future generations to come.. i will like you to patner with me in  making sure that this Formular is looked into by the respective authorities and also funded in order to produce it in large form for the utilization of the public.I will like to quickly add here that what i have is the natural ingredients and the formular, it will be modified through my key which i will give to the scientists in charge for futher confirmations and experimentations and funding proper.If you are going to help me i will be very glad and i assure you that i will do your wish in whatever you want me to do in this situation.I will be looking forward to here from you. Please  if you want to partner with me in this,contact me through my email and i will be willing to answer all you need to know about this.

Regards

SHEDRACK



From normalperson at yhbt.net  Fri Oct  8 17:37:30 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 8 Oct 2010 14:37:30 -0700
Subject: internal API changes in Unicorn 2.x
Message-ID: <20101008213730.GA30777@dcvr.yhbt.net>

Here's a non-exhaustive list of internal API changes for Unicorn 2.x.

This will NOT affect most users, only those who use/rely on the
undocumented API which we've never recommended people to use.

This mainly affects Rainbows!  development.  Nearly all of these changes
are to make life easier for Rainbows! which Unicorn 2.x is intended to
support.

* Util.tmpio -> TmpIO.new

* HttpParser and HttpRequest become the same class
  - not finalized, yet, one name will go away, probably in 3.x
  - the Rack "env" hash and "buf" String buffer belong
    to the C struct since they're always used together
  - the outward API should be mostly compatible with 1.1.x
  - PARSER, BUF, REQ constants removed from HttpRequest,
    the new HttpParser holds all these on a per-socket basis
  - HttpParser#parse method added to work on the env/buf
    elements

* TeeInput.new takes only two parameters, socket and the
  parser/request object

* HttpRequest::LOCALHOST constant removed, use Kgio::LOCALHOST

* RACK_INPUT and REMOTE_ADDR constants moved from Const to HttpRequest
  since they're unused outside of HttpRequest


If there are other apps/libraries who depend on some things we changed
and you really can't have it any other way, we _may_ try to accomodate
you if it's not too much effort.

-- 
Eric Wong

From jimmy.soho at gmail.com  Sun Oct 10 04:23:43 2010
From: jimmy.soho at gmail.com (Jimmy Soho)
Date: Sun, 10 Oct 2010 19:23:43 +1100
Subject: timestamping log entries
Message-ID: <AANLkTikta3kbVfyfPL4qwuxtJbOk4D+vznSBsA3G=cSn@mail.gmail.com>

Hi,

When I look in the unicorn.stdout.log and/or unicorn.stderr.log files
some lines are timestamped, some aren't. How could I get all lines
timestamped?


Cheers,
Jimmy


For example:

I, [2010-10-06T05:51:49.589454 #10512]  INFO -- : Refreshing Gem list
reaped #<Process::Status: pid=7850,exited(0)> worker=1
reaped #<Process::Status: pid=7851,exited(0)> worker=2
reaped #<Process::Status: pid=7849,exited(0)> worker=0
reaped #<Process::Status: pid=7852,exited(0)> worker=3
master complete
master process ready
worker=0 ready
worker=2 ready
worker=3 ready
worker=1 ready
reaped #<Process::Status: pid=10531,exited(0)> worker=0
reaped #<Process::Status: pid=10532,exited(0)> worker=1
reaped #<Process::Status: pid=10533,exited(0)> worker=2
reaped #<Process::Status: pid=10534,exited(0)> worker=3
master complete

From zblut at cerego.co.jp  Mon Oct 11 22:20:45 2010
From: zblut at cerego.co.jp (Zev Blut)
Date: Tue, 12 Oct 2010 11:20:45 +0900
Subject: timestamping log entries
In-Reply-To: <AANLkTikta3kbVfyfPL4qwuxtJbOk4D+vznSBsA3G=cSn@mail.gmail.com>
References: <AANLkTikta3kbVfyfPL4qwuxtJbOk4D+vznSBsA3G=cSn@mail.gmail.com>
Message-ID: <4CB3C5FD.4000409@cerego.co.jp>

Hello,

On 10/10/2010 05:23 PM, Jimmy Soho wrote:

> When I look in the unicorn.stdout.log and/or unicorn.stderr.log files
> some lines are timestamped, some aren't. How could I get all lines
> timestamped?

+1
I too would really like to see some sort of time stamp for the log files.

--
Zev

From normalperson at yhbt.net  Tue Oct 12 04:43:07 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 12 Oct 2010 01:43:07 -0700
Subject: timestamping log entries
In-Reply-To: <4CB3C5FD.4000409@cerego.co.jp>
References: <AANLkTikta3kbVfyfPL4qwuxtJbOk4D+vznSBsA3G=cSn@mail.gmail.com>
	<4CB3C5FD.4000409@cerego.co.jp>
Message-ID: <20101012084307.GA14304@dcvr.yhbt.net>

Zev Blut <zblut at cerego.co.jp> wrote:
> Hello,
>
> On 10/10/2010 05:23 PM, Jimmy Soho wrote:
>
>> When I look in the unicorn.stdout.log and/or unicorn.stderr.log files
>> some lines are timestamped, some aren't. How could I get all lines
>> timestamped?
>
> +1
> I too would really like to see some sort of time stamp for the log files.

They're enabled, but Rails does something funky after it's loaded:
>From the FAQ, http://unicorn.bogomips.org/FAQ.html

| === Why are log messages from Unicorn are unformatted when using Rails?
| 
| Current versions of Rails unfortunately overrides the default Logger
| formatter.
| 
| You can undo this behavior with the default logger in your Unicorn
| config file:
| 
|   Configurator::DEFAULTS[:logger].formatter = Logger::Formatter.new
| 
| Of course you can specify an entirely different logger as well
| with the "logger" directive described by Unicorn::Configurator.

(sorry for the delayed reply, on vacation).

-- 
Eric Wong

From justin.giancola at gmail.com  Thu Oct 14 00:58:49 2010
From: justin.giancola at gmail.com (Justin Giancola)
Date: Thu, 14 Oct 2010 00:58:49 -0400
Subject: completely stumped by problem encountered replacing running unicorn
Message-ID: <AANLkTinQUf-DOWk=wzxdfZTTKrKy5Kvw+xaQZrHv8Q+Z@mail.gmail.com>

Hello,

First off, many thanks to all those who have contributed to this great
project. I have been running Unicorn in production for quite some time
now. It hass been very reliable and I have really enjoyed being able
to deploy frequently without downtime or even dropped connections.

I have used the USR2+QUIT upgrade procedure as described in SIGNALS
since I started using Unicorn and it works great. However, while
upgrading Bundler on my server the other day, an upgrade failed for
the first time. The upgrade failed due to a LoadError -- <application
root>/gems/environment.rb is no longer used in Bundler 1.0 so it was
missing from the new application tree.

I am using a Capistrano setup so my app is located at
/path/to/app/current where current is a symlink to a timestamped
release directory. I am calling #working_directory in my unicorn
config to set it to the symlinked current path. I am also calling
unicorn via a binstub generated by Bundler located at
/path/to/app/current/bin/unicorn.

I had expected that reexec would pick up the new unicorn and app code
because the path to the unicorn executable script, the path to my app,
and config files were all identical to those used in the previous
deploy. However, this is not what is happening. Rather, it looks like
the previous version of the code is trying to reload iteself in the
new app tree where files are missing.

I am having a very difficult time understanding exactly what files are
being reloaded during the reexec. Even my attempts at troubleshooting
where the load error is originating have left me completely baffled.
In the old app gems/environment.rb is required in a few places but
none of those files seem to be reloaded during the reexec. Further, I
cannot get this executable swap to work no matter what I try. I have
even tried getting rid of the symlinks entirely and changing the
contents of the /path/to/app/current in place during the upgrade.

I feel like I may be missing something obvious or else I have
completely misunderstood how the reload process works. Any insight as
to what I should be doing differently would be greatly appreciated.


Justin

From lawrence.pit at gmail.com  Thu Oct 14 01:50:29 2010
From: lawrence.pit at gmail.com (Lawrence Pit)
Date: Thu, 14 Oct 2010 16:50:29 +1100
Subject: completely stumped by problem encountered replacing running
	unicorn
In-Reply-To: <AANLkTinQUf-DOWk=wzxdfZTTKrKy5Kvw+xaQZrHv8Q+Z@mail.gmail.com>
References: <AANLkTinQUf-DOWk=wzxdfZTTKrKy5Kvw+xaQZrHv8Q+Z@mail.gmail.com>
Message-ID: <4CB69A25.3000000@gmail.com>

  Hi Justin,

See the paragraph "Other ENV pollution issues" at 
http://unicorn.bogomips.org/Sandbox.html


Cheers,
Lawrence

> Hello,
>
> First off, many thanks to all those who have contributed to this great
> project. I have been running Unicorn in production for quite some time
> now. It hass been very reliable and I have really enjoyed being able
> to deploy frequently without downtime or even dropped connections.
>
> I have used the USR2+QUIT upgrade procedure as described in SIGNALS
> since I started using Unicorn and it works great. However, while
> upgrading Bundler on my server the other day, an upgrade failed for
> the first time. The upgrade failed due to a LoadError --<application
> root>/gems/environment.rb is no longer used in Bundler 1.0 so it was
> missing from the new application tree.
>
> I am using a Capistrano setup so my app is located at
> /path/to/app/current where current is a symlink to a timestamped
> release directory. I am calling #working_directory in my unicorn
> config to set it to the symlinked current path. I am also calling
> unicorn via a binstub generated by Bundler located at
> /path/to/app/current/bin/unicorn.
>
> I had expected that reexec would pick up the new unicorn and app code
> because the path to the unicorn executable script, the path to my app,
> and config files were all identical to those used in the previous
> deploy. However, this is not what is happening. Rather, it looks like
> the previous version of the code is trying to reload iteself in the
> new app tree where files are missing.
>
> I am having a very difficult time understanding exactly what files are
> being reloaded during the reexec. Even my attempts at troubleshooting
> where the load error is originating have left me completely baffled.
> In the old app gems/environment.rb is required in a few places but
> none of those files seem to be reloaded during the reexec. Further, I
> cannot get this executable swap to work no matter what I try. I have
> even tried getting rid of the symlinks entirely and changing the
> contents of the /path/to/app/current in place during the upgrade.
>
> I feel like I may be missing something obvious or else I have
> completely misunderstood how the reload process works. Any insight as
> to what I should be doing differently would be greatly appreciated.
>
>
> Justin
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>


From justin.giancola at gmail.com  Thu Oct 14 19:01:03 2010
From: justin.giancola at gmail.com (Justin Giancola)
Date: Thu, 14 Oct 2010 19:01:03 -0400
Subject: completely stumped by problem encountered replacing running
	unicorn
In-Reply-To: <4CB69A25.3000000@gmail.com>
References: <AANLkTinQUf-DOWk=wzxdfZTTKrKy5Kvw+xaQZrHv8Q+Z@mail.gmail.com>
	<4CB69A25.3000000@gmail.com>
Message-ID: <AANLkTimZoONd1h_agW11K1Odq9yDHochSE7ZG9BdcRpB@mail.gmail.com>

Thanks, Lawrence.

ENV pollution was indeed the problem. The aforementioned exception was
being caused by by Bundler 0.7 having set ENV['RUBYOPT'] to
"-r<application root>/gems/environment.rb". After clearing that and
setting my paths properly it worked.

Thank you very much for pointing me in the right direction.


Justin

On Thu, Oct 14, 2010 at 1:50 AM, Lawrence Pit <lawrence.pit at gmail.com> wrote:
> ?Hi Justin,
>
> See the paragraph "Other ENV pollution issues" at
> http://unicorn.bogomips.org/Sandbox.html
>
>
> Cheers,
> Lawrence
>
>> Hello,
>>
>> First off, many thanks to all those who have contributed to this great
>> project. I have been running Unicorn in production for quite some time
>> now. It hass been very reliable and I have really enjoyed being able
>> to deploy frequently without downtime or even dropped connections.
>>
>> I have used the USR2+QUIT upgrade procedure as described in SIGNALS
>> since I started using Unicorn and it works great. However, while
>> upgrading Bundler on my server the other day, an upgrade failed for
>> the first time. The upgrade failed due to a LoadError --<application
>> root>/gems/environment.rb is no longer used in Bundler 1.0 so it was
>> missing from the new application tree.
>>
>> I am using a Capistrano setup so my app is located at
>> /path/to/app/current where current is a symlink to a timestamped
>> release directory. I am calling #working_directory in my unicorn
>> config to set it to the symlinked current path. I am also calling
>> unicorn via a binstub generated by Bundler located at
>> /path/to/app/current/bin/unicorn.
>>
>> I had expected that reexec would pick up the new unicorn and app code
>> because the path to the unicorn executable script, the path to my app,
>> and config files were all identical to those used in the previous
>> deploy. However, this is not what is happening. Rather, it looks like
>> the previous version of the code is trying to reload iteself in the
>> new app tree where files are missing.
>>
>> I am having a very difficult time understanding exactly what files are
>> being reloaded during the reexec. Even my attempts at troubleshooting
>> where the load error is originating have left me completely baffled.
>> In the old app gems/environment.rb is required in a few places but
>> none of those files seem to be reloaded during the reexec. Further, I
>> cannot get this executable swap to work no matter what I try. I have
>> even tried getting rid of the symlinks entirely and changing the
>> contents of the /path/to/app/current in place during the upgrade.
>>
>> I feel like I may be missing something obvious or else I have
>> completely misunderstood how the reload process works. Any insight as
>> to what I should be doing differently would be greatly appreciated.
>>
>>
>> Justin
>> _______________________________________________
>> Unicorn mailing list - mongrel-unicorn at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
>> Do not quote signatures (like this one) or top post when replying
>>
>
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>

From Postmaster at company.mail  Thu Oct 21 03:16:48 2010
From: Postmaster at company.mail (Postmaster at company.mail)
Date: Thu, 21 Oct 2010 14:16:48 +0700
Subject: MDaemon Notification -- Attachment Removed
Message-ID: <MDAEMON0016201010211416.AA1648896@company.mail>

The following message contained restricted attachment(s) which have been removed:

>From      : mongrel-unicorn at rubyforge.org
To        : quangnv at pc2.vn
Subject   : Delivery reports about your e-mail
Message-ID: 

Attachment(s) removed:
-----------------------------------------
Text.pif



From nareshov at gmail.com  Fri Oct 22 15:50:19 2010
From: nareshov at gmail.com (Naresh V)
Date: Sat, 23 Oct 2010 01:20:19 +0530
Subject: 502 bad gateway on nginx with recv() failed
Message-ID: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>

Hi,

I'm serving the puppetmaster application with its config.ru through
unicorn - proxied by nginx.
I'm using unix sockets, 4 workers, and 2048 backlog.

The clients - after their typical "puppet run" - send back a report to
the master in YAML.
Some clients whose reports tend to be large (close to 2mb) get a 502
bad gateway error and error out.

nginx log:

2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
Connection reset by peer) while reading response header from upstream,
client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
HTTP/1.1", upstream:
"http://unix:/tmp/.sock:/production/report/nagios", host:
"puppet:8140"

I was getting the same thing earlier when I had unicorn listening on
TCP sockets instead of UNIX sockets. And I had a lot of connections in
TIME_WAIT:

tcp        0      0 127.0.0.1:8141              127.0.0.1:54507
     TIME_WAIT   -
tcp        0      0 127.0.0.1:8141              127.0.0.1:57322
     TIME_WAIT   -

Fluctuating all the way from 20 to 800. A quick restart of nginx
tended to bring the number down.


-Naresh V.

From normalperson at yhbt.net  Fri Oct 22 17:14:46 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 22 Oct 2010 14:14:46 -0700
Subject: 502 bad gateway on nginx with recv() failed
In-Reply-To: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>
References: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>
Message-ID: <20101022211446.GA22603@dcvr.yhbt.net>

Naresh V <nareshov at gmail.com> wrote:
> Hi,
> 
> I'm serving the puppetmaster application with its config.ru through
> unicorn - proxied by nginx.
> I'm using unix sockets, 4 workers, and 2048 backlog.
> 
> The clients - after their typical "puppet run" - send back a report to
> the master in YAML.
> Some clients whose reports tend to be large (close to 2mb) get a 502
> bad gateway error and error out.
> 
> nginx log:
> 
> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
> Connection reset by peer) while reading response header from upstream,
> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
> HTTP/1.1", upstream:
> "http://unix:/tmp/.sock:/production/report/nagios", host:
> "puppet:8140"

Hi Naresh, do you see anything in the Unicorn stderr log file?

Is the 2mb report part of the response or request?  Unicorn should
have no problems accepting large requests (Rainbows! defaults the
client_max_body_size to 1mb, just like nginx).

It could be Unicorn's internal (default 60s) timeout kicking
in because puppet is slowly reading/generating the 2mb body.

Also, which version of Unicorn and nginx is this?

> I was getting the same thing earlier when I had unicorn listening on
> TCP sockets instead of UNIX sockets. And I had a lot of connections in
> TIME_WAIT:
> 
> tcp        0      0 127.0.0.1:8141              127.0.0.1:54507
>      TIME_WAIT   -
> tcp        0      0 127.0.0.1:8141              127.0.0.1:57322
>      TIME_WAIT   -
> 
> Fluctuating all the way from 20 to 800. A quick restart of nginx
> tended to bring the number down.

Having many TIME_WAIT sockets is normal and expected when you're
starting/stopping lots of TCP connections.  It's nothing to worry about
unless you get several thousands of requests/second, and then you should
apply tcp_tw_reuse/tcp_tw_recycle as mentioned in
http://unicorn.bogomips.org/TUNING.html (or switch to UNIX domain
sockets and use nginx for keepalive).

-- 
Eric Wong

From nareshov at gmail.com  Sat Oct 23 00:48:07 2010
From: nareshov at gmail.com (Naresh V)
Date: Sat, 23 Oct 2010 10:18:07 +0530
Subject: 502 bad gateway on nginx with recv() failed
In-Reply-To: <20101022211446.GA22603@dcvr.yhbt.net>
References: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>
	<20101022211446.GA22603@dcvr.yhbt.net>
Message-ID: <AANLkTinhwsSGgYb5qF-6x8w1iT36jufE=S_CqFJZXEKv@mail.gmail.com>

On 23 October 2010 02:44, Eric Wong <normalperson at yhbt.net> wrote:
> Naresh V <nareshov at gmail.com> wrote:
>> Hi,
>>
>> I'm serving the puppetmaster application with its config.ru through
>> unicorn - proxied by nginx.
>> I'm using unix sockets, 4 workers, and 2048 backlog.
>>
>> The clients - after their typical "puppet run" - send back a report to
>> the master in YAML.
>> Some clients whose reports tend to be large (close to 2mb) get a 502
>> bad gateway error and error out.
>>
>> nginx log:
>>
>> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
>> Connection reset by peer) while reading response header from upstream,
>> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
>> HTTP/1.1", upstream:
>> "http://unix:/tmp/.sock:/production/report/nagios", host:
>> "puppet:8140"
>
> Hi Naresh, do you see anything in the Unicorn stderr log file?

Hi Eric, I think I caught it:

E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
timeout (60.207392s > 60s), killing
I, [2010-10-22T23:03:31.212533 #10184]  INFO -- : reaped
#<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
I, [2010-10-22T23:03:31.214768 #10490]  INFO -- : worker=2 spawned pid=10490
I, [2010-10-22T23:03:31.221748 #10490]  INFO -- : worker=2 ready


> Is the 2mb report part of the response or request? ?Unicorn should
> have no problems accepting large requests (Rainbows! defaults the
> client_max_body_size to 1mb, just like nginx).

It's part of the PUT request, I guess.

> It could be Unicorn's internal (default 60s) timeout kicking
> in because puppet is slowly reading/generating the 2mb body.

I raised the timeout first to 120, then 180 - and I continued to get
the 502 (with the logs as above)
When I raised it upto 240, puppetd complained:

#-(1)> puppetd -t -v  --trace
notice: Ignoring --listen on onetime run
info: Caching catalog for nagios
info: Applying configuration version '1287807847'
notice: Finished catalog run in 25.86 seconds
/usr/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:56:in `timeout'
/usr/lib/ruby/1.8/timeout.rb:76:in `timeout'
/usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1050:in `request'
/usr/lib/ruby/1.8/net/http.rb:1037:in `request'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/net/http.rb:1035:in `request'
/usr/lib/ruby/1.8/net/http.rb:857:in `put'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb:90:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:253:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector.rb:64:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:178:in `send_report'
/usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:172:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent/locker.rb:21:in `lock'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/1.8/sync.rb:229:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:39:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:103:in `with_client'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:37:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:171:in `call'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:171:in `controlled_run'
/usr/lib/ruby/site_ruby/1.8/puppet/agent.rb:35:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:114:in `onetime'
/usr/lib/ruby/site_ruby/1.8/puppet/application/agent.rb:88:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:393:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:287:in `run'
/usr/sbin/puppetd:4
err: Could not run Puppet configuration client: execution expired


> Also, which version of Unicorn and nginx is this?

unicorn (1.1.4) and nginx-0.8.49-1.el5

>> I was getting the same thing earlier when I had unicorn listening on
>> TCP sockets instead of UNIX sockets. And I had a lot of connections in
>> TIME_WAIT:
>>
>> tcp ? ? ? ?0 ? ? ?0 127.0.0.1:8141 ? ? ? ? ? ? ?127.0.0.1:54507
>> ? ? ?TIME_WAIT ? -
>> tcp ? ? ? ?0 ? ? ?0 127.0.0.1:8141 ? ? ? ? ? ? ?127.0.0.1:57322
>> ? ? ?TIME_WAIT ? -
>>
>> Fluctuating all the way from 20 to 800. A quick restart of nginx
>> tended to bring the number down.
>
> Having many TIME_WAIT sockets is normal and expected when you're
> starting/stopping lots of TCP connections. ?It's nothing to worry about
> unless you get several thousands of requests/second, and then you should
> apply tcp_tw_reuse/tcp_tw_recycle as mentioned in
> http://unicorn.bogomips.org/TUNING.html (or switch to UNIX domain
> sockets and use nginx for keepalive).
>
> --
> Eric Wong


-Naresh V.

From normalperson at yhbt.net  Sat Oct 23 19:22:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 23 Oct 2010 16:22:31 -0700
Subject: 502 bad gateway on nginx with recv() failed
In-Reply-To: <AANLkTinhwsSGgYb5qF-6x8w1iT36jufE=S_CqFJZXEKv@mail.gmail.com>
References: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>
	<20101022211446.GA22603@dcvr.yhbt.net>
	<AANLkTinhwsSGgYb5qF-6x8w1iT36jufE=S_CqFJZXEKv@mail.gmail.com>
Message-ID: <20101023232231.GC11427@dcvr.yhbt.net>

Naresh V <nareshov at gmail.com> wrote:
> On 23 October 2010 02:44, Eric Wong <normalperson at yhbt.net> wrote:
> > Naresh V <nareshov at gmail.com> wrote:
> >> I'm serving the puppetmaster application with its config.ru through
> >> unicorn - proxied by nginx.
> >> I'm using unix sockets, 4 workers, and 2048 backlog.
> >>
> >> The clients - after their typical "puppet run" - send back a report to
> >> the master in YAML.
> >> Some clients whose reports tend to be large (close to 2mb) get a 502
> >> bad gateway error and error out.
> >>
> >> nginx log:
> >>
> >> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
> >> Connection reset by peer) while reading response header from upstream,
> >> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
> >> HTTP/1.1", upstream:
> >> "http://unix:/tmp/.sock:/production/report/nagios", host:
> >> "puppet:8140"
> >
> > Hi Naresh, do you see anything in the Unicorn stderr log file?
> 
> Hi Eric, I think I caught it:
> 
> E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
> timeout (60.207392s > 60s), killing
> I, [2010-10-22T23:03:31.212533 #10184]  INFO -- : reaped
> #<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
> I, [2010-10-22T23:03:31.214768 #10490]  INFO -- : worker=2 spawned pid=10490
> I, [2010-10-22T23:03:31.221748 #10490]  INFO -- : worker=2 ready
> 
> > Is the 2mb report part of the response or request? ?Unicorn should
> > have no problems accepting large requests (Rainbows! defaults the
> > client_max_body_size to 1mb, just like nginx).
> 
> It's part of the PUT request, I guess.
> 
> > It could be Unicorn's internal (default 60s) timeout kicking
> > in because puppet is slowly reading/generating the 2mb body.
> 
> I raised the timeout first to 120, then 180 - and I continued to get
> the 502 (with the logs as above)
> When I raised it upto 240, puppetd complained:

Interesting.  I'm not familiar with Puppet internals, but is there any
valid reason it would be taking this long?

Can you tell if the Unicorn worker is doing anything (using up CPU time
in top) or just blocked on some socket connection to a database or DNS?
(strace/truss will help).

You should definitely talk to Puppet developers/users about why it's
taking so long.  HTTP requests taking anywhere near 60s is an eternity,
I wonder if your Puppet is somehow misconfigured.

-- 
Eric Wong

From nareshov at gmail.com  Sun Oct 24 02:00:52 2010
From: nareshov at gmail.com (Naresh V)
Date: Sun, 24 Oct 2010 11:30:52 +0530
Subject: 502 bad gateway on nginx with recv() failed
In-Reply-To: <20101023232231.GC11427@dcvr.yhbt.net>
References: <AANLkTimDFC1KRSKATDWSnhDay9esLmMHObx7V4FXGb_z@mail.gmail.com>
	<20101022211446.GA22603@dcvr.yhbt.net>
	<AANLkTinhwsSGgYb5qF-6x8w1iT36jufE=S_CqFJZXEKv@mail.gmail.com>
	<20101023232231.GC11427@dcvr.yhbt.net>
Message-ID: <AANLkTingjB9yFAPr1QwX5h_5Q_DKLU8hu5=K+qQd8V2z@mail.gmail.com>

On 24 October 2010 04:52, Eric Wong <normalperson at yhbt.net> wrote:
> Naresh V <nareshov at gmail.com> wrote:
>> On 23 October 2010 02:44, Eric Wong <normalperson at yhbt.net> wrote:
>> > Naresh V <nareshov at gmail.com> wrote:
>> >> I'm serving the puppetmaster application with its config.ru through
>> >> unicorn - proxied by nginx.
>> >> I'm using unix sockets, 4 workers, and 2048 backlog.
>> >>
>> >> The clients - after their typical "puppet run" - send back a report to
>> >> the master in YAML.
>> >> Some clients whose reports tend to be large (close to 2mb) get a 502
>> >> bad gateway error and error out.
>> >>
>> >> nginx log:
>> >>
>> >> 2010/10/22 14:20:27 [error] 19461#0: *17115 recv() failed (104:
>> >> Connection reset by peer) while reading response header from upstream,
>> >> client: 1x.yy.zz.x4, server: , request: "PUT /production/report/nagios
>> >> HTTP/1.1", upstream:
>> >> "http://unix:/tmp/.sock:/production/report/nagios", host:
>> >> "puppet:8140"
>> >
>> > Hi Naresh, do you see anything in the Unicorn stderr log file?
>>
>> Hi Eric, I think I caught it:
>>
>> E, [2010-10-22T23:03:30.207455 #10184] ERROR -- : worker=2 PID:10206
>> timeout (60.207392s > 60s), killing
>> I, [2010-10-22T23:03:31.212533 #10184] ?INFO -- : reaped
>> #<Process::Status: pid=10206,signaled(SIGKILL=9)> worker=2
>> I, [2010-10-22T23:03:31.214768 #10490] ?INFO -- : worker=2 spawned pid=10490
>> I, [2010-10-22T23:03:31.221748 #10490] ?INFO -- : worker=2 ready
>>
>> > Is the 2mb report part of the response or request? ?Unicorn should
>> > have no problems accepting large requests (Rainbows! defaults the
>> > client_max_body_size to 1mb, just like nginx).
>>
>> It's part of the PUT request, I guess.
>>
>> > It could be Unicorn's internal (default 60s) timeout kicking
>> > in because puppet is slowly reading/generating the 2mb body.
>>
>> I raised the timeout first to 120, then 180 - and I continued to get
>> the 502 (with the logs as above)
>> When I raised it upto 240, puppetd complained:
>
> Interesting. ?I'm not familiar with Puppet internals, but is there any
> valid reason it would be taking this long?
>
> Can you tell if the Unicorn worker is doing anything (using up CPU time
> in top) or just blocked on some socket connection to a database or DNS?
> (strace/truss will help).
>
> You should definitely talk to Puppet developers/users about why it's
> taking so long. ?HTTP requests taking anywhere near 60s is an eternity,
> I wonder if your Puppet is somehow misconfigured.
>

Yes, puppet-server was mis-configured and went unnoticed. The reports
are supposed to be accepted by another URL and that had to be
configured appropriately.

Sorry for the false alarm.

-Naresh.

From normalperson at yhbt.net  Wed Oct 27 20:19:42 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 28 Oct 2010 00:19:42 +0000
Subject: [ANN] Unicorn 2.0.0, 1.1.5, 1.0.2 released!
Message-ID: <20101028001942.GA22467@dcvr.yhbt.net>

tldr:
2.0.0 - cleanups for Rainbows!, but should be ready for general use
        barring portability issues
1.1.5 - bug fixes
1.0.2 - bug fixes

Please send any and all feedback to the mailing list.  Do not waste
bandwidth with HTML or signature attachments.

email: mongrel-unicorn at rubyforge.org
git: git://git.bogomips.org/unicorn.git

The longer change summaries/release notes:

== unicorn 2.0.0 - mostly internal cleanups

Despite the version number, this release mostly features
internal cleanups for future versions of Rainbows!.  User
visible changes include reductions in CPU wakeups on idle sites
using high timeouts.

Barring possible portability issues due to the introduction of
the kgio library, this release should be ready for all to use.
However, 1.1.x (and possibly 1.0.x) will continue to be
maintained.  Unicorn 1.1.5 and 1.0.2 have also been released
with bugfixes found during development of 2.0.0.

== unicorn 1.1.5

This maintenance release fixes several long-standing but
recently-noticed bugs.  SIGHUP reloading now correctly restores
default values if they're erased or commented-out in the Unicorn
configuration file.  Delays/slowdowns in signal handling since
0.990 are fixed, too.

== unicorn 1.0.2

This is the latest maintenance release of the 1.0.x series.
All users are encouraged to upgrade to 1.1.x stable series
and report bugs there.

Shortlog of changes since 1.0.1:

Eric Wong (8):
      SIGTTIN works after SIGWINCH
      fix delays in signal handling
      Rakefile: don't post freshmeat on empty changelogs
      Rakefile: capture prerelease tags
      configurator: use "__send__" instead of "send"
      configurator: reloading with unset values restores default
      gemspec: depend on Isolate 3.0.0 for dev
      doc: stop using deprecated rdoc CLI options

==
Have fun!

-- 
Eric Wong

From antonvesnin at gmail.com  Fri Oct 29 07:16:29 2010
From: antonvesnin at gmail.com (Anton A. Vesnin)
Date: Fri, 29 Oct 2010 15:16:29 +0400
Subject: stop/start workers "by need"
Message-ID: <201010291516.31851.antonvesnin@gmail.com>

Hello to everybody.
I making shared hosting for rails and rack applications. For app. server i 
userd phusion passenger, but after some issues with that, want to migrate to 
unicorn. 

Is there any way to make unicorn stop all workers and start them on first 
request? It'll be really great to have this opportunity. 

>No subscription is needed to post to the mailing list, let us know that we 
need to Cc: replies to you if you?re unsubscribed.
Reply with Cc:, pleace. Thank you.

--
With best regards.

From normalperson at yhbt.net  Fri Oct 29 13:40:51 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 29 Oct 2010 10:40:51 -0700
Subject: stop/start workers "by need"
In-Reply-To: <201010291516.31851.antonvesnin@gmail.com>
References: <201010291516.31851.antonvesnin@gmail.com>
Message-ID: <20101029174051.GA25861@dcvr.yhbt.net>

"Anton A. Vesnin" <antonvesnin at gmail.com> wrote:
> Hello to everybody.
> I making shared hosting for rails and rack applications. For app. server i 
> userd phusion passenger, but after some issues with that, want to migrate to 
> unicorn. 
> 
> Is there any way to make unicorn stop all workers and start them on first 
> request? It'll be really great to have this opportunity. 

Hi Anton, there's no current way to do this.

Unlike Apache/Passenger, Unicorn should only be tuned for backend
requirements, never to actual connected clients.  Unicorn isn't too
different from prefork Apache when serving requests, it's just the
"marketing" for Unicorn focuses _very_ strongly on its limitations :)

On a memory-constrained box, maybe tailing logs or similar and sending
TTIN/TTOU signals to increment/decrement workers as needed can work.  A
smarter version of this would even recognize which endpoints are more/less
expensive and tune the workers accordingly.

A dumb version of this would just be a cronjob that
increments/decrements worker depending on the time of day.  If you're
running batch processes at off hours, maybe the dumb version is all you
need :)

I'm strongly against having this logic in Unicorn itself since apps
behave wildly differently depending on how they're implemented.  And
most deployments I've supported over the years were better off with
fixed/dedicated resources and letting the OS manage things.

> Reply with Cc:, pleace. Thank you.

-- 
Eric Wong

From jamie at jamiedubs.com  Fri Oct 29 13:55:03 2010
From: jamie at jamiedubs.com (Jamie Wilkinson)
Date: Fri, 29 Oct 2010 10:55:03 -0700
Subject: stop/start workers "by need"
In-Reply-To: <201010291516.31851.antonvesnin@gmail.com>
References: <201010291516.31851.antonvesnin@gmail.com>
Message-ID: <490C27DA-A84B-4612-BE07-10564866DC11@jamiedubs.com>

On Oct 29, 2010, at 4:16 AM, "Anton A. Vesnin" <antonvesnin at gmail.com> wrote:

> Hello to everybody.
> I making shared hosting for rails and rack applications. For app. server i 
> userd phusion passenger, but after some issues with that, want to migrate to 
> unicorn. 
> 
> Is there any way to make unicorn stop all workers and start them on first 
> request? It'll be really great to have this opportunity. 

You'd need to keep a master running, but TTOU and TTIN signals let you arbitrarily increase and reduce the number of workers. I think disabling app preloading might shave some memory from the master as well

What issues were you having with passenger? Passenger and unicorn were created with quite different goals but I'd be curious if this works for you




> 
>> No subscription is needed to post to the mailing list, let us know that we 
> need to Cc: replies to you if you?re unsubscribed.
> Reply with Cc:, pleace. Thank you.
> 
> --
> With best regards.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying