From normalperson at yhbt.net  Thu Dec  2 20:29:01 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 3 Dec 2010 01:29:01 +0000
Subject: [ANN] unicorn 3.0.1 - one bugfix for Rainbows!
Message-ID: <20101203012901.GA32087@dcvr.yhbt.net>

...and only Rainbows!  This release fixes HTTP pipelining for
requests with bodies for users of synchronous Rainbows!
concurrency models.

Since Unicorn itself does not support keepalive nor pipelining,
Unicorn-only users need not upgrade.

* http://unicorn.bogomips.org/
* mongrel-unicorn at rubyforge.org
* git://git.bogomips.org/unicorn.git

-- 
Eric Wong

From normalperson at yhbt.net  Thu Dec  2 20:30:43 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 3 Dec 2010 01:30:43 +0000
Subject: [ANN] Rainbows! 2.0.1 - upload pipelining fixes
In-Reply-To: <20101203012901.GA32087@dcvr.yhbt.net>
References: <20101203012901.GA32087@dcvr.yhbt.net>
Message-ID: <20101203013043.GB32087@dcvr.yhbt.net>

For HTTP clients living on the edge and pipelining uploads, we
now fully support pipelined requests (as long as the application
consumes each request in its entirety).

Rainbows! is an HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

-- 
Eric Wong

From normalperson at yhbt.net  Mon Dec  6 19:20:06 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 6 Dec 2010 16:20:06 -0800
Subject: Zbatery/Rainbows keepalive problem?
In-Reply-To: <AANLkTimE+t6o1U--N3MP7btZa0xKXEMHnhTTW=8Sn_x0@mail.gmail.com>
References: <AANLkTi=oKL0Saw9sjf8Lrw-JakmOUKHnM363Qw=6JQO8@mail.gmail.com>
	<20101124221402.GA28625@dcvr.yhbt.net>
	<AANLkTi=J9cXKj+7Wi5AbL6FvwsqdJeb9UH6e-Yfs05v-@mail.gmail.com>
	<20101127085011.GA8663@dcvr.yhbt.net>
	<AANLkTimkvFSjiz60Bm90NQUa0+7-UE7-R35nVwU8LcQi@mail.gmail.com>
	<AANLkTimPgtZ47OVJtBz5JMtHs8U5wk6=ULMSou_3e18z@mail.gmail.com>
	<20101128100705.GA23779@dcvr.yhbt.net>
	<AANLkTi=bt+cMTYdc+SMi0DJ3ZBB2iV8PryaWPm5-YqC-@mail.gmail.com>
	<20101129211236.GA18934@dcvr.yhbt.net>
	<AANLkTimE+t6o1U--N3MP7btZa0xKXEMHnhTTW=8Sn_x0@mail.gmail.com>
Message-ID: <20101207002006.GA23431@dcvr.yhbt.net>

Jake Douglas <jakecdouglas at gmail.com> wrote:
> On Mon, Nov 29, 2010 at 1:12 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > Jake Douglas <jakecdouglas at gmail.com> wrote:
> >> Adding the Server header does not change anything. The only other
> >> difference I can see is that Rainbows! includes the Date header.
> >> However, removing it also does not change the behavior. I am at a
> >> complete loss now for what is causing this.
> >
> > A few more things, are these GET or POST requests (or something else)?
> > Is it done via AJAX/XmlHttpRequest?
> >
> > Can you put together a redistributable test case as a standalone app
> > that I can hammer on locally?
> >
> They are normal GET requests, not XHRs. I'll try to get you a simple
> test case sometime this week.

Hi Jake, is this still a problem?

I'm only testing with Rack::File, but can't seem to reproduce it
(using Firefox 3.6.12 (ugh)), either.

I just opened up http://rainbows.bogomips.org:8080/README.html
and started clicking some links.  I don't imagine it's too different
based on what the headers show...

-- 
Eric Wong

From ghazel at gmail.com  Sun Dec 12 19:14:58 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Sun, 12 Dec 2010 16:14:58 -0800
Subject: rails 2 and slow external services
Message-ID: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>

Hi,

Some of my page loads (currently serviced by Unicorn) spend a great
deal of time waiting for external services (OpenID, OAuth, etc over
Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
Which concurrency model in Rainbows! is best suited for this sort of
page? I'm not totally clear on which parts if any of Rails are
thread-safe.

Thanks in advance!

-Greg

From ghazel at gmail.com  Mon Dec 13 04:04:45 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Mon, 13 Dec 2010 01:04:45 -0800
Subject: /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
Message-ID: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>

Hi,

I installed the rainbows gem and symlinked rainbows-2.0.1/bin/rainbows
to /usr/local/bin/rainbows
However, running rainbows gives: -bash: /usr/local/bin/rainbows:
/this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby: bad
interpreter: No such file or directory

Any ideas what I should have done instead?

-Greg

From normalperson at yhbt.net  Mon Dec 13 05:34:20 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 13 Dec 2010 10:34:20 +0000
Subject: /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
In-Reply-To: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>
References: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>
Message-ID: <20101213103420.GA28718@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> Hi,
> 
> I installed the rainbows gem and symlinked rainbows-2.0.1/bin/rainbows
> to /usr/local/bin/rainbows
> However, running rainbows gives: -bash: /usr/local/bin/rainbows:
> /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby: bad
> interpreter: No such file or directory
> 
> Any ideas what I should have done instead?

You shouldn't have had to make a symlink, if you used gems, then
RubyGems will have put a "rainbows" executable wrapper in the same
prefix as your other gems (probably same path as the "gem" command) that
loads and wraps the $GEM_HOME/gems/rainbows-2.0.1/bin/rainbows script...

In short, it works the same way as unicorn, mongrel, thin or any
other RubyGem that distributes executable scripts.

-- 
Eric Wong

From normalperson at yhbt.net  Mon Dec 13 05:39:36 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Mon, 13 Dec 2010 02:39:36 -0800
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
Message-ID: <20101213103936.GA8440@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> Hi,
> 
> Some of my page loads (currently serviced by Unicorn) spend a great
> deal of time waiting for external services (OpenID, OAuth, etc over
> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> Which concurrency model in Rainbows! is best suited for this sort of
> page? I'm not totally clear on which parts if any of Rails are
> thread-safe.

Hi, if you want compatibility over the greatest number of existing Gems,
then ThreadSpawn and ThreadPool are the safest options.

I'm fairly certain Rails 2.3 (or even 2.2) are _advertised_ as
thread-safe but I've never tested it heavily.  Hopefully other folks can
give some feedback here.

If you're running Ruby 1.9, then RevThreadSpawn and RevThreadPool should
be good options if you do heavy keepalive, too, but they're a bit iffy
under 1.8 due to incompatibilities with libev and the MRI 1.8 threading
model.

-- 
Eric Wong

From russm-rubyforge at slofith.org  Mon Dec 13 05:52:13 2010
From: russm-rubyforge at slofith.org (russell muetzelfeldt)
Date: Mon, 13 Dec 2010 21:52:13 +1100
Subject: /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
In-Reply-To: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>
References: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>
Message-ID: <81A07824-4000-4E2B-A353-19C6986F526A@slofith.org>

On 13/12/2010, at 8:04 PM, ghazel at gmail.com wrote:

> I installed the rainbows gem and symlinked rainbows-2.0.1/bin/rainbows
> to /usr/local/bin/rainbows
> However, running rainbows gives: -bash: /usr/local/bin/rainbows:
> /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby: bad
> interpreter: No such file or directory
> 
> Any ideas what I should have done instead?

what platform are you on? you shouldn't have to link gem executable scripts anywhere for them to work.

when I install gems, the system creates a wrapper executable in an existing bin/ directory (/usr/bin/ on OSX, /var/lib/gems/1.9.1/bin/ on Debian Squeeze) that lets you force a version for the loaded gem and then loads the original executable script without launching a new interpreter - this ignores the supposed #! line in the gem's internal bin/rainbows, which is why that file says its #! will be overwritten or wrapped.

If installing rainbows didn't leave an executable script somewhere in your $PATH you might need to find where your local gem package is putting the wrappers and add that to your search path.

cheers

Russell

-----
Russell Muetzelfeldt <russm at slofith.org>
Mundus vult decipi, ergo decipiatur.


From ghazel at gmail.com  Mon Dec 13 13:28:09 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Mon, 13 Dec 2010 10:28:09 -0800
Subject: /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby
In-Reply-To: <81A07824-4000-4E2B-A353-19C6986F526A@slofith.org>
References: <AANLkTikfzyTV4Ymbk2vPHfq2VwYpM9f3mZuL0ep_eZ-O@mail.gmail.com>
	<81A07824-4000-4E2B-A353-19C6986F526A@slofith.org>
Message-ID: <AANLkTin7ABGGRuRp9mtW8f3VCY=77zpqpBmvJJ78MeKY@mail.gmail.com>

On Mon, Dec 13, 2010 at 2:52 AM, russell muetzelfeldt
<russm-rubyforge at slofith.org> wrote:
> On 13/12/2010, at 8:04 PM, ghazel at gmail.com wrote:
>
>> I installed the rainbows gem and symlinked rainbows-2.0.1/bin/rainbows
>> to /usr/local/bin/rainbows
>> However, running rainbows gives: -bash: /usr/local/bin/rainbows:
>> /this/will/be/overwritten/or/wrapped/anyways/do/not/worry/ruby: bad
>> interpreter: No such file or directory
>>
>> Any ideas what I should have done instead?
>
> what platform are you on? you shouldn't have to link gem executable scripts anywhere for them to work.
>
> when I install gems, the system creates a wrapper executable in an existing bin/ directory (/usr/bin/ on OSX, /var/lib/gems/1.9.1/bin/ on Debian Squeeze) that lets you force a version for the loaded gem and then loads the original executable script without launching a new interpreter - this ignores the supposed #! line in the gem's internal bin/rainbows, which is why that file says its #! will be overwritten or wrapped.
>
> If installing rainbows didn't leave an executable script somewhere in your $PATH you might need to find where your local gem package is putting the wrappers and add that to your search path.

Ah! For me it's /usr/local/ruby-enterprise-1.8.7-2010.01/bin

That makes much more sense. Thanks!

-Greg

From ghazel at gmail.com  Mon Dec 13 15:40:48 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Mon, 13 Dec 2010 12:40:48 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101213103936.GA8440@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
Message-ID: <AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>

On Mon, Dec 13, 2010 at 2:39 AM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> Hi,
>>
>> Some of my page loads (currently serviced by Unicorn) spend a great
>> deal of time waiting for external services (OpenID, OAuth, etc over
>> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
>> Which concurrency model in Rainbows! is best suited for this sort of
>> page? I'm not totally clear on which parts if any of Rails are
>> thread-safe.
>
> Hi, if you want compatibility over the greatest number of existing Gems,
> then ThreadSpawn and ThreadPool are the safest options.
>
> I'm fairly certain Rails 2.3 (or even 2.2) are _advertised_ as
> thread-safe but I've never tested it heavily. ?Hopefully other folks can
> give some feedback here.
>
> If you're running Ruby 1.9, then RevThreadSpawn and RevThreadPool should
> be good options if you do heavy keepalive, too, but they're a bit iffy
> under 1.8 due to incompatibilities with libev and the MRI 1.8 threading
> model.

This is Ruby 1.8.7 (REE). Is there any interesting difference between
ThreadPool and ThreadSpawn in this environment?

I also make use of a (heavily modified, which is another topic) OobGC.
Does anyone know if garbage collection in ruby 1.8.7 is reasonably
threadable? I expect not, but one can hope.

-Greg

From normalperson at yhbt.net  Mon Dec 13 23:57:20 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 14 Dec 2010 04:57:20 +0000
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
Message-ID: <20101214045720.GC5051@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> > ghazel at gmail.com wrote:
> >> Some of my page loads (currently serviced by Unicorn) spend a great
> >> deal of time waiting for external services (OpenID, OAuth, etc over
> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> 
> This is Ruby 1.8.7 (REE). Is there any interesting difference between
> ThreadPool and ThreadSpawn in this environment?

ThreadPool is generally more predictable, but ThreadSpawn has lower
memory usage if your traffic spikes are intermittent or low.

ThreadSpawn is much like the original Mongrel and ThreadPool was an
experiment with Ruby 1.9 in mind.  1.9 has more expensive (but slightly
more concurrent) threading.  If your bottlenecks are external HTTP
requests on 1.8, but first instinct would be to use ThreadSpawn.

Ruby 1.9 + ThreadPool would probably be well-suited for large file
serving to LAN clients with many slowish disks as it can use sendfile
via IO.copy_stream), but if you can afford the constant memory overhead,
it could be good in 1.8, too.

> I also make use of a (heavily modified, which is another topic) OobGC.
> Does anyone know if garbage collection in ruby 1.8.7 is reasonably
> threadable? I expect not, but one can hope.

It is not, the entire interpreter stops running every single thread for
GC.  I don't think using OobGC with any of the Rainbows! concurrency
models will work, only :Base and Unicorn.

-- 
Eric Wong

From ghazel at gmail.com  Tue Dec 14 01:14:53 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Mon, 13 Dec 2010 22:14:53 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101214045720.GC5051@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
Message-ID: <AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>

On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> > ghazel at gmail.com wrote:
>> >> Some of my page loads (currently serviced by Unicorn) spend a great
>> >> deal of time waiting for external services (OpenID, OAuth, etc over
>> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
>>
>> This is Ruby 1.8.7 (REE). Is there any interesting difference between
>> ThreadPool and ThreadSpawn in this environment?
>
> ThreadPool is generally more predictable, but ThreadSpawn has lower
> memory usage if your traffic spikes are intermittent or low.
>
> ThreadSpawn is much like the original Mongrel and ThreadPool was an
> experiment with Ruby 1.9 in mind. ?1.9 has more expensive (but slightly
> more concurrent) threading. ?If your bottlenecks are external HTTP
> requests on 1.8, but first instinct would be to use ThreadSpawn.
>
> Ruby 1.9 + ThreadPool would probably be well-suited for large file
> serving to LAN clients with many slowish disks as it can use sendfile
> via IO.copy_stream), but if you can afford the constant memory overhead,
> it could be good in 1.8, too.

I don't mind constant memory overhead -  actually I prefer it to
spikey memory usage with an unknown peak. Otherwise they should be the
same?

>> I also make use of a (heavily modified, which is another topic) OobGC.
>> Does anyone know if garbage collection in ruby 1.8.7 is reasonably
>> threadable? I expect not, but one can hope.
>
> It is not, the entire interpreter stops running every single thread for
> GC. ?I don't think using OobGC with any of the Rainbows! concurrency
> models will work, only :Base and Unicorn.

Well, not too surprising, but thanks for specifying.

I'm running a bit of my traffic through some Rainbows! right now, but I got:

2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
referrer: "https://foofoo.com"
2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"

>From nginx in the error log. My proxy_read_timeout is 300, and my
listen backlog is 2048 (for now...). Basically my Rainbows! config is
identical to my Unicorn config, where I have not seen that happen,
except I added "Rainbows! { use :ThreadPool; worker_connections 100
}".

Any ideas?

-Greg

From normalperson at yhbt.net  Tue Dec 14 01:35:52 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 14 Dec 2010 06:35:52 +0000
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
Message-ID: <20101214063552.GA12020@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > ghazel at gmail.com wrote:
> >> > ghazel at gmail.com wrote:
> >> >> Some of my page loads (currently serviced by Unicorn) spend a great
> >> >> deal of time waiting for external services (OpenID, OAuth, etc over
> >> >> Net::HTTP and curb), so I'm looking at Rainbows!. I use Rails 2.3.10.
> >>
> >> This is Ruby 1.8.7 (REE). Is there any interesting difference between
> >> ThreadPool and ThreadSpawn in this environment?
> >
> > ThreadPool is generally more predictable, but ThreadSpawn has lower
> > memory usage if your traffic spikes are intermittent or low.
> >
> > ThreadSpawn is much like the original Mongrel and ThreadPool was an
> > experiment with Ruby 1.9 in mind. ?1.9 has more expensive (but slightly
> > more concurrent) threading. ?If your bottlenecks are external HTTP
> > requests on 1.8, but first instinct would be to use ThreadSpawn.
> >
> > Ruby 1.9 + ThreadPool would probably be well-suited for large file
> > serving to LAN clients with many slowish disks as it can use sendfile
> > via IO.copy_stream), but if you can afford the constant memory overhead,
> > it could be good in 1.8, too.
> 
> I don't mind constant memory overhead -  actually I prefer it to
> spikey memory usage with an unknown peak. Otherwise they should be the
> same?

Having idle threads with ThreadPool would affect GC performance,
actually.

> I'm running a bit of my traffic through some Rainbows! right now, but I got:
> 
> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
> Connection timed out) while reading response header from upstream,
> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
> referrer: "https://foofoo.com"
> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
> Connection timed out) while reading response header from upstream,
> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
> 
> From nginx in the error log. My proxy_read_timeout is 300, and my
> listen backlog is 2048 (for now...). Basically my Rainbows! config is
> identical to my Unicorn config, where I have not seen that happen,
> except I added "Rainbows! { use :ThreadPool; worker_connections 100
> }".

Was your app hitting the Unicorn kill -9 timeout frequently before?  In
Rainbows!, the kill -9 timeout only kicks in when the entire
interpreter/VM is stuck due to a broken C extension or bug in Ruby.

If it's some component of your app taking a long time (and relying on
the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

-- 
Eric Wong

From ghazel at gmail.com  Tue Dec 14 02:13:33 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Mon, 13 Dec 2010 23:13:33 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101214063552.GA12020@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
Message-ID: <AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>

On Mon, Dec 13, 2010 at 10:35 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> On Mon, Dec 13, 2010 at 8:57 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> > ghazel at gmail.com wrote:
>> I'm running a bit of my traffic through some Rainbows! right now, but I got:
>>
>> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
>> referrer: "https://foofoo.com"
>> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
>>
>> From nginx in the error log. My proxy_read_timeout is 300, and my
>> listen backlog is 2048 (for now...). Basically my Rainbows! config is
>> identical to my Unicorn config, where I have not seen that happen,
>> except I added "Rainbows! { use :ThreadPool; worker_connections 100
>> }".
>
> Was your app hitting the Unicorn kill -9 timeout frequently before? ?In
> Rainbows!, the kill -9 timeout only kicks in when the entire
> interpreter/VM is stuck due to a broken C extension or bug in Ruby.
>
> If it's some component of your app taking a long time (and relying on
> the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
> middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

I'm not sure how I would know that - I'm not actually sure which
timeout you mean. "timeout" in the config/unicorn.rb and
config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
you saying when that is hit that Rainbows! and Unicorn act
differently?

-Greg

From normalperson at yhbt.net  Tue Dec 14 02:49:44 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 14 Dec 2010 07:49:44 +0000
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
Message-ID: <20101214074944.GA13496@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> I'm not sure how I would know that - I'm not actually sure which
> timeout you mean. "timeout" in the config/unicorn.rb and
> config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
> you saying when that is hit that Rainbows! and Unicorn act
> differently?

Yes, the purpose of the "timeout" in the Unicorn config has always been
to kill unrecoverably stuck processes, Rainbows! just enforces that
more strictly.

The simple nature of Unicorn allows lazy people can use it to avoid
adding timeouts to their code without causing harm to other clients.

Since Rainbows! is designed to serve multiple clients in the same
process, killing a process would break all the clients on the process,
not just the one client that triggered the timeout.  So using Rainbows!
requires more discipline from the application authors than Unicorn.

-- 
Eric Wong

From ghazel at gmail.com  Tue Dec 14 03:03:12 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Tue, 14 Dec 2010 00:03:12 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101214074944.GA13496@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
Message-ID: <AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>

On Mon, Dec 13, 2010 at 11:49 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> I'm not sure how I would know that - I'm not actually sure which
>> timeout you mean. "timeout" in the config/unicorn.rb and
>> config/rainbows.rb is 300, same as proxy_read_timeout. Is that it? Are
>> you saying when that is hit that Rainbows! and Unicorn act
>> differently?
>
> Yes, the purpose of the "timeout" in the Unicorn config has always been
> to kill unrecoverably stuck processes, Rainbows! just enforces that
> more strictly.
>
> The simple nature of Unicorn allows lazy people can use it to avoid
> adding timeouts to their code without causing harm to other clients.
>
> Since Rainbows! is designed to serve multiple clients in the same
> process, killing a process would break all the clients on the process,
> not just the one client that triggered the timeout. ?So using Rainbows!
> requires more discipline from the application authors than Unicorn.

Hm. Well I was unaware that there was any timeout issue with my
application. When a Unicorn process timed out and died, how did the
request not timeout with nginx? Was it re-submitted to another worker?

-Greg

From normalperson at yhbt.net  Tue Dec 14 12:27:48 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 14 Dec 2010 17:27:48 +0000
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
Message-ID: <20101214172748.GA18131@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> Hm. Well I was unaware that there was any timeout issue with my
> application. When a Unicorn process timed out and died, how did the
> request not timeout with nginx? Was it re-submitted to another worker?

A timeout issue is one *possible* cause of the errors you were seeing
from nginx.  Of course you know the application better than I do, so,
I'm not certain it was a timeout issue, just a likely cause of the
errors.  Did your Unicorn error logs tell you if there were any
timeouts?

Anyways I'd always put timeouts around any code that accesses remote
services since the Internet is unreliable.  It's also pretty easy to
setup an evil OpenID provider server that can DoS web apps that don't
timeout themselves.

-- 
Eric Wong

From ghazel at gmail.com  Tue Dec 14 21:01:06 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Tue, 14 Dec 2010 18:01:06 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101214172748.GA18131@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
Message-ID: <AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>

On Tue, Dec 14, 2010 at 9:27 AM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> Hm. Well I was unaware that there was any timeout issue with my
>> application. When a Unicorn process timed out and died, how did the
>> request not timeout with nginx? Was it re-submitted to another worker?
>
> A timeout issue is one *possible* cause of the errors you were seeing
> from nginx. ?Of course you know the application better than I do, so,
> I'm not certain it was a timeout issue, just a likely cause of the
> errors. ?Did your Unicorn error logs tell you if there were any
> timeouts?

Nothing in the Unicorn logs ever indicated an error like that. I
believe I found the cause though, and this did not seem to affect
Unicorn:

 - I was not using config.threadsafe! (should probably mention that
with the Rainbows! + Rails docs somewhere)
 - I was attempting to JSON serialize request.env. It seems there are
real objects in env, which when you try to serialize them cause your
application to hang forever. Bleh.

-Greg

From normalperson at yhbt.net  Tue Dec 14 21:40:43 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 14 Dec 2010 18:40:43 -0800
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
References: <20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
	<AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
Message-ID: <20101215024043.GA10349@dcvr.yhbt.net>

ghazel at gmail.com wrote:
>  - I was not using config.threadsafe! (should probably mention that
> with the Rainbows! + Rails docs somewhere)

Done and pushed out.  Thanks for following up with your resolution
so others can benefit from your discoveries.

>From 7e2bb251228a30c0d4e66029b20bbbf85bc99a09 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Tue, 14 Dec 2010 18:36:34 -0800
Subject: [PATCH] FAQ: add a note about config.threadsafe!

At least one user ran into it:
http://mid.gmane.org/AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE at mail.gmail.com
---
 FAQ |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/FAQ b/FAQ
index 7609d55..f45568b 100644
--- a/FAQ
+++ b/FAQ
@@ -98,3 +98,9 @@ For older versions of Rails, the following config.ru will work:
 
 One thing to watch out for is that RAILS_ENV will not be set in the
 environment for you, thus we set it to match RACK_ENV.
+
+=== I'm using threads and Rails is misbehaving!
+
+If you use any of the threaded concurrency models, you will need to use
+{config.threadsafe!}[http://m.onkey.org/thread-safety-for-your-rails]
+in your config/environments/$RAILS_ENV.rb
-- 
Eric Wong

From ghazel at gmail.com  Tue Dec 14 23:30:29 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Tue, 14 Dec 2010 20:30:29 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101215024043.GA10349@dcvr.yhbt.net>
References: <20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
	<AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
	<20101215024043.GA10349@dcvr.yhbt.net>
Message-ID: <AANLkTi=mjEc2z96LY=MmFSFjp2NJ_66Lb=KVhLLpZEWy@mail.gmail.com>

On Tue, Dec 14, 2010 at 6:40 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> ?- I was not using config.threadsafe! (should probably mention that
>> with the Rainbows! + Rails docs somewhere)
>
> Done and pushed out. ?Thanks for following up with your resolution
> so others can benefit from your discoveries.
>
> >From 7e2bb251228a30c0d4e66029b20bbbf85bc99a09 Mon Sep 17 00:00:00 2001
> From: Eric Wong <normalperson at yhbt.net>
> Date: Tue, 14 Dec 2010 18:36:34 -0800
> Subject: [PATCH] FAQ: add a note about config.threadsafe!
>
> At least one user ran into it:
> http://mid.gmane.org/AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE at mail.gmail.com
> ---
> ?FAQ | ? ?6 ++++++
> ?1 files changed, 6 insertions(+), 0 deletions(-)
>
> diff --git a/FAQ b/FAQ
> index 7609d55..f45568b 100644
> --- a/FAQ
> +++ b/FAQ
> @@ -98,3 +98,9 @@ For older versions of Rails, the following config.ru will work:
>
> ?One thing to watch out for is that RAILS_ENV will not be set in the
> ?environment for you, thus we set it to match RACK_ENV.
> +
> +=== I'm using threads and Rails is misbehaving!
> +
> +If you use any of the threaded concurrency models, you will need to use
> +{config.threadsafe!}[http://m.onkey.org/thread-safety-for-your-rails]
> +in your config/environments/$RAILS_ENV.rb

Sadly, also:
https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/3457-actioncontrollercachingsweeper-controller-instance-is-not-thread-safe

-Greg

From ghazel at gmail.com  Wed Dec 15 17:28:38 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Wed, 15 Dec 2010 14:28:38 -0800
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
	<AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
Message-ID: <AANLkTimyy5XyVFF4+MWWgxfb63fCoAmWRYM=MEjVWn96@mail.gmail.com>

On Tue, Dec 14, 2010 at 6:01 PM,  <ghazel at gmail.com> wrote:
> On Tue, Dec 14, 2010 at 9:27 AM, Eric Wong <normalperson at yhbt.net> wrote:
>> ghazel at gmail.com wrote:
>>> Hm. Well I was unaware that there was any timeout issue with my
>>> application. When a Unicorn process timed out and died, how did the
>>> request not timeout with nginx? Was it re-submitted to another worker?
>>
>> A timeout issue is one *possible* cause of the errors you were seeing
>> from nginx. ?Of course you know the application better than I do, so,
>> I'm not certain it was a timeout issue, just a likely cause of the
>> errors. ?Did your Unicorn error logs tell you if there were any
>> timeouts?
>
> Nothing in the Unicorn logs ever indicated an error like that.

Also with Rainbows! I see a good number of:
2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
Broken pipe) while sending request to upstream,
2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
Connection reset by peer) while reading upstream,

on our photo upload page. These errors did not occur with Unicorn, and
the Unicorn logs said nothing about a failure either.

-Greg

From ghazel at gmail.com  Thu Dec 16 09:39:52 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Thu, 16 Dec 2010 06:39:52 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101214063552.GA12020@dcvr.yhbt.net>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
Message-ID: <AANLkTi=7OhyTwkHsp_rXU7Gp1PokihiQ9bJigpO-BfN6@mail.gmail.com>

On Mon, Dec 13, 2010 at 10:35 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> I'm running a bit of my traffic through some Rainbows! right now, but I got:
>>
>> 2010/12/14 02:30:24 [error] 3183#0: *9229917 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 1.2.3.4, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com",
>> referrer: "https://foofoo.com"
>> 2010/12/14 04:28:25 [error] 3182#0: *9440717 upstream timed out (110:
>> Connection timed out) while reading response header from upstream,
>> client: 5.6.7.8, server: mysite.com, request: "GET /blah HTTP/1.1",
>> upstream: "http://unix:/tmp/rainbows.sock:/blah", host: "mysite.com"
>>
>> From nginx in the error log. My proxy_read_timeout is 300, and my
>> listen backlog is 2048 (for now...). Basically my Rainbows! config is
>> identical to my Unicorn config, where I have not seen that happen,
>> except I added "Rainbows! { use :ThreadPool; worker_connections 100
>> }".
>
> Was your app hitting the Unicorn kill -9 timeout frequently before? ?In
> Rainbows!, the kill -9 timeout only kicks in when the entire
> interpreter/VM is stuck due to a broken C extension or bug in Ruby.
>
> If it's some component of your app taking a long time (and relying on
> the Unicorn kill -9 timeout), you can try the Rainbows::ThreadTimeout
> middleware: http://rainbows.rubyforge.org/Rainbows/ThreadTimeout.html

I believe I found a bug in ThreadTimeout. It seems to be terminating a
request immediately every @timeout seconds. I'm using :ThreadSpawn and
my initializer is:
use Rainbows::ThreadTimeout, :timeout => 299

The observed behavior is that the request dies < 1 second later with
Rainbows::ThreadTimeout::ExecutionExpired. If I look 299 seconds back
in the log, I see another request killed prematurely. If I look 299
further back from that, I see the process started at this time.
Glancing at the code:

        now = Time.now
        @lock.synchronize do
          @active.delete_if do |thread, time|
            time >= now and thread.raise(ExecutionExpired).nil?
          end
        end

The "time >= now" seems incorrect to me. Since "time" is set to
"Time.now + @timeout" it will be greater than Time.now while the
request still has time left. I believe that should read "now >= time".

-Greg

From normalperson at yhbt.net  Sun Dec 19 20:00:15 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 19 Dec 2010 17:00:15 -0800
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTi=7OhyTwkHsp_rXU7Gp1PokihiQ9bJigpO-BfN6@mail.gmail.com>
References: <AANLkTikYESRSJbG3TViBPcsj7PDwcQ4O__xdtidvn_z1@mail.gmail.com>
	<20101213103936.GA8440@dcvr.yhbt.net>
	<AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTi=7OhyTwkHsp_rXU7Gp1PokihiQ9bJigpO-BfN6@mail.gmail.com>
Message-ID: <20101220010015.GA20300@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> I believe I found a bug in ThreadTimeout. It seems to be terminating a
> request immediately every @timeout seconds. I'm using :ThreadSpawn and
> my initializer is:
> use Rainbows::ThreadTimeout, :timeout => 299
> 
> The observed behavior is that the request dies < 1 second later with
> Rainbows::ThreadTimeout::ExecutionExpired. If I look 299 seconds back
> in the log, I see another request killed prematurely. If I look 299
> further back from that, I see the process started at this time.
> Glancing at the code:
> 
>         now = Time.now
>         @lock.synchronize do
>           @active.delete_if do |thread, time|
>             time >= now and thread.raise(ExecutionExpired).nil?
>           end
>         end
> 
> The "time >= now" seems incorrect to me. Since "time" is set to
> "Time.now + @timeout" it will be greater than Time.now while the
> request still has time left. I believe that should read "now >= time".

Yup, pushing out a trivial fix in a bit along with a test case.
Thanks!

(sorry for the delayed response, been under the weather lately)
-- 
Eric Wong

From normalperson at yhbt.net  Sun Dec 19 21:52:56 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Sun, 19 Dec 2010 18:52:56 -0800
Subject: rails 2 and slow external services
In-Reply-To: <AANLkTimyy5XyVFF4+MWWgxfb63fCoAmWRYM=MEjVWn96@mail.gmail.com>
References: <AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
	<AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
	<AANLkTimyy5XyVFF4+MWWgxfb63fCoAmWRYM=MEjVWn96@mail.gmail.com>
Message-ID: <20101220025256.GB20300@dcvr.yhbt.net>

ghazel at gmail.com wrote:
> Also with Rainbows! I see a good number of:
> 2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
> Broken pipe) while sending request to upstream,
> 2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
> Connection reset by peer) while reading upstream,
> 
> on our photo upload page. These errors did not occur with Unicorn, and
> the Unicorn logs said nothing about a failure either.

Could this be related to the ThreadTimeout middleware bug?

Still working on making the tests run properly for that, but I seem to
be hitting another bug somewhere...

-- 
Eric Wong

From ghazel at gmail.com  Sun Dec 19 22:04:55 2010
From: ghazel at gmail.com (ghazel at gmail.com)
Date: Sun, 19 Dec 2010 19:04:55 -0800
Subject: rails 2 and slow external services
In-Reply-To: <20101220025256.GB20300@dcvr.yhbt.net>
References: <AANLkTikD5qSSxb_qtdMXF4pznu3EPxG7+6nVJLRkjttH@mail.gmail.com>
	<20101214045720.GC5051@dcvr.yhbt.net>
	<AANLkTi=FN02-OJc3Utzh74zMA+MX4KqHc6hUHjR5a=a_@mail.gmail.com>
	<20101214063552.GA12020@dcvr.yhbt.net>
	<AANLkTik+HpLvQTrNtLFRZ4xYtN4i_F8JVsBTT8odi4tm@mail.gmail.com>
	<20101214074944.GA13496@dcvr.yhbt.net>
	<AANLkTiks_hBMMTHMEDmaxnya=pogx67AzK4Dzz+Kfbmz@mail.gmail.com>
	<20101214172748.GA18131@dcvr.yhbt.net>
	<AANLkTikegPX2-6Q93++bz_aLt+9nLPJXjg+NkL8tDjeE@mail.gmail.com>
	<AANLkTimyy5XyVFF4+MWWgxfb63fCoAmWRYM=MEjVWn96@mail.gmail.com>
	<20101220025256.GB20300@dcvr.yhbt.net>
Message-ID: <AANLkTikcS9OoCbCJO7wjyGcBNse4JGXokxKSwMV9KVEU@mail.gmail.com>

On Sun, Dec 19, 2010 at 6:52 PM, Eric Wong <normalperson at yhbt.net> wrote:
> ghazel at gmail.com wrote:
>> Also with Rainbows! I see a good number of:
>> 2010/12/15 21:37:42 [error] 2104#0: *174851 sendfile() failed (32:
>> Broken pipe) while sending request to upstream,
>> 2010/12/15 21:37:55 [error] 2090#0: *175527 readv() failed (104:
>> Connection reset by peer) while reading upstream,
>>
>> on our photo upload page. These errors did not occur with Unicorn, and
>> the Unicorn logs said nothing about a failure either.
>
> Could this be related to the ThreadTimeout middleware bug?

I don't think so. I added the ThreadTimeout middleware after many of
these had occurred.

-Greg

From nishibushiti92meomy at msn.com  Tue Dec 21 18:47:36 2010
From: nishibushiti92meomy at msn.com (Louis)
Date: Wed, 22 Dec 2010 07:47:36 +0800
Subject: =?GB2312?B?M0QvMkQgQW5pbWF0aW9uIHNlcnZpY2VzIA==?=
	=?GB2312?B?LSBDYXJ0b29uIERlc2lnbiAtIDNEIG1vZA==?=
	=?GB2312?B?ZWxpbmc=?=
Message-ID: <NS2xfiIBFdtaFcFzr8c00004e4b@inpix.com>

You are receiving this email because we wish you to use our 3D/2D animation services.

We are a China based animation studio. We are specialized in providing 3D designing/modelling/animation services across the globe. We utilize the finest equipment available in the industry, offer efficient data delivery and unrivaled quality and work until the client is fully satisfied with the end product. We pushed past the limitations of technology and hardware to bring 3D walk-through animations and animated effects to the Web, VHS, DVD and now High-Definition. 

3d Animation services  
3D Modeling  
Character.Set.Props. Modeling , and Environment Creation 
Painting Textures 
layout Uv,painting the Colour.Bump. Normal Map .etc  
3D Animation 
Rigging.Skinning.Aanimation for game and film  

Custom 3D Modeling  
Whether you're looking for an inexpensive alternative to 3D model conversion software or just need your latest creative masterpiece to import correctly into your new favorite software, we can help!

Architectures Services  
Interior/exterior architecture rendering 
Architecture animation 

Core Offerings
3D Animation
4D Animation
3D Modelling
3D Rendering
2D Animation
2D/3D Game Animation
2D/3D Game Development
Cartoon 
Architecture Visualization 
Character Animation 
Animated Commercial 
Product Modelling 
Animated Presentation 
Walkthrough Animation 
Flash Presentation 
Logo Animation 
Flash Animation 
Flash Games
Product Demos 
Medical Animation 
Animated Movies 
Flash Gateways 
Animated Wallpapers 
Animated Graphics 

Best regards,
Louis
Tanfeffson Animation Services
Contact: ibanicontact at yeah.net























Pls send address to aniremove at yeah.net for remove   

                                                                                                                                                                 

From ibc at aliax.net  Wed Dec 22 06:24:26 2010
From: ibc at aliax.net (=?UTF-8?Q?I=C3=B1aki_Baz_Castillo?=)
Date: Wed, 22 Dec 2010 12:24:26 +0100
Subject: [ANN] kgio library / RubyGem
In-Reply-To: <20100928030302.GA3452@dcvr.yhbt.net>
References: <20100928030302.GA3452@dcvr.yhbt.net>
Message-ID: <AANLkTiksx3defTUbaC_30do=7b925i-f-wwdWoeraVFv@mail.gmail.com>

2010/9/28 Eric Wong <normalperson at yhbt.net>:
> I've released kgio, a kinder, gentler I/O library for Ruby. ?Some of its
> features are useful for Unicorn, and all of it is useful for Rainbows!
>
> I intend to make kgio a requirement for both Unicorn and
> Rainbows!/Zbatery. ?I'm comfortable with the code, but extra testers and
> extra eyes to review it would be helpful (it's nearly all C).

Very interesting project :)

A typo in the doc:

---------------------
kgio_wait_readable()
Blocks the running Thread indefinitely until self IO object is writable
---------------------

Should be "readable", right? :)


-- 
I?aki Baz Castillo
<ibc at aliax.net>

From normalperson at yhbt.net  Wed Dec 22 15:27:31 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 22 Dec 2010 12:27:31 -0800
Subject: [ANN] kgio library / RubyGem
In-Reply-To: <AANLkTiksx3defTUbaC_30do=7b925i-f-wwdWoeraVFv@mail.gmail.com>
References: <20100928030302.GA3452@dcvr.yhbt.net>
	<AANLkTiksx3defTUbaC_30do=7b925i-f-wwdWoeraVFv@mail.gmail.com>
Message-ID: <20101222202731.GA16060@dcvr.yhbt.net>

I?aki Baz Castillo <ibc at aliax.net> wrote:
> A typo in the doc:
> 
> ---------------------
> kgio_wait_readable()
> Blocks the running Thread indefinitely until self IO object is writable
> ---------------------
> 
> Should be "readable", right? :)

Fixed, thanks!

-- 
Eric Wong

From normalperson at yhbt.net  Tue Dec 28 21:28:32 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 29 Dec 2010 02:28:32 +0000
Subject: [ANN] Rainbows! 2.1.0 - Cool.io, bugfixes and more!
Message-ID: <20101229022832.GA5862@dcvr.yhbt.net>

Rainbows! is an HTTP server for sleepy Rack applications.  It is based on
Unicorn, but designed to handle applications that expect long
request/response times and/or slow clients.

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/rainbows.git

Changes:

Cool.io (new version of Rev) support is explicitly added
(it always worked before).  ":Coolio" may be used in place
of ":Rev" anywhere in your Rainbows! config file.

There is a new "keepalive_requests" config directive to limit
the number of requests a single connection may make (default:
100, same as nginx).  This may be useful for better
load-balancing characteristics.

The old "Rev" prefixes remain supported as long as Cool.io
remains compatible with Rev (likely forever).

Bug fixes:

* Rainbows::ThreadTimeout middleware with multiple clients
* large, pipelined upload errors with Revactor+Coolio(Rev)
* high CPU usage for maintaining idle keepalive on *Fiber*
* needless ThreadPool wakeups
* request env prematurely cleared keepalive requests,
  breaking some middlewares such as Clogger.
* "close" not called on body if wrapper and sendfile used together

Various code cleanups, and our RDoc website is JavaScript-free.
See the ChangeLog[1] or git for all changes.

[1] http://rainbows.rubyforge.org/ChangeLog.html

-- 
Eric Wong

From normalperson at yhbt.net  Wed Dec 29 04:35:09 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 29 Dec 2010 09:35:09 +0000
Subject: [ANN] Zbatery 0.6.0 - Rainbows! - Rainbows! 2.1.x resync
In-Reply-To: <20101229022832.GA5862@dcvr.yhbt.net>
References: <20101229022832.GA5862@dcvr.yhbt.net>
Message-ID: <20101229093509.GA27384@dcvr.yhbt.net>

Zbatery is an HTTP server for Rack applications on systems that either
do not support fork(), or have no memory (nor need) to run the
master/worker model.  It is based on Rainbows! (which is based on
Unicorn (which is based on Mongrel)) and inherits parts of each.
Zbatery supports your choice of all the thread/fiber/event/actor-based
concurrency models and Rack middleware that Rainbows! supports (or will
ever support) in a single process.

* http://zbatery.bogomip.org/
* rainbows-talk at rubyforge.org
* git://git.bogomips.org/zbatery.git

Changes:

All the latest and greatest changes from Rainbows! 2.1.0:
  http://git.bogomips.org/cgit/rainbows.git/tag/?id=v2.1.0
-- 
Eric Wong

From rjolivier at mac.com  Thu Dec 30 15:28:34 2010
From: rjolivier at mac.com (Robert Olivier)
Date: Thu, 30 Dec 2010 12:28:34 -0800
Subject: Sunshowers status
Message-ID: <ifiq1i$nmb$1@dough.gmane.org>

I was working with sunshowers today and found that it hasn't been 
updated to the latest protocol spec and isnt' working with safari 5 / 
chrome due to a malformed server handshake.

Is anyone working on this?  I'm going to try and patch it myself and 
would like to submit my patch if I get it working.

rjo



From normalperson at yhbt.net  Thu Dec 30 23:15:16 2010
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 30 Dec 2010 20:15:16 -0800
Subject: Sunshowers status
In-Reply-To: <ifiq1i$nmb$1@dough.gmane.org>
References: <ifiq1i$nmb$1@dough.gmane.org>
Message-ID: <20101231041516.GA7966@dcvr.yhbt.net>

Robert Olivier <rjolivier at mac.com> wrote:
> I was working with sunshowers today and found that it hasn't been  
> updated to the latest protocol spec and isnt' working with safari 5 /  
> chrome due to a malformed server handshake.
>
> Is anyone working on this?  I'm going to try and patch it myself and  
> would like to submit my patch if I get it working.

This project has been stagnant for a while, mainly because I haven't
found much use for it (I barely use a web browser myself).

I'll gladly accept any patches/pull requests you send to update this for
the spec, though I seem to recall some required invasive modifications
to Rainbows!  Let me know what you find, thanks!


-- 
Eric Wong