From andrew at andrewloe.com  Wed Dec  5 01:19:28 2012
From: andrew at andrewloe.com (W. Andrew Loe III)
Date: Tue, 4 Dec 2012 17:19:28 -0800
Subject: Sinatra stream(:keep_open) not holding connections open
Message-ID: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>

I'm running Rainbows with EventMachine and struggling to get it to
keep the connections open.

First, here is my code: https://gist.github.com/c33598db8a113b574165

This application works on Thin.

The main loop handles the HTTP request and creates a Celluloid Actor
that hangs out waiting on messages from Redis, when it receives a
message it dispatches it to the stream which EM then pushes out to the
client. Eventually the client will disconnect (this is Server-Sent
Events) and the Actor is torn down.

I want to run under Rainbows! for the hot upgrades and to re-use all
of the wonderful scripts I have for managing my Rails/Unicorn stack.
What I see is the GET handler executed, the Actor spawned, then the
connection is closed and the Actor is terminated. The browser then
reconnects. It doesn't ever have the opportunity to dispatch a message
the Actor doesn't even connect to Redis before it is terminated.
Response times are something like 0.0011 seconds, nothing is being
done.

I'd prefer not to pull in Sinatra::Async, I'd like to know why it
works with Thin out-of-the-box, but not Rainbows!

From normalperson at yhbt.net  Wed Dec  5 01:44:54 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 5 Dec 2012 01:44:54 +0000
Subject: Sinatra stream(:keep_open) not holding connections open
In-Reply-To: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>
References: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>
Message-ID: <20121205014454.GA24401@dcvr.yhbt.net>

"W. Andrew Loe III" <andrew at andrewloe.com> wrote:
> I'm running Rainbows with EventMachine and struggling to get it to
> keep the connections open.
> 
> First, here is my code: https://gist.github.com/c33598db8a113b574165

I'll take a closer look later today.

Can you use a public gist (or any other public git repo) instead of a
private one?  Private gists make no sense for public bug reports.

> This application works on Thin.
> 
> The main loop handles the HTTP request and creates a Celluloid Actor
> that hangs out waiting on messages from Redis, when it receives a
> message it dispatches it to the stream which EM then pushes out to the
> client. Eventually the client will disconnect (this is Server-Sent
> Events) and the Actor is torn down.

Can you reproduce the issue with a stripped down application that
does not depend on Redis?  How about one that does not depend on
Celluloid?

> I want to run under Rainbows! for the hot upgrades and to re-use all
> of the wonderful scripts I have for managing my Rails/Unicorn stack.
> What I see is the GET handler executed, the Actor spawned, then the
> connection is closed and the Actor is terminated. The browser then
> reconnects. It doesn't ever have the opportunity to dispatch a message
> the Actor doesn't even connect to Redis before it is terminated.
> Response times are something like 0.0011 seconds, nothing is being
> done.
> 
> I'd prefer not to pull in Sinatra::Async, I'd like to know why it
> works with Thin out-of-the-box, but not Rainbows!

I would too!  I wasn't aware of stream(:keep_open) until now.
I'll check it out soon.

From andrew at andrewloe.com  Wed Dec  5 02:16:15 2012
From: andrew at andrewloe.com (W. Andrew Loe III)
Date: Tue, 4 Dec 2012 18:16:15 -0800
Subject: Sinatra stream(:keep_open) not holding connections open
In-Reply-To: <20121205014454.GA24401@dcvr.yhbt.net>
References: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>
	<20121205014454.GA24401@dcvr.yhbt.net>
Message-ID: <CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg@mail.gmail.com>

I've pushed a very very stripped down example that just echos the time.
https://github.com/loe/sinatra-sse

$ bundle install
$ foreman start rainbows
$ foreman start thin

Rainbows listens on 5000 and hangs up immediately.
Thin listens on 5001 and keeps the connection open, you'll see the
time in your browser.

On Tue, Dec 4, 2012 at 5:44 PM, Eric Wong <normalperson at yhbt.net> wrote:
> "W. Andrew Loe III" <andrew at andrewloe.com> wrote:
>> I'm running Rainbows with EventMachine and struggling to get it to
>> keep the connections open.
>>
>> First, here is my code: https://gist.github.com/c33598db8a113b574165
>
> I'll take a closer look later today.
>
> Can you use a public gist (or any other public git repo) instead of a
> private one?  Private gists make no sense for public bug reports.
>
>> This application works on Thin.
>>
>> The main loop handles the HTTP request and creates a Celluloid Actor
>> that hangs out waiting on messages from Redis, when it receives a
>> message it dispatches it to the stream which EM then pushes out to the
>> client. Eventually the client will disconnect (this is Server-Sent
>> Events) and the Actor is torn down.
>
> Can you reproduce the issue with a stripped down application that
> does not depend on Redis?  How about one that does not depend on
> Celluloid?
>
>> I want to run under Rainbows! for the hot upgrades and to re-use all
>> of the wonderful scripts I have for managing my Rails/Unicorn stack.
>> What I see is the GET handler executed, the Actor spawned, then the
>> connection is closed and the Actor is terminated. The browser then
>> reconnects. It doesn't ever have the opportunity to dispatch a message
>> the Actor doesn't even connect to Redis before it is terminated.
>> Response times are something like 0.0011 seconds, nothing is being
>> done.
>>
>> I'd prefer not to pull in Sinatra::Async, I'd like to know why it
>> works with Thin out-of-the-box, but not Rainbows!
>
> I would too!  I wasn't aware of stream(:keep_open) until now.
> I'll check it out soon.
> _______________________________________________
> Rainbows! mailing list - rainbows-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying

From normalperson at yhbt.net  Wed Dec  5 03:18:48 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Wed, 5 Dec 2012 03:18:48 +0000
Subject: Sinatra stream(:keep_open) not holding connections open
In-Reply-To: <CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg@mail.gmail.com>
References: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>
	<20121205014454.GA24401@dcvr.yhbt.net>
	<CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg@mail.gmail.com>
Message-ID: <20121205031848.GA32757@dcvr.yhbt.net>

"W. Andrew Loe III" <andrew at andrewloe.com> wrote:
> I've pushed a very very stripped down example that just echos the time.
> https://github.com/loe/sinatra-sse

Thanks for the test case, I've just pushed out the following fix
to "master" of git://bogomips.org/rainbows.git

I've pushed a rainbows 4.4.1.1.gd5c8c prerelease to RubyGems.org
It should be installable with: gem install --pre rainbows

I expect this to work for you, but be sure to let me know if it doesn't :x

Since this is a pretty small fix, I think I'll push 4.4.2 final
very soon.

>From d5c8cc8b51619f0d33f75036c53e3936ad2749b2 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson at yhbt.net>
Date: Wed, 5 Dec 2012 03:08:19 +0000
Subject: [PATCH] event_machine: properly defer body.close for async

Calling body.close in the normal write_response() code path
is incorrect, and only worked out of sheer luck with
Cramp and async_sinata.

This change allows stream(:keep_open) in Sinatra to work
properly.

Thanks to W. Andrew Loe III for the informative bug report
and reproducible test case.

ref: http://mid.gmane.org/CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg at mail.gmail.com
---
 lib/rainbows/event_machine/client.rb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/rainbows/event_machine/client.rb b/lib/rainbows/event_machine/client.rb
index e56931f..fc0dfe3 100644
--- a/lib/rainbows/event_machine/client.rb
+++ b/lib/rainbows/event_machine/client.rb
@@ -64,8 +64,11 @@ class Rainbows::EventMachine::Client < EM::Connection
     @state = :headers if alive
     if body.respond_to?(:errback) && body.respond_to?(:callback)
       @deferred = body
+      write_headers(status, headers, alive)
+      write_body_each(body)
       deferred_errback(body)
       deferred_callback(body, alive)
+      return
     elsif body.respond_to?(:to_path)
       st = File.stat(path = body.to_path)
 
-- 
Eric Wong

From andrew at andrewloe.com  Wed Dec  5 06:09:53 2012
From: andrew at andrewloe.com (W. Andrew Loe III)
Date: Tue, 4 Dec 2012 22:09:53 -0800
Subject: Sinatra stream(:keep_open) not holding connections open
In-Reply-To: <20121205031848.GA32757@dcvr.yhbt.net>
References: <CA+-9oNeKevgZOvKUPs-3VF7qstn8BcNw3_Py0M50h-j5-GzvOA@mail.gmail.com>
	<20121205014454.GA24401@dcvr.yhbt.net>
	<CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg@mail.gmail.com>
	<20121205031848.GA32757@dcvr.yhbt.net>
Message-ID: <CA+-9oNfR9zM4phTShBsyjweAZBBYWF5egHw7MU8gcBf3f=fAJw@mail.gmail.com>

Works great! Thank you very much for the quick response and patch!

Feature Request: https://github.com/celluloid/celluloid-io
Celluloid::IO backend, the spiritual successor to cool.io. I'm going
to take a stab at this soon, but wanted to get something out.

On Tue, Dec 4, 2012 at 7:18 PM, Eric Wong <normalperson at yhbt.net> wrote:
> "W. Andrew Loe III" <andrew at andrewloe.com> wrote:
>> I've pushed a very very stripped down example that just echos the time.
>> https://github.com/loe/sinatra-sse
>
> Thanks for the test case, I've just pushed out the following fix
> to "master" of git://bogomips.org/rainbows.git
>
> I've pushed a rainbows 4.4.1.1.gd5c8c prerelease to RubyGems.org
> It should be installable with: gem install --pre rainbows
>
> I expect this to work for you, but be sure to let me know if it doesn't :x
>
> Since this is a pretty small fix, I think I'll push 4.4.2 final
> very soon.
>
> >From d5c8cc8b51619f0d33f75036c53e3936ad2749b2 Mon Sep 17 00:00:00 2001
> From: Eric Wong <normalperson at yhbt.net>
> Date: Wed, 5 Dec 2012 03:08:19 +0000
> Subject: [PATCH] event_machine: properly defer body.close for async
>
> Calling body.close in the normal write_response() code path
> is incorrect, and only worked out of sheer luck with
> Cramp and async_sinata.
>
> This change allows stream(:keep_open) in Sinatra to work
> properly.
>
> Thanks to W. Andrew Loe III for the informative bug report
> and reproducible test case.
>
> ref: http://mid.gmane.org/CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg at mail.gmail.com
> ---
>  lib/rainbows/event_machine/client.rb | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/lib/rainbows/event_machine/client.rb b/lib/rainbows/event_machine/client.rb
> index e56931f..fc0dfe3 100644
> --- a/lib/rainbows/event_machine/client.rb
> +++ b/lib/rainbows/event_machine/client.rb
> @@ -64,8 +64,11 @@ class Rainbows::EventMachine::Client < EM::Connection
>      @state = :headers if alive
>      if body.respond_to?(:errback) && body.respond_to?(:callback)
>        @deferred = body
> +      write_headers(status, headers, alive)
> +      write_body_each(body)
>        deferred_errback(body)
>        deferred_callback(body, alive)
> +      return
>      elsif body.respond_to?(:to_path)
>        st = File.stat(path = body.to_path)
>
> --
> Eric Wong
> _______________________________________________
> Rainbows! mailing list - rainbows-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying

From normalperson at yhbt.net  Thu Dec  6 11:46:26 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Thu, 6 Dec 2012 11:46:26 +0000
Subject: [ANN] Rainbows! 4.4.2 - EventMachine async.callback fix
Message-ID: <20121206114626.GA1805@dcvr.yhbt.net>

Changes:

One bugfix allows stream(:keep_open) in Sinatra to work
properly.  No changes for non-EM users.

Thanks to W. Andrew Loe III for the informative bug report
and reproducible test case.

ref: http://mid.gmane.org/CA+-9oNd1EFqsniPkkPTwu5opTCinbM7-2KHoXov7+y3LE4s4Tg at mail.gmail.com

* http://rainbows.rubyforge.org/
* rainbows-talk at rubyforge.org
* git://bogomips.org/rainbows.git
* http://rainbows.rubyforge.org/NEWS.atom.xml

From godfat at godfat.org  Tue Dec 18 11:09:51 2012
From: godfat at godfat.org (Lin Jen-Shin (godfat))
Date: Tue, 18 Dec 2012 19:09:51 +0800
Subject: negative timeout in Rainbows::Fiber::Base
In-Reply-To: <20120928192449.GB14292@dcvr.yhbt.net>
References: <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ@mail.gmail.com>
	<20120829211707.GA22726@dcvr.yhbt.net>
	<CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ@mail.gmail.com>
	<20120831013731.GA16613@dcvr.yhbt.net>
	<CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ@mail.gmail.com>
	<20120905232739.GA25153@dcvr.yhbt.net>
	<CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw@mail.gmail.com>
	<20120922194222.GA6839@dcvr.yhbt.net>
	<CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw@mail.gmail.com>
	<20120928191132.GA14292@dcvr.yhbt.net>
	<20120928192449.GB14292@dcvr.yhbt.net>
Message-ID: <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q@mail.gmail.com>

On Sat, Sep 29, 2012 at 3:24 AM, Eric Wong <normalperson at yhbt.net> wrote:
> Eric Wong <normalperson at yhbt.net> wrote:
>> So the _actual_ Content-Length that's sent is zero?
>>
>> Rainbows! should drop a connection if an exception is raised while
>> sending the response body, perhaps the heroku router is confused
>> by that?
>
> Instead of trying "keepalive_timeout 0", you can also try using the
> following middleware.  This only works for Content-Length responses
> (which seems to be your case)
>
> Of course, like all middleware, this makes your stack deeper, so
> maybe it's not good, either...
>
> Furthermore, this code is totally untested, it may not even compile,
> but I hope you get the idea and fix trivial errors :)
[...]

Thank you again for the helps.  Since the conference (http://rubyconf.tw/2012)
had ended, I got much more time to work with this now.

I didn't try this middleware because now I feel it might be the problem
in their application, given that it used a ton of gems which might have
some thread-safety issues.  On the other hand, the person who asked
me about this telling me that it's fine to run Rainbows! with EventMachine,
but not with ThreadPool.  So it is very likely that it's their issues.

As for the issue on my own last time, since it only happened on an
application which we no longer run, and Rainbows! (Zbatery) runs
totally fine on our true production side, I would guess it's a Heroku
issue at that time and it might have been fixed already?

As a result, I want to ignore this issue from now on, and move forward.

I'll send a patch which contains the concurrency models we use on the
production site later in a new thread.  Also I might start trying to implement
a new concurrency model based on celluloid-io which I believe the API
would be similar to cool.io, thus might not be hard to implement.
Hope we could get there soon :)

From normalperson at yhbt.net  Tue Dec 18 19:19:43 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 18 Dec 2012 19:19:43 +0000
Subject: negative timeout in Rainbows::Fiber::Base
In-Reply-To: <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q@mail.gmail.com>
References: <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ@mail.gmail.com>
	<20120831013731.GA16613@dcvr.yhbt.net>
	<CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ@mail.gmail.com>
	<20120905232739.GA25153@dcvr.yhbt.net>
	<CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw@mail.gmail.com>
	<20120922194222.GA6839@dcvr.yhbt.net>
	<CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw@mail.gmail.com>
	<20120928191132.GA14292@dcvr.yhbt.net>
	<20120928192449.GB14292@dcvr.yhbt.net>
	<CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q@mail.gmail.com>
Message-ID: <20121218191943.GA7743@dcvr.yhbt.net>

"Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> Thank you again for the helps.  Since the conference (http://rubyconf.tw/2012)
> had ended, I got much more time to work with this now.

No problem!

> I didn't try this middleware because now I feel it might be the problem
> in their application, given that it used a ton of gems which might have
> some thread-safety issues.  On the other hand, the person who asked
> me about this telling me that it's fine to run Rainbows! with EventMachine,
> but not with ThreadPool.  So it is very likely that it's their issues.

They probably a thread-safety issue.  I've used ThreadPool/Spawn heavily
myself (much more than EM) with great success.  (I always verify any
gems I use are thread-safe)

> As for the issue on my own last time, since it only happened on an
> application which we no longer run, and Rainbows! (Zbatery) runs
> totally fine on our true production side, I would guess it's a Heroku
> issue at that time and it might have been fixed already?

*shrug*  Maybe check with Heroku changelogs/news releases if they
publish those.

> As a result, I want to ignore this issue from now on, and move forward.
> 
> I'll send a patch which contains the concurrency models we use on the
> production site later in a new thread.  Also I might start trying to implement
> a new concurrency model based on celluloid-io which I believe the API
> would be similar to cool.io, thus might not be hard to implement.
> Hope we could get there soon :)

Alright, looking forward to it.  I was hoping to look at celluloid-io
myself last week, but got sidetracked by something else

From godfat at godfat.org  Tue Dec 18 20:36:17 2012
From: godfat at godfat.org (Lin Jen-Shin (godfat))
Date: Wed, 19 Dec 2012 04:36:17 +0800
Subject: [PATCH] close_connection_after_writing only if not deferred,
	as in cool.io
In-Reply-To: <CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
Message-ID: <CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>

I don't have 100% confidence about this,
but running test suite "t/t0002-graceful.sh"
during my development for EventMachineThreadPool
would need this to properly pass the tests.

I think this would be needed while using
`throw :async' as well?

The patch is also available on Github:
https://github.com/godfat/rainbows/pull/1

>From bd5c38005f44bbe003bd416f7f871f15b0010c85 Mon Sep 17 00:00:00 2001
From: Lin Jen-Shin <godfat at godfat.org>
Date: Wed, 19 Dec 2012 04:03:55 +0800
Subject: [PATCH] close_connection_after_writing only if not deferred, as in
 cool.io

---
 lib/rainbows/event_machine/client.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/rainbows/event_machine/client.rb
b/lib/rainbows/event_machine/client.rb
index fc0dfe3..ebb6f17 100644
--- a/lib/rainbows/event_machine/client.rb
+++ b/lib/rainbows/event_machine/client.rb
@@ -28,7 +28,7 @@ class Rainbows::EventMachine::Client < EM::Connection

   def quit
     super
-    close_connection_after_writing
+    close_connection_after_writing if nil == @deferred
   end

   def app_call input
-- 
1.8.0.2

From normalperson at yhbt.net  Tue Dec 18 21:45:38 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 18 Dec 2012 21:45:38 +0000
Subject: [PATCH] close_connection_after_writing only if not deferred,
	as in cool.io
In-Reply-To: <CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
Message-ID: <20121218214538.GA12275@dcvr.yhbt.net>

"Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> I don't have 100% confidence about this,
> but running test suite "t/t0002-graceful.sh"
> during my development for EventMachineThreadPool

Heh, I gave that a shot way back in the day but never got it working
to my satisfaction.  Perhaps your fix is what is needed...

> would need this to properly pass the tests.
> 
> I think this would be needed while using
> `throw :async' as well?

I don't think so, you just need to set @deferred=nil in a few places
before calling quit.

I've updated the patch and commit message.  Will apply unless
you have objections:

--------------------------------8<----------------------------------

>From 42bf1f6de55b82af46fd8255453036c6582b7f19 Mon Sep 17 00:00:00 2001
From: Lin Jen-Shin <godfat at godfat.org>
Date: Wed, 19 Dec 2012 04:03:55 +0800
Subject: [PATCH] event_machine: avoid close on deferred response

close_connection_after_writing only if not deferred, as in
cool.io

Deferred responses may buffer more data down the line, so
keep the connection alive if we have a deferred response
body.

[ew: clear @deferred when we really want to quit,
 updated commit message]

Acked-by: Eric Wong <normalperson at yhbt.net>
---
 lib/rainbows/event_machine/client.rb | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/rainbows/event_machine/client.rb b/lib/rainbows/event_machine/client.rb
index fc0dfe3..f3c2070 100644
--- a/lib/rainbows/event_machine/client.rb
+++ b/lib/rainbows/event_machine/client.rb
@@ -28,7 +28,7 @@ class Rainbows::EventMachine::Client < EM::Connection
 
   def quit
     super
-    close_connection_after_writing
+    close_connection_after_writing if nil == @deferred
   end
 
   def app_call input
@@ -48,6 +48,7 @@ class Rainbows::EventMachine::Client < EM::Connection
   def deferred_errback(orig_body)
     @deferred.errback do
       orig_body.close if orig_body.respond_to?(:close)
+      @deferred = nil
       quit
     end
   end
@@ -103,7 +104,8 @@ class Rainbows::EventMachine::Client < EM::Connection
 
   def next!
     @deferred.close if @deferred.respond_to?(:close)
-    @hp.keepalive? ? receive_data(@deferred = nil) : quit
+    @deferred = nil
+    @hp.keepalive? ? receive_data(nil) : quit
   end
 
   def unbind
-- 
Eric Wong

From godfat at godfat.org  Tue Dec 18 22:20:22 2012
From: godfat at godfat.org (Lin Jen-Shin (godfat))
Date: Wed, 19 Dec 2012 06:20:22 +0800
Subject: [PATCH] close_connection_after_writing only if not deferred, as
	in cool.io
In-Reply-To: <20121218214538.GA12275@dcvr.yhbt.net>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
Message-ID: <CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>

On Wed, Dec 19, 2012 at 5:45 AM, Eric Wong <normalperson at yhbt.net> wrote:
> Heh, I gave that a shot way back in the day but never got it working
> to my satisfaction.  Perhaps your fix is what is needed...

Glad to know that. Hope this time we could reach your satisfaction :P
I have some hard time running the test suites though :(
I can't even pass all tests without any of my patches with:

    make EventMachine

This works better, but still cannot pass everything:

    make -j8 EventMachine

My linux crashed at the moment, and I am too lazy to fix it right now,
thus running tests on my mac. I'll try to make sure that both EventMachine
and EventMachineThreadSpawn fail on the same tests before sending
the patch. Hope this would be good enough.

I'll continue to work on EventMachineThreadPool and EventMachineFiberSpawn
only if EventMachineThreadSpawn works.

>> would need this to properly pass the tests.
>>
>> I think this would be needed while using
>> `throw :async' as well?
>
> I don't think so, you just need to set @deferred=nil in a few places
> before calling quit.

I am not sure if we read it the same way, but what I mean is that
if the application is using `throw :async', we still need the check in
this patch to avoid dropping connections while receiving SIGQUIT.
Thus even if EventMachineThreadSpawn is not included, we still
need that check for regular EventMachine to quit gracefully for
applications use `throw :async'

But anyway, I still don't really understand all the details in Rainbows,
so of course it's very likely that I am simply wrong :P

> I've updated the patch and commit message.  Will apply unless
> you have objections:

Thank you for the corrections and more explanation in the commit log.
I am all for improving it, so credits don't really matter :) You could also
change all the codes and logs in the patch, like fixing my broken English
as well :P

From normalperson at yhbt.net  Tue Dec 18 23:59:54 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Tue, 18 Dec 2012 23:59:54 +0000
Subject: [PATCH] close_connection_after_writing only if not deferred,
	as in cool.io
In-Reply-To: <CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
	<CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
Message-ID: <20121218235954.GA14404@dcvr.yhbt.net>

"Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> On Wed, Dec 19, 2012 at 5:45 AM, Eric Wong <normalperson at yhbt.net> wrote:
> > Heh, I gave that a shot way back in the day but never got it working
> > to my satisfaction.  Perhaps your fix is what is needed...
> 
> Glad to know that. Hope this time we could reach your satisfaction :P
> I have some hard time running the test suites though :(
> I can't even pass all tests without any of my patches with:
> 
>     make EventMachine
> 
> This works better, but still cannot pass everything:
> 
>     make -j8 EventMachine
> 
> My linux crashed at the moment, and I am too lazy to fix it right now,
> thus running tests on my mac. I'll try to make sure that both EventMachine
> and EventMachineThreadSpawn fail on the same tests before sending
> the patch. Hope this would be good enough.

I've had no trouble running tests with various Linux distros.
It's been a long time since I've tested on FreeBSD, and I've
not tested on any other *BSD.  I cannot support non-Free OSes.

Anything in the *err logs?  The tests should leave them around on
failure.

> I'll continue to work on EventMachineThreadPool and EventMachineFiberSpawn
> only if EventMachineThreadSpawn works.
> 
> >> would need this to properly pass the tests.
> >>
> >> I think this would be needed while using
> >> `throw :async' as well?
> >
> > I don't think so, you just need to set @deferred=nil in a few places
> > before calling quit.
> 
> I am not sure if we read it the same way, but what I mean is that
> if the application is using `throw :async', we still need the check in
> this patch to avoid dropping connections while receiving SIGQUIT.
> Thus even if EventMachineThreadSpawn is not included, we still
> need that check for regular EventMachine to quit gracefully for
> applications use `throw :async'
> 
> But anyway, I still don't really understand all the details in Rainbows,
> so of course it's very likely that I am simply wrong :P

The catch :async will set @deferred=true (obfuscated :x)
I'll commit this to clarify the code:

--- a/lib/rainbows/event_machine/client.rb
+++ b/lib/rainbows/event_machine/client.rb
@@ -38,14 +38,17 @@ class Rainbows::EventMachine::Client < EM::Connection
     @env[ASYNC_CALLBACK] = method(:write_async_response)
     @env[ASYNC_CLOSE] = EM::DefaultDeferrable.new
     status, headers, body = catch(:async) {
       APP.call(@env.merge!(RACK_DEFAULTS))
     }
 
-    (nil == status || -1 == status) ? @deferred = true :
+    if (nil == status || -1 == status)
+      @deferred = true
+    else
       ev_write_response(status, headers, body, @hp.next?)
+    end
   end
 
   def deferred_errback(orig_body)
     @deferred.errback do
       orig_body.close if orig_body.respond_to?(:close)
       @deferred = nil

> > I've updated the patch and commit message.  Will apply unless
> > you have objections:
> 
> Thank you for the corrections and more explanation in the commit log.
> I am all for improving it, so credits don't really matter :) You could also
> change all the codes and logs in the patch, like fixing my broken English
> as well :P

Your actual fix was more important than my cleanups :)

From godfat at godfat.org  Fri Dec 28 04:45:24 2012
From: godfat at godfat.org (Lin Jen-Shin (godfat))
Date: Fri, 28 Dec 2012 12:45:24 +0800
Subject: [PATCH] close_connection_after_writing only if not deferred, as
	in cool.io
In-Reply-To: <20121218235954.GA14404@dcvr.yhbt.net>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
	<CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
	<20121218235954.GA14404@dcvr.yhbt.net>
Message-ID: <CAA2_N1ujCb8Yt4FGJNaTN0FdSk6g_iqL5N747Vcxt5etp9RnMw@mail.gmail.com>

On Wed, Dec 19, 2012 at 7:59 AM, Eric Wong <normalperson at yhbt.net> wrote:
> I've had no trouble running tests with various Linux distros.
> It's been a long time since I've tested on FreeBSD, and I've
> not tested on any other *BSD.  I cannot support non-Free OSes.
>
> Anything in the *err logs?  The tests should leave them around on
> failure.

>From last time I checked, there's nothing left in the log. I would
guess it's because some command line tools behave differently,
but Rainbows should work fine.

Here's what I need to comment out: (3 tests, ab, rack-fiber_pool,
and "Content-Length is set correctly in headers" (I guess it's grep
behave differently. I could try to install GNU's version and try again.)

diff --git a/t/t0012-spurious-wakeups-quiet.sh
b/t/t0012-spurious-wakeups-quiet.sh
index 23557b7..b63870f 100755
--- a/t/t0012-spurious-wakeups-quiet.sh
+++ b/t/t0012-spurious-wakeups-quiet.sh
@@ -10,11 +10,11 @@ then
 	AB=$(PATH=/usr/local/sbin:/usr/sbin:$PATH which ab 2>/dev/null || :)
 fi

-if test -z "$AB"
-then
+#if test -z "$AB"
+#then
 	t_info "skipping $T since 'ab' could not be found"
 	exit 0
-fi
+#fi

 t_plan 4 "quiet spurious wakeups for $model"
diff --git a/t/t0600-rack-fiber_pool.sh b/t/t0600-rack-fiber_pool.sh
index 01f28b5..ca8fcb1 100755
--- a/t/t0600-rack-fiber_pool.sh
+++ b/t/t0600-rack-fiber_pool.sh
@@ -10,7 +10,7 @@ esac

 require_check rack/fiber_pool Rack::FiberPool

-t_plan 7 "basic test with rack-fiber_pool gem"
+t_plan 6 "basic test with rack-fiber_pool gem"

 CONFIG_RU=rack-fiber_pool/app.ru

@@ -42,8 +42,8 @@ t_begin "no errors from curl" && {

 t_begin "no errors in stderr" && check_stderr

-t_begin "ensure we hit 3 separate fibers" && {
-	test x3 = x"$(sort < $curl_out | uniq | wc -l)"
-}
+# t_begin "ensure we hit 3 separate fibers" && {
+# 	test x3 = x"$(sort < $curl_out | uniq | wc -l)"
+# }

 t_done
diff --git a/t/t9001-sendfile-to-path.sh b/t/t9001-sendfile-to-path.sh
index 5a9fdcd..7ca450a 100755
--- a/t/t9001-sendfile-to-path.sh
+++ b/t/t9001-sendfile-to-path.sh
@@ -2,7 +2,7 @@
 . ./test-lib.sh
 skip_models StreamResponseEpoll

-t_plan 7 "Sendfile middleware test for $model"
+t_plan 6 "Sendfile middleware test for $model"

 t_begin "configure and start" && {
 	rtmpfiles curl_err
@@ -39,9 +39,9 @@ t_begin "X-Sendfile does not show up in headers" && {
 	fi
 }

-t_begin "Content-Length is set correctly in headers" && {
-	expect=$(wc -c < random_blob)
-	grep "^< Content-Length: $expect" $curl_err
-}
+# t_begin "Content-Length is set correctly in headers" && {
+# 	expect=$(wc -c < random_blob)
+# 	grep "^< Content-Length: $expect" $curl_err
+# }

 t_done

From normalperson at yhbt.net  Fri Dec 28 08:43:37 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Fri, 28 Dec 2012 08:43:37 +0000
Subject: [PATCH] close_connection_after_writing only if not deferred, as
	in cool.io
In-Reply-To: <CAA2_N1ujCb8Yt4FGJNaTN0FdSk6g_iqL5N747Vcxt5etp9RnMw@mail.gmail.com>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
	<CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
	<20121218235954.GA14404@dcvr.yhbt.net>
	<CAA2_N1ujCb8Yt4FGJNaTN0FdSk6g_iqL5N747Vcxt5etp9RnMw@mail.gmail.com>
Message-ID: <20121228084337.GB19512@dcvr.yhbt.net>

"Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> On Wed, Dec 19, 2012 at 7:59 AM, Eric Wong <normalperson at yhbt.net> wrote:
> > I've had no trouble running tests with various Linux distros.
> > It's been a long time since I've tested on FreeBSD, and I've
> > not tested on any other *BSD.  I cannot support non-Free OSes.
> >
> > Anything in the *err logs?  The tests should leave them around on
> > failure.
> 
> >From last time I checked, there's nothing left in the log. I would
> guess it's because some command line tools behave differently,
> but Rainbows should work fine.
> 
> Here's what I need to comment out: (3 tests, ab, rack-fiber_pool,
> and "Content-Length is set correctly in headers" (I guess it's grep
> behave differently. I could try to install GNU's version and try again.)

I definitely don't want to rely on GNU-isms in shell scripts,
it should rely on POSIX behavior.

It /should/ work on *BSD, I'll try again on FreeBSD tomorrow.

From godfat at godfat.org  Fri Dec 28 11:26:12 2012
From: godfat at godfat.org (Lin Jen-Shin (godfat))
Date: Fri, 28 Dec 2012 19:26:12 +0800
Subject: [PATCH] close_connection_after_writing only if not deferred, as
	in cool.io
In-Reply-To: <20121228084337.GB19512@dcvr.yhbt.net>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
	<CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
	<20121218235954.GA14404@dcvr.yhbt.net>
	<CAA2_N1ujCb8Yt4FGJNaTN0FdSk6g_iqL5N747Vcxt5etp9RnMw@mail.gmail.com>
	<20121228084337.GB19512@dcvr.yhbt.net>
Message-ID: <CAA2_N1tCYbJjh_NvdVOQzJ5hvdv5NYnBM+VhP105uRvCzbnF_A@mail.gmail.com>

On Fri, Dec 28, 2012 at 4:43 PM, Eric Wong <normalperson at yhbt.net> wrote:
> I definitely don't want to rely on GNU-isms in shell scripts,
> it should rely on POSIX behavior.
>
> It /should/ work on *BSD, I'll try again on FreeBSD tomorrow.

I got managed to get them all passed now :D
The problem is at `wc`. On my mac, wc would print
some leading spaces even if you use `wc -c < random_blob`.

$ echo 's' | wc -c
       2

After using coreutils from GNU or use sed to strip the spaces,
everything passed, except for the one for Apache ab. I've heard
that ab is broken on mac, and after install ab from:
https://github.com/Homebrew/homebrew-dupes/pull/102
Tests using ab are all passed too.
ref: http://simon.heimlicher.com/articles/2012/07/08/fix-apache-bench-ab-on-os-x-lion

Nothing to do with grep.

Also, after confirming all tests are fine, the only test which cannot
pass for eventmachine with threads is:

    "send big pipelined identity requests"

As you mentioned before, the issue is that there is no easy way to
disable a connection and enable it later on? I tried `pause' and
`resume', but it doesn't seem to work correctly? Or I might have
done something wrong or missed something.

If I took out this line to make Rainbows! buffer everything, it would
also work.

    @_io.shutdown(Socket::SHUT_RD) if @buf.size > 0x1c000

But I guess we shouldn't really buffer them in Ruby. I am still trying
to solve this :/ You can find my working copy here:
https://github.com/godfat/rainbows/pull/2/files
I don't paste it inline here because it's large and I guess we should
make all tests pass. Or we could skip that test for this model?

Another concern is that, the most straightforward implementation
would be subclass Rainbows::EventMachine::Client to make it
app_call in a thread. But there's em_client_class option which
users might provide their own eventmachine connection.

If I did this in a client class, then users who provide their own
client class won't really respect the threads, making using
EventMachine or EventMachineThreadSpawn basically the same.
I think this is somehow unexpected.

But then I realized using another option for threads or not really
complicates the original implementation, this is not what I really
want either. I am not sure if there's a perfect solution for this,
so I just concentrate on passing all tests first...

From normalperson at yhbt.net  Sat Dec 29 13:26:01 2012
From: normalperson at yhbt.net (Eric Wong)
Date: Sat, 29 Dec 2012 13:26:01 +0000
Subject: [PATCH] close_connection_after_writing only if not deferred, as
	in cool.io
In-Reply-To: <CAA2_N1tCYbJjh_NvdVOQzJ5hvdv5NYnBM+VhP105uRvCzbnF_A@mail.gmail.com>
References: <CAA2_N1tHbvs=J--dPZLnq6_owEx3JJZbFoRzjFKcCLBONW=iGA@mail.gmail.com>
	<CAA2_N1s+SefHd7Dotw53+5b=-EeBt_O5Q-LmoKuUy65jZiZ+tA@mail.gmail.com>
	<CAA2_N1scEcW3J=4dmUVvd+UgLwuayqkpfjNfVGLCJw-dXm9_cA@mail.gmail.com>
	<20121218214538.GA12275@dcvr.yhbt.net>
	<CAA2_N1ur-dxw4i9dvMhTNAcskJe+N=4hU2Yuq341TxvJ2KE4kw@mail.gmail.com>
	<20121218235954.GA14404@dcvr.yhbt.net>
	<CAA2_N1ujCb8Yt4FGJNaTN0FdSk6g_iqL5N747Vcxt5etp9RnMw@mail.gmail.com>
	<20121228084337.GB19512@dcvr.yhbt.net>
	<CAA2_N1tCYbJjh_NvdVOQzJ5hvdv5NYnBM+VhP105uRvCzbnF_A@mail.gmail.com>
Message-ID: <20121229132601.GA16997@dcvr.yhbt.net>

"Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> On Fri, Dec 28, 2012 at 4:43 PM, Eric Wong <normalperson at yhbt.net> wrote:
> > I definitely don't want to rely on GNU-isms in shell scripts,
> > it should rely on POSIX behavior.
> >
> > It /should/ work on *BSD, I'll try again on FreeBSD tomorrow.
> 
> I got managed to get them all passed now :D
> The problem is at `wc`. On my mac, wc would print
> some leading spaces even if you use `wc -c < random_blob`.
> 
> $ echo 's' | wc -c
>        2
> 
> After using coreutils from GNU or use sed to strip the spaces,
> everything passed, except for the one for Apache ab. I've heard

Ah, probably good to add a byte_count() function to test-lib.sh
to get around this...  FreeBSD 9.0 does this, too.
I didn't manage to work more on FreeBSD, got distracted by bugs
in other projects...

> Also, after confirming all tests are fine, the only test which cannot
> pass for eventmachine with threads is:
> 
>     "send big pipelined identity requests"
> 
> As you mentioned before, the issue is that there is no easy way to
> disable a connection and enable it later on? I tried `pause' and
> `resume', but it doesn't seem to work correctly? Or I might have
> done something wrong or missed something.

I think that was another issue I had with the EM thread
implementation I tried...

> If I took out this line to make Rainbows! buffer everything, it would
> also work.
> 
>     @_io.shutdown(Socket::SHUT_RD) if @buf.size > 0x1c000
> 
> But I guess we shouldn't really buffer them in Ruby. I am still trying
> to solve this :/ You can find my working copy here:
> https://github.com/godfat/rainbows/pull/2/files
> I don't paste it inline here because it's large and I guess we should
> make all tests pass. Or we could skip that test for this model?

Tests for features are fine to skip, but being open to such a
trivial DoS is not fine...  Perhaps disable keepalive support
entirely?  (you'd have to disable a lot of tests, perhaps
like StreamResponseEpoll :x)

> Another concern is that, the most straightforward implementation
> would be subclass Rainbows::EventMachine::Client to make it
> app_call in a thread. But there's em_client_class option which
> users might provide their own eventmachine connection.

That was for Cramp, I can't remember exactly how it worked...

> If I did this in a client class, then users who provide their own
> client class won't really respect the threads, making using
> EventMachine or EventMachineThreadSpawn basically the same.
> I think this is somehow unexpected.

They'd have to pick the threaded concurrency model.  Maybe
more Ruby code is thread-safe nowadays...

> But then I realized using another option for threads or not really
> complicates the original implementation, this is not what I really
> want either. I am not sure if there's a perfect solution for this,
> so I just concentrate on passing all tests first...

Yes, Rainbows! already has so many options it's hard to keep
things straight :x

I believe EM is one of the concurrency options people actually use in
Rainbows!, so it's important to not break it.  I don't mind copy+pasting
first and eventually factoring out the common parts if possible.