negative timeout in Rainbows::Fiber::Base
Lin Jen-Shin (godfat)
godfat at godfat.org
Sat Sep 22 09:52:11 UTC 2012
On Thu, Sep 6, 2012 at 7:27 AM, Eric Wong <normalperson at yhbt.net> wrote:
> Simple: we don't read from the socket at all while processing a request
> Extra data the client sends gets buffered in the kernel, eventually TCP
> backoff will kick in and the Internet stays usable :>
Understood, thanks! I guess this is simple enough that it's a nice
trade off, since not every clients would do heavy pipelining.
> With the inability to easily stop read callbacks via EM, the socket
> buffers constantly get drained so the clients are able to keep sending
> data. The only option I've found for Rainbows! + EM was to issue
> shutdown(SHUT_RD) if a client attempts to pipeline too much (which
> never happens with legitimate clients AFAIK).
And the buffer size you set: 0x1c000, I guess that's large enough
for most cases.
> It shouldn't matter for nginx, I don't think nginx will (ever) pipeline
> to a backend. Nowadays nginx can do persistent connections to backends,
> though I'm not sure how much of a benefit it is for local sockets.
Here's a problem troubles me for a long time, and I failed to figure out
what's going on and cannot find a proper fix. I didn't mention this before,
because I don't think it's Zbatery/Rainbows' issue, but it turns out that
it might be related, and perhaps you would be interested to provide
some hints, I would much appreciate it.
So we're running all our apps on http://www.heroku.com
It's a hosting service that you simply `git push` to deploy your app,
worrying not all the complexity regarding deployment.
There's a very weird issue running Zbatery/Rainbows on it.
I don't remember if Unicorn would be suffering from that
since I don't run Unicorn directly for a while now.
The problem comes with Rails 3 and its "assets pipeline" feature.
css, etc) no longer simple assets which could be served directly
via nginx or some other static web server.
Like, developers could name their coffeescripts files as:
And when a client is requesting "application.js", that "assets pipeline"
and return it to the client.
We can also "precompile" those assets before deploying to production.
Heroku would do this if it detects that the application is deploying is
built with Rails 3.
If this precompile thing is done successfully, then there's no problems,
at least in some of the cases.... But what if precompile failed? In my cases,
it would usually cause system stack overflow whenever it tries to compile
it on the fly.
Oh, suddenly I think maybe it's because I would wrap a fiber around
each request in some cases... and it just used too many method calls.
I am not sure about that... might check it next time I saw it.
Somehow I feel this might be related to persistent connections or
pipelined requests, because last time I didn't write the client class
correctly to handle pipelined requests as you pointed out, it would
also cause some issues regarding "assets pipeline" on Heroku.
I believe Heroku is running nginx in front off Zbatery, and if I didn't
get pipelined requests correctly, it would result timeout for many of
assets the client is asking about. I feel Heroku is using pipelined
requests if the client is using this.
And the most weird thing I can't understand is, I am trying to convince
one of my friend to try out Zbatery, but using ThreadSpawn model
would usually cause serving assets timeout, just like in my case
where I didn't get pipelined requests correct. Switching to EventMachine
solved the problem! Huh?
Moreover, once there are some assets timeout issues on EventMachine,
too. When I tried to debug this, I put some traces into Rainbows,
realizing that sometimes EventMachine didn't call `receive_data'
when receiving some pipelined requests. Could it be an eventmachine bug!?
Sorry that I wrote so much and it's unclear what's going on, since
I don't understand so I can't remember correctly...
If you have any idea for a specific case, I can try to provide all the
information I could collect. I can't get the nginx configuration Heroku
is using though :(
> I think your comment is unfortunately representative of a lot of
> software development nowadays.
> Embrace pessimism and let it be your guide :)
I hope I could understand/realize this sooner :)
More information about the rainbows-talk