Release Name: 0.9.0
Notes:
Rack HTTP server for Unix, fast clients and nothing else
unicorn 0.9.0
We now have support for "Transfer-Encoding: chunked" bodies in
requests. Not only that, Rack applications reading input bodies
get that data streamed off to the client socket on an as-needed
basis. This allows the application to do things like upload
progress notification and even tunneling of arbitrary stream
protocols via bidirectional chunked encoding.
See Unicorn::App::Inetd and examples/git.ru (including the
comments) for an example of tunneling the git:// protocol over
HTTP.
This release also gives applications the ability to respond
positively to "Expect: 100-continue" headers before being rerun
without closing the socket connection. See Unicorn::App::Inetd
for an example of how this is used.
This release is NOT recommended for production use.
Eric Wong (43):
http_request: no need to reset the request
http_request: StringIO is binary for empty bodies (1.9)
http_request: fix typo for 1.9
Transfer-Encoding: chunked streaming input support
Unicorn::App::Inetd: reinventing Unix, poorly :)
README: update with mailing list info
local.mk.sample: publish_doc gzips all html, js, css
Put copyright text in new files, include GPL2 text
examples/cat-chunk-proxy: link to proposed curl(1) patch
Update TODO
Avoid duplicating the "Z" constant
Optimize body-less GET/HEAD requests (again)
tee_input: Don't expose the @rd object as a return value
exec_cgi: small cleanups
README: another note about older Sinatra
tee_input: avoid defining a @rd.size method
Make TeeInput easier to use
test_upload: add tests for chunked encoding
GNUmakefile: more stringent error checking in tests
test_upload: fix ECONNRESET with 1.9
GNUmakefile: allow TRACER= to be specified for tests
test_rails: workaround long-standing 1.9 bug
tee_input: avoid rereading fresh data
"Fix" tests that break with stream_input=false
inetd: fix broken constant references
configurator: provide stream_input (true|false) option
chunked_reader: simpler interface
http_request: force BUFFER to be Encoding::BINARY
ACK clients on "Expect: 100-continue" header
Only send "100 Continue" when no body has been sent
http_request: tighter Transfer-Encoding: "chunked" check
Add trailer_parser for parsing trailers
chunked_reader: Add test for chunk parse failure
TeeInput: use only one IO for tempfile
trailer_parser: set keys with "HTTP_" prefix
TrailerParser integration into ChunkedReader
Unbind listeners as before stopping workers
Retry listen() on EADDRINUSE 5 times ever 500ms
Re-add support for non-portable socket options
Move "Expect: 100-continue" handling to the app
tee_input: avoid ignoring initial body blob
Force streaming input onto apps by default
unicorn 0.9.0
Changes:
commit b67ff587730abfd4e1f78c448ef128315fbf786d
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jul 1 14:15:34 2009 -0700
unicorn 0.9.0
commit 90c0e4666185f29d3484c12ea6d451fcac528376
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jul 1 15:16:49 2009 -0700
Remove cat-chunk-proxy, curl CVS supports non-blocking stdin
Now that upstream curl supports this functionality, there's
no reason to duplicate it here as an example.
commit 06bf73975864b8e16ef1ee977f8424a0e5517fd6
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jul 1 13:59:40 2009 -0700
Force streaming input onto apps by default
This change gives applications full control to deny clients
from uploading unwanted message bodies. This also paves the
way for doing things like upload progress notification within
applications in a Rack::Lint-compatible manner.
Since we don't support HTTP keepalive, so we have more freedom
here by being able to close TCP connections and deny clients the
ability to write to us (and thus wasting our bandwidth).
While I could've left this feature off by default indefinitely
for maximum backwards compatibility (for arguably broken
applications), Unicorn is not and has never been about
supporting the lowest common denominator.
commit ec5d374768ced6aba3fed8a9481d2ac3c07cdb98
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jul 1 13:49:29 2009 -0700
tee_input: avoid ignoring initial body blob
This was causing the first part of the body to be missing when
an HTTP client failed to delay between sending the header and
body in the request.
commit ec14a20474575e77a23b713ee8fcda1e71b1d018
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jul 1 13:32:33 2009 -0700
Move "Expect: 100-continue" handling to the app
This gives the app ability to deny clients with 417 instead of
blindly making the decision for the underlying application. Of
course, apps must be made aware of this.
commit 563d03f649ef31d2aec3505cbbed1e015493b8fc
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 19:16:43 2009 -0700
Re-add support for non-portable socket options
Now that we support tunnelling arbitrary protocols over HTTP as
well as "100 Continue" responses, TCP_NODELAY actually becomes
useful to us. TCP_NODELAY is actually reasonably portable
nowadays; even.
While we're adding non-portable options, TCP_CORK/TCP_NOPUSH can
be enabled, too. Unlike some other servers, these can't be
disabled explicitly/intelligently to force a flush, however.
However, these may still improve performance with "normal" HTTP
applications (Mongrel has always had TCP_CORK enabled in Linux).
While we're adding OS-specific features, we might as well
support TCP_DEFER_ACCEPT in Linux and FreeBSD the "httpready"
accept filter to prevent abuse.
These options can all be enabled on a per-listener basis.
commit d247b5d95a3ad2de65cc909db21fdfbc6194b4c9
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 17:47:55 2009 -0700
Retry listen() on EADDRINUSE 5 times ever 500ms
This number of retries and delay taken directly from nginx
commit 8c2040127770e40e344a927ddc187bf801073e33
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 17:33:15 2009 -0700
Unbind listeners as before stopping workers
This allows another process to take our listeners
sooner rather than later.
commit 20bf660a3efff9229c81b3b3a0feb6844bc27a7c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 17:15:56 2009 -0700
TrailerParser integration into ChunkedReader
Support for the "Trailer:" header and associated Trailer
lines should be reasonably well supported now
commit f01c1d4071e8ce30aa6806fd3cd8eec7491bf06c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 17:15:34 2009 -0700
trailer_parser: set keys with "HTTP_" prefix
commit 8f648549d57df3d1b9796c70d2e11ba6ffb19486
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 17:11:31 2009 -0700
TeeInput: use only one IO for tempfile
We can actually just use one IO and file descriptor here and
simplify the code while we're at it.
commit 2d3f12f0ca25ddfdec5b7d92f76489dcc1f26cf7
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 14:15:57 2009 -0700
chunked_reader: Add test for chunk parse failure
I'd honestly be more comfortable doing this in C (and possibly
adapting the code from the libcurl internals since that code has
been very well-tested).
commit 608e6243a2b15bfc28c3524ed45d5fc7598e8565
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 30 13:47:41 2009 -0700
Add trailer_parser for parsing trailers
Eventually this (and ChunkedReader) may be done in C/Ragel
along with the existing HttpParser.
commit 86cd3fec9a02352d10b53bac02f98267d2b77bdd
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 22:27:44 2009 -0700
http_request: tighter Transfer-Encoding: "chunked" check
Don't allow misbehaving clients to mispell "chunked"
commit 7e36e29a69c4adb777e82a91a381bd680967c055
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 21:24:59 2009 -0700
Only send "100 Continue" when no body has been sent
Under slow/inconsistent network conditions or overly aggressive
clients, there is a possibility we could've already started
reading the body. In those cases, don't bother responding
to the expectation to continue since the client has already
started sending a message body.
commit 516c4a8686911a3b6c5e1837d183cd6f515e877c
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 19:21:34 2009 -0700
ACK clients on "Expect: 100-continue" header
By responding with a "HTTP/1.1 100 Continue" response to
encourage a client to send the rest of the body.
This is part of the HTTP/1.1 standard but not often implemented
by servers:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3
This will speed up curl uploads since curl sleeps up to 1 second if
no response is received:
http://curl.haxx.se/docs/faq.html#My_HTTP_POST_or_PUT_requests_are
commit ec3ac2e4291026a3ebf687d7d0c45c34acac111e
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 18:21:52 2009 -0700
http_request: force BUFFER to be Encoding::BINARY
Not sure why this hasn't been an issue yet, but better
safe than sorry with data integrity...
commit 41f6f65a6cb7152b5181704373d7da62704a62e8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 18:01:41 2009 -0700
chunked_reader: simpler interface
This won't be heavily used enough to make preallocation worth
the effort. While we're at it, don't enforce policy by forcing
the readpartial buffer to be Encoding::BINARY (even though it
/should/ be :), it's up to the user of the interface to decide.
commit 4f05fb1a3b44f8eab1a9dda26d5b115f33a149cd
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 05:27:18 2009 -0700
configurator: provide stream_input (true|false) option
The default is false because some applications were not
written to handle partial reads (even though IO#read allows
it, not just IO#readpartial).
commit 1e10654f81e74c4d11ab538b16dcc1b7bd36cb7f
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 11:33:02 2009 -0700
inetd: fix broken constant references
This has been totally broken since
commit b0013b043a15d77d810d5965157766c1af364db2
"Avoid duplicating the "Z" constant"
commit 2c53b32bf651966d3eb39850cc6be019eda51df0
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 05:11:13 2009 -0700
"Fix" tests that break with stream_input=false
commit 0d3aa1eb6bc7646c9bde78b0fae4e34bc5876421
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 04:58:21 2009 -0700
tee_input: avoid rereading fresh data
Oops!
commit cfed47020a39291504877ce2a3463ed268fed9c8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 03:59:00 2009 -0700
test_rails: workaround long-standing 1.9 bug
Now that I've beefed out my Makefile to detect errors,
I've noticed this test has been failing under 1.9 for
a while now. Currently no released version of Rack(1.0)
or Rails(2.3.2.1) supports this.
commit 0127374ed2e4028ca8e164c2cfba357b53960a10
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 03:06:56 2009 -0700
GNUmakefile: allow TRACER= to be specified for tests
This can allow you to run make with:
TRACER='strace -f -o $(t).strace -s 100000'
to debug a test failure (it should be usable with truss,
ltrace, and other similar tools).
commit 93f55bb8802866e8f9a2d741ecec6a75973f2c72
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 03:03:35 2009 -0700
test_upload: fix ECONNRESET with 1.9
This has been broken since
6945342a1f0a4caaa918f2b0b1efef88824439e0
"Transfer-Encoding: chunked streaming input support" but
somehow never caught by me or anyone else.
commit d7b3fb8053315c2ecf4d66199b028df5070e1ab8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jun 29 02:39:04 2009 -0700
GNUmakefile: more stringent error checking in tests
commit 7cbdf6ba4c3463dfc1cac5cb01d4241c02be7fde
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Jun 27 16:00:02 2009 -0700
test_upload: add tests for chunked encoding
Additionally, provide verifications for sizes after-the-fact
to avoid slamming all of our input into the server.
commit 14ce889cc12628405b0f6a1842b7e1ad09ca4b7b
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Jun 27 15:58:45 2009 -0700
Make TeeInput easier to use
The complexity of making the object persistent isn't worth the
potential performance gain here.
commit 665717d9d339748447e26d3eb0e34c9f6c64ce73
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 26 15:52:39 2009 -0700
tee_input: avoid defining a @rd.size method
We don't ever expose the @rd object to the public so
Rack-applications won't ever call size() on it.
commit 1f0d2e953af07b5bb904953767ee8c66deea669a
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Jun 25 20:25:08 2009 -0700
README: another note about older Sinatra
Older Sinatra would blindly try to run Mongrel or Thin at_exit.
This causes strange behavior to happen when Unicorn workers are
exited.
commit 8e030bcd34da6296b1bda95f151c2ac4b777f617
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Jun 25 16:38:13 2009 -0700
exec_cgi: small cleanups
* avoid '' strings for GC-friendliness
* Ensure the '' we do need is binary for 1.9
* Disable passing the raw rack.input object to the child process
This is never possible with our new TeeInput wrapper.
commit 1574d8ce531d79fc905be2f319fa703fb0710e96
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Jun 25 16:37:01 2009 -0700
tee_input: Don't expose the @rd object as a return value
Pay a performance penalty and always proxy reads through our
TeeInput object to ensure nobody closes our internal reader.
commit 4651cd8b7c10d18bc745b84ebc7a55aad07d6077
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jun 10 16:00:08 2009 -0700
Optimize body-less GET/HEAD requests (again)
No point in making syscalls to deal with empty bodies.
Reinstate usage of the NULL_IO object which allows us
to avoid allocating new objects.
commit b0013b043a15d77d810d5965157766c1af364db2
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jun 9 16:22:55 2009 -0700
Avoid duplicating the "Z" constant
Trying not to repeat ourselves. Unfortunately, Ruby 1.9 forces
us to actually care about encodings of arbitrary byte sequences.
commit f3e1cd9b7728b85f346d5588268066c9ef9d3cf3
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Jun 7 13:28:25 2009 -0700
Update TODO
commit 67c2b02fdb4a863f1a7934b2c907b6d11c236f75
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Jun 7 03:41:20 2009 -0700
examples/cat-chunk-proxy: link to proposed curl(1) patch
Then hopefully soon we'll be able to get rid of this script...
commit a834b9fe0f3085f979d7f378d3cb72e0a052d85c
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Jun 6 13:53:35 2009 -0700
Put copyright text in new files, include GPL2 text
Just clarifying the license terms of the new code. Other files
should really have this notice in there as well.
commit 7fdf4388a637b687c3996a4e5943db4610ba6f6e
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Jun 6 13:41:19 2009 -0700
local.mk.sample: publish_doc gzips all html, js, css
While we're at it, use the rsyncable flag with gzip here
to reduce bandwidth usage on my end.
commit c302b3281b71897f6e2854562e6f77447831e61e
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Jun 6 13:14:20 2009 -0700
README: update with mailing list info
commit a64913eafbee3501a677b1232470838a4ad0fc65
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 5 22:16:47 2009 -0700
Unicorn::App::Inetd: reinventing Unix, poorly :)
This includes an example of tunneling the git protocol inside a
TE:chunked HTTP request. The example is unfortunately contrived
in that it relies on the custom examples/cat-chunk-proxy.rb
script in the client. My initial wish was to have a generic
tool like curl(1) operate like this:
cat > ~/bin/cat-chunk-proxy.sh <<EOF
#!/bin/sh
exec curl -sfNT- http://$1:$2/
EOF
chmod +x ~/bin/cat-chunk-proxy.sh
GIT_PROXY_COMMAND=cat-chunk-proxy.sh git clone git://0:8080/foo
Unfortunately, curl will attempt a blocking read on stdin before
reading the TCP socket; causing the git-clone consumer to
starve. This does not appear to be a problem with the new
server code for handling chunked requests.
commit 6945342a1f0a4caaa918f2b0b1efef88824439e0
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 5 18:03:46 2009 -0700
Transfer-Encoding: chunked streaming input support
This adds support for handling POST/PUT request bodies sent with
chunked transfer encodings ("Transfer-Encoding: chunked").
Attention has been paid to ensure that a client cannot OOM us by
sending an extremely large chunk.
This implementation is pure Ruby as the Ragel-based
implementation in rfuzz didn't offer a streaming interface. It
should be reasonably close to RFC-compliant but please test it
in an attempt to break it.
The more interesting part is the ability to stream data to the
hosted Rack application as it is being transferred to the
server. This can be done regardless if the input is chunked or
not, enabling the streaming of POST/PUT bodies can allow the
hosted Rack application to process input as it receives it. See
examples/echo.ru for an example echo server over HTTP.
Enabling streaming also allows Rack applications to support
upload progress monitoring previously supported by Mongrel
handlers.
Since Rack specifies that the input needs to be rewindable, this
input is written to a temporary file (a la tee(1)) as it is
streamed to the application the first time. Subsequent rewinded
reads will read from the temporary file instead of the socket.
Streaming input to the application is disabled by default since
applications may not necessarily read the entire input body
before returning. Since this is a completely new feature we've
never seen in any Ruby HTTP application server before, we're
taking the safe route by leaving it disabled by default.
Enabling this can only be done globally by changing the
Unicorn HttpRequest::DEFAULTS hash:
Unicorn::HttpRequest::DEFAULTS["unicorn.stream_input"] = true
Similarly, a Rack application can check if streaming input
is enabled by checking the value of the "unicorn.stream_input"
key in the environment hashed passed to it.
All of this code has only been lightly tested and test coverage
is lacking at the moment.
[1] - http://tools.ietf.org/html/rfc2616#section-3.6.1
commit a48695449f49e6900819fed472f23408c62b5501
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 5 00:19:05 2009 -0700
http_request: fix typo for 1.9
commit 4716aab9530cabb14448a5123865b9f79b77f40c
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat May 30 09:15:50 2009 -0700
http_request: StringIO is binary for empty bodies (1.9)
commit 34b916e3c237f32d5455869d2823eccb27e9ff24
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat May 30 09:10:29 2009 -0700
http_request: no need to reset the request
That method no longer exists, but Ruby would never know until it
tried to run it. Yes, I miss my compiled languages.
|
