[Mongrel] http parser

Francis Cianfrocca garbagecat10 at gmail.com
Sun Aug 20 17:04:41 EDT 2006

On 8/20/06, Zed Shaw <zedshaw at zedshaw.com> wrote:
> The EventMachine stuff doesn't really buy you much since they use Ruby's
> select still to process connections.  That means you're still stuck with
> 1024 max files and still stuck with Ruby's threads.  All they do is then
> move IO processing out to a *large* amount of convoluted C++ code.  I'm
> sure you'll love security auditing that. :-)
> They are also working on their Mongrel competitor, but being in true
> open source NIH form couldn't be bothered to borrow the parser I wrote
> and decided to write their own by hand (it's like I don't even say
> anything).  For whatever reason I've always thought those guys were a
> little on the shady side.  Not sure why, they just seem desperate for
> cash or something.
> I'd say play around with it and if you get something interesting working
> then that's cool, but in the end adding even more compiled code that I'd
> have to distribute that's then tied to a group of developers who're
> obviously trying to make bank off my work isn't going to go over well.
> FYI, I'm also working on a different approach that moves all of the HTTP
> processing into an ultra fast ucontext/poll based IO system.  I'm hoping
> it'll work with any language too (Python baby).
> But don't let this discourage you, Mongrel is open source so people can
> try these things out.  You never know, it could end up being the bee's
> knees on a platform like win32.

It's hard to know how to answer this, given that I've felt the Ruby
community has generally been about looking for good answers to
technical problems, and not about making unsupported statements that
have a strong odor of suspicion and insecurity about them.

EventMachine is not and has never been fundamentally about HTTP. It
started out as a general framework for writing fast, event-driven
programs without having to use threads, and it wasn't designed only
for working with Ruby. It is stuck with a limit of the number of
descriptors when used with Ruby because we hook Ruby's select
implementation in order to interoperate with Ruby threads, which EM
does with no problems whatsoever.

I wrote the EM I/O engine in C++ for two primary reasons: first
because this code base builds on more than ten years of experience
with extremely high-speed network servers on a range of different
platforms. And second, because when I wrote it (last April), Ruby
didn't have proper nonblocking I/O. The Ruby-core guys generously
added proper nonblocking support at the end of May, and at this point
EM has a nearly-complete pure Ruby implementation, that has an
identical API and will be used transparently on platforms where the
extension fails to compile.

I just checked EM's C++ code, and it's only 4200 lines. That's not a
*large* amount of code. It's a *small* amount of code. If you think
it's "convoluted," then I have to say you probably haven't tried to
read it. (It's also possible that you're not that strong a C++
programmer.) I've been through some hard-core source-code audits in my
time, and I'm confident this code would pass without an abnormal
amount of effort. And at any rate, it's wide open on Rubyforge SCM, so
anyone and everyone is invited to take a crack at finding the security
holes in it. We'll gratefully accept any patch correcting security

Working on a Mongrel competitor: I'm not particularly interested in
competing against Mongrel for a couple of reasons. Most importantly,
I'm not entirely sure what Mongrel's goal in life is. It seems nice to
have a better-performing Webrick, but you've also said that you expect
Mongrel to be deployed in conjunction with Apache or Lighty, so I'm
left assuming the primary value-added is that people can save the
trouble of configuring FastCGI or such stuff. Also, it seems nice to
be able to cluster Rails instances intelligently. And you have said to
me yourself that adding faster I/O to Rails is of limited value
because there are so many performance problems in Rails itself. And
none of those are problems I'm interested in solving any differently
than you have.

I'm not quite sure what you think anything I'm working on is that
might be competitive to Mongrel. I know that Mongrel has an HTTP
parser in it, and we have one too, but that's not a whole lot of
overlap. I've read your statements on parsing HTTP, and I'm not
terribly convinced by your approach, which seems to be all about
implementing a very complex RFC as completely and strictly as
possible. I've been writing practical high-speed HTTP parsers for
almost ten years now, and I know a lot about which parts of the
standard(s) really matter and which matter less, and also about which
parts should be treated in a relaxed way. You and I are trying to
solve different problems, and for my problem-domain, your chosen
approach isn't too interesting. You're trying to write a well-behaved
proxying layer between Rails and standard web servers. I'm trying to
write a general framework for event-driven applications that can
easily incoroporate a range of different protocols. Speed and ease of
development matter a lot more to me than strict conformance with RFC

>>>For whatever reason I've always thought those guys were a
little on the shady side.  Not sure why, they just seem desperate for
cash or something.<<<
This borders on libel. I challenge you to produce the evidence on the
basis of which you make these statements. I've been a senior
technology executive and investor for quite a long time. I don't
contribute to open-source projects in order to make a living. Are you
willing to make the same statement in regard to yourself?

>>>but in the end adding even more compiled code that I'd
have to distribute that's then tied to a group of developers who're
obviously trying to make bank off my work isn't going to go over well.<<<
To my knowledge, no one in my group is trying to induce you to add
anything, compiled or otherwise, to your code base. Your statement
sounds like more than just protesting too much: it sounds insecure. I
can't speak for your motivations, but the rest of us just want to work
with the best possible technology, and investigate new ideas to see if
they lead to interesting places.

Zed, you're an important guy in the community and you've written some
impressive technology. But you're showing a personal side that is
quite disappointing.

More information about the Mongrel-users mailing list