[Mongrel] Uploading Large (100mb+) files

Zed A. Shaw zedshaw at zedshaw.com
Tue Nov 28 17:59:20 EST 2006

On Tue, 28 Nov 2006 15:26:04 -0600 (CST)
Philip Hallstrom <mongrel at philip.pjkh.com> wrote:

> > Quick clarification to what Philip said:  The actual upload doesn't 
> > block mongrel since it's done in a thread before rails gets the uploaded 
> > body.  Now, once Rails gets it the cgi.rb multipart mime processing is 
> > run on the resulting file which can eat your CPU for about 20 
> > seconds/100mb of file (based on my observations).
> You lost me a bit... are you saying the actual upload doesn't block 
> mongrel with or without the plugin?  And when you say "mongrel" do you 
> really mean "rails within mongrel" or just mongrel?
> I'm thinking you mean that uploads don't block *mongrel* ever, but without 
> the plugin uploads block rails within mongrel which affectively blocks any 
> other rails requests.

Here's the process:

1) Mongrel accepts the client, creates a thread, and processes the headers into an HttpRequest object.
2) Then it hits the body, it looks at the content-length and sets up to read that much.
2a) If the body is over 16k then it is streamed to a Tempfile.  Otherwise into a StringIO.
3) Mongrel then reads 16k chunks and writes the results into the IO (Tempfile or StringIO).
4) Once the body is complete, and the content-length checks out, then the HttpRequest is complete.
5) The HttpRequest and an HttpResponse is passed to the RailsHandler for processing.
6) LOCK!
7) Rails runs cgi.rb and a bunch of other stuff on the uploaded file.  At this point you aren't blocked by the remote client's IO since it's already saved by Mongrel, but cgi.rb is really slow so it'll take a while depending on the size of the file to parse out the multi-part mime.
9) Rails processes the request and sends back the response (which is also stuffed into a StringIO).
11) Mongrel now takes the response headers and response body and sends those to the remote client fully threaded.

So, there's only a short part of the process where Rails is actually blocking the upload.

Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

More information about the Mongrel-users mailing list