[Mongrel] POST with huge HTTP body

Daniel Brahneborg basic70 at gmail.com
Tue Oct 16 04:02:36 EDT 2007


It looks like it's CGI::QueryExtension#read_body in ActionPack that is the
first one to cause problems. When a PUT or POST request arrives, it reads
the entire thing into memory. Even if that method is patched, there are tons
of other methods that wants to mess around with the request body.
According to the documentation, the CGI class is supposed to handle
Tempfile objects, but obviously not in this case.

Seems like the way to go is to just use Mongrel and write a HttpHandler
as a standalone server without both Rails and Railsdav, at least for a first
version.

Lots of thanks for pointing me in the right direction.

/Daniel

On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:
> Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe it has something to do with your Railsdav plugin?
>
>
> I just tested an 80 MB upload with my Apache 2.2.6 reverse proxy and Mongrel 1.0.1 cluster and it does save the object first to /tmp on disk and not in memory. The memory used by the mongrel process before I started uploading was 50 MB. While uploading it peaked at 57 MB but was mostly stable around 50-52 MB. And after the upload was done it was back again at 50 MB.
>
>
> Maybe your $TMPDIR is not writable for the user you are running mongrel under?
>
>
> Just before the upload was finished:
> $ ls -hl /tmp/
> total 160288
> -rw-------  1 mongrel  wheel  78.2M Oct 15 16:31 mongrel.25.0
>
>
>
>
>
> I would start looking in the Railsdav plugin you are using, seems like they changed the default of saving to disk, to memory.
>
>
>
> Cheers.
>
> ----- Original Message ----
> From: Daniel Brahneborg <basic70 at gmail.com>
> To: mongrel-users at rubyforge.org
> Sent: Monday, October 15, 2007 4:10:06 PM
> Subject: Re: [Mongrel] POST with huge HTTP body
>
>
> Thanks for the replies.
>
> The fact that Mongrel shouldn't be the first one to get the POST/PUT
>  data
> was a good point that I hadn't thought of. However, even when talking
> directly to Mongrel, the memory consumption of the application
>  increases
> until it crashes if too much data is sent. In my case, I was sending a
>  500MB
> file from a Webdav client. At some point all of it was loaded into RAM,
>  which
> doesn't work.
>
> Are there any more productive alternatives than writing the entire
>  thing
> from scratch in C? Please? :)
>
> /Daniel
>
> On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:
> > A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel
>  does save uploads to a tempfile on disk - if the upload is bigger than
>  about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR,
>  /tmp on most systems).
> >
> >
> > The request is handed over to Rails after it's fully received by
>  Mongrel.
> >
> >
> > I'm not sure if this "saving to disk" works the same with chunked
>  uploads (uploads without a Content-Length header).
> >
> >
> >
> > Maybe my plugin can help you:
>  http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is
>  not very elegant).
> >
> >
> > Gl,
> >
> >
> > Tim
> >
> > ----- Original Message ----
> > From: Daniel Brahneborg <basic70 at gmail.com>
> > To: mongrel-users at rubyforge.org
> > Sent: Monday, October 15, 2007 2:00:20 PM
> > Subject: [Mongrel] POST with huge HTTP body
> >
> >
> > Hi,
> >
> > I found the following page describing how to stream data from the
> > server to the client:
> >
> > http://api.rubyonrails.com/classes/ActionController/Streaming.html
> >
> > I want to do the opposite, streaming data from the client to the
> >  server,
> > letting the controller saving the data while it's being received.
> > In particular, I want to upload large files to the RailsDav
> > (http://www.liverail.net/railsdav) plugin, without having to store
> > the entire file in memory during the operation.
> >
> > This seems to be a Mongrel problem, so I started looking at this
> >  plugin:
> >
> > http://mongrel.rubyforge.org/docs/upload_progress.html
> >
> > How do I get Mongrel to store the incoming POST  data in a tempfile,
> > passing a handler to that file as the raw_post field in the Rails
> >  request?
> > Is it even possible?
> > How is the "request" parameter to the process() method related to the
> > "request" object in the Rails controllers?
> >
> > /Daniel
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
> >
> >
> >
> >
> >
>  ____________________________________________________________________________________
> > Shape Yahoo! in your own image.  Join our Network Research Panel
>  today!   http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
> >
> >
> > _______________________________________________
> > Mongrel-users mailing list
> > Mongrel-users at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/mongrel-users
> >
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
>
>
>
>
> ____________________________________________________________________________________
> Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more.
> http://mobile.yahoo.com/go?refer=1GNXIC
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>


More information about the Mongrel-users mailing list