[Mongrel] POST with huge HTTP body

Tim Kuijsten kuijsten at yahoo.com
Mon Oct 15 10:37:57 EDT 2007


Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe it has something to do with your Railsdav plugin?


I just tested an 80 MB upload with my Apache 2.2.6 reverse proxy and Mongrel 1.0.1 cluster and it does save the object first to /tmp on disk and not in memory. The memory used by the mongrel process before I started uploading was 50 MB. While uploading it peaked at 57 MB but was mostly stable around 50-52 MB. And after the upload was done it was back again at 50 MB.


Maybe your $TMPDIR is not writable for the user you are running mongrel under?


Just before the upload was finished:
$ ls -hl /tmp/
total 160288
-rw-------  1 mongrel  wheel  78.2M Oct 15 16:31 mongrel.25.0





I would start looking in the Railsdav plugin you are using, seems like they changed the default of saving to disk, to memory.



Cheers.

----- Original Message ----
From: Daniel Brahneborg <basic70 at gmail.com>
To: mongrel-users at rubyforge.org
Sent: Monday, October 15, 2007 4:10:06 PM
Subject: Re: [Mongrel] POST with huge HTTP body


Thanks for the replies.

The fact that Mongrel shouldn't be the first one to get the POST/PUT
 data
was a good point that I hadn't thought of. However, even when talking
directly to Mongrel, the memory consumption of the application
 increases
until it crashes if too much data is sent. In my case, I was sending a
 500MB
file from a Webdav client. At some point all of it was loaded into RAM,
 which
doesn't work.

Are there any more productive alternatives than writing the entire
 thing
from scratch in C? Please? :)

/Daniel

On 10/15/07, Tim Kuijsten <kuijsten at yahoo.com> wrote:
> A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel
 does save uploads to a tempfile on disk - if the upload is bigger than
 about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR,
 /tmp on most systems).
>
>
> The request is handed over to Rails after it's fully received by
 Mongrel.
>
>
> I'm not sure if this "saving to disk" works the same with chunked
 uploads (uploads without a Content-Length header).
>
>
>
> Maybe my plugin can help you:
 http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is
 not very elegant).
>
>
> Gl,
>
>
> Tim
>
> ----- Original Message ----
> From: Daniel Brahneborg <basic70 at gmail.com>
> To: mongrel-users at rubyforge.org
> Sent: Monday, October 15, 2007 2:00:20 PM
> Subject: [Mongrel] POST with huge HTTP body
>
>
> Hi,
>
> I found the following page describing how to stream data from the
> server to the client:
>
> http://api.rubyonrails.com/classes/ActionController/Streaming.html
>
> I want to do the opposite, streaming data from the client to the
>  server,
> letting the controller saving the data while it's being received.
> In particular, I want to upload large files to the RailsDav
> (http://www.liverail.net/railsdav) plugin, without having to store
> the entire file in memory during the operation.
>
> This seems to be a Mongrel problem, so I started looking at this
>  plugin:
>
> http://mongrel.rubyforge.org/docs/upload_progress.html
>
> How do I get Mongrel to store the incoming POST  data in a tempfile,
> passing a handler to that file as the raw_post field in the Rails
>  request?
> Is it even possible?
> How is the "request" parameter to the process() method related to the
> "request" object in the Rails controllers?
>
> /Daniel
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
>
>
>
>
>      
 ____________________________________________________________________________________
> Shape Yahoo! in your own image.  Join our Network Research Panel
 today!   http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>
_______________________________________________
Mongrel-users mailing list
Mongrel-users at rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users





       
____________________________________________________________________________________
Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. 
http://mobile.yahoo.com/go?refer=1GNXIC


More information about the Mongrel-users mailing list