[Mongrel] Uploading Large (100mb+) files

Ezra Zygmuntowicz ezmobius at gmail.com
Tue Nov 28 16:53:39 EST 2006

	There is an example merb controller for use with the  
mongrel_upload_progress if  anyone is interested in using a small  
merb app for their upload app server. We have a number of customers  
doing this and some of them are handling a ton of file uploads all  
the time. Merb can also be a nice secure download manager. The Files  
controller in the example app in the merb svn shows how easy it is to  


	You can do a svn checkout of the merb svn here:

$ svn co http://svn.devjavu.com/merb

And build it and play with the example app like this:

$ cd merb
$ sudo rake install
$ cd examples/sample_app
$ merb -p 4000

	Then go to http://localhost:4000/files/start in your browser to see  
an example of the upload progress. Sice it is running locally you  
shoudl try to upload a file at least 100Mb or bigger or else it will  
happen too fast and you won't see the progress bar.

	The nice thing about the way merb works is that it doesn't use  
cgi.rb for mime parsing and it is written in a more thread safe way  
then actionpack. So multiple concurrent requests can be processed and  
have their mime parsing done all at the same time. Merb only does a  
mutex around the smallest possible place where you might be calling  
ActiveRecord code. With the rails dispatcher, the mutex surrounds the  
entire routing code and mime parsing and your controler code. Merb  
only locks around the action in your controller code. It does route  
recognition and mime parsing outside the mutex in a multi threaded  
way. This allows us to use ActiveRecord in single threaded mode which  
is better then AR's multi threaded mode for performance. So we use  
the smallest sized lock we can in order to still support this, but we  
do all the heavyweight routing and mime parsing concurrently.

	I promised Zed I would do a write up about using a merb app for a  
dedicated upload server in conjunction with a rails app. I will try  
to get something written this week and posted. Until then I am open  
to and questions about merb and you can file tickets for support or  
feature requests here:


	Zed and I hope to come up with a solution so that we can use the  
upload progress without reparsing the mime twice per file. Right now  
mongrel streams the upload to a tmp file and increments the progress  
bar all the way to 100% before it hands control to your rails or merb  
controller to deal with the file. But then rails or merb has to parse  
the mime boundaries of the entire file again. So you may notice with  
big files that right when the progress bar makes it to 95-100% it  
will pause while rails or meb parses the mime again into another  
tmpfile. This is obviously not an optimal way to do this. Zed has  
started an awesome fast C mime carver and I will be working with him  
to integrate this so uploads can be parsed once with the fast C mimer  
and then just handed off to the next handler with no need to reparse.


-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Mongrel-users mailing list