large uploads

Eric Wong normalperson at
Thu Sep 29 14:47:11 EDT 2011

John Joseph Bachir <j at> wrote:
> My application accepts uploads from users, which can be quite huge in
> some cases. This of course requires setting the unicorn timeout to
> something much higher than 60 seconds, more like 10 minutes.
> Are there any drawbacks to doing this, other than the obvious drawback
> of not killing off long-running requests that are illegitimate?

Are your users remote? (outside of your immediate LAN).

The upload speed (and thus timeout) for unicorn should be based on
the nginx <-> unicorn transfer rate.  Unicorn should never talk to users
directly and users can upload slowly to nginx which buffers requests to the
filesystem, first.

> I've googled quite a bit about this and have found surprisingly little
> -- I guess people who have apps that receive uploads just generally
> don't use unicorn?

I have a LAN-only application that regularly processes uploads several
hundreds of megabytes (via PUT) directly to Unicorn.  The local disk I/O
is often the limiting factor since the parallel requests fill up the
kernel buffers and wait on disk I/O.  The Rack application itself
unfortunately needs to seek/rewind so it must be in the filesystem.

Rainbows! with ThreadSpawn/ThreadPool can process uploads without
buffering them to disk first (but the Rack multipart parser may not).
I often stream several hundred megabytes of data directly to apps on
Rainbows! via PUT requests (curl -T), too.

More information about the mongrel-unicorn mailing list