Is a client uploading a file a slow client from unicorn's point of view?

Eric Wong normalperson at
Tue Oct 9 01:58:00 UTC 2012

Jimmy Soho <jimmy.soho at> wrote:
> Hi All,
> I was wondering what would happen when large files were uploaded to
> our system in parallel to endpoints that don't process file uploads.
> In particular I was wondering if we're vulnerable to a simple DoS
> attack.

nginx will protect you by buffering large requests to disk, so slow
requests are taken care of (of course you may still run out of disk

> The setup I tested with was nginx v1.2.4 with upload module (v2.2.0)
> configured only for location /uploads with 2 unicorn (v4.3.1) workers
> with timeout 30 secs, all running on 1 small unix box.
> In a few terminals I started this command 3 times in parallel:
>    $ curl -i -F importer_input=@/Users/admin/largefile.tar.gz
> In a browser I then tried to go a page that would be served by a unicorn worker.
> My expectation was that I would not get to see the web page as all
> unicorn workers would be busy with receiving / saving the upload. As
> discussed in for example this article:
> Or as describes it:
>   "Processing and saving file uploads are typically long running tasks
> and should be done in a background process."

That is true.  It's good to move slow jobs to background processes if
possible if the bottleneck is either:

a) your application processing
b) the storage destination of your app (e.g. cloud storage)

However, if your only bottleneck is client <-> your app, then nginx
will take care of that part for you.

> But I don't see this. The page is served just fine in my setup. The
> requests for the file uploads appear in the nginx access log at the
> same time the curl upload command eventually finishes minutes later
> client side, and then it's handed off to a unicorn/rack worker
> process, which quickly returns a 404 page not found. Response times of
> less than 50ms.
> What am I missing here? I'm starting to wonder what's the use of the
> nginx upload module? My understanding was that it's use was to keep
> unicorn workers available as long as a file upload was in progress,
> but it seems that without that module it does the same thing.

I'm not familiar with the nginx upload module, but stock nginx will
already do full request buffering for you.  It looks like the nginx
upload module[1] is mostly meant for standalone apps written for
nginx, and not when nginx is used as a proxy for Rails app...


> Another question (more an nginx question though I guess): is there a
> way to kill an upload request as early as possible if the request is
> not made against known / accepted URI locations, instead of waiting
> for it to be completely uploaded to our system and/or waiting for it
> to reach the unicorn workers?

I'm not sure if nginx has this functionality, but unicorn lazily buffers
uploads.  So your upload will be fully read by nginx, but unicorn
will only read the uploaded request body if your application wants to
read it.

Unfortunately, I think most application frameworks (e.g. Rails) will
attempt to do all the multipart parsing up front.  To get around this,
you'll probably want some middleware along the following lines (and
placed in front of whichever part of your stack calls

class BadUploadStopper
  def initialize(app)
    @app = app

  def call(env)
    case env["REQUEST_METHOD"]
    when "POST", "PUT"
      case env["PATH_INFO"]
      when "/upload_allowed" # forward to the app
        # bad path, don't waste time with
        [ 403, {}, [ "Go away\n" ] ]
    else  # forward to the app

------------------- ---------------------
use BadUploadStopper

More information about the mongrel-unicorn mailing list