murdering high-memory workers and auto-scaling

Clifton King cliftonk at
Fri Mar 2 00:12:20 UTC 2012

We use the following at the bottom of a God config. I believe it's in an example

Bloated worker gets sent a QUIT and it will finish processing the request and
exits gracefully. We never really use TTOU and TTIN since the # of workers used
is basically determined by the ram in the machine in question.

# unicorn workers
unicorn_worker_memory_limit = 220_000 do
  loop do
      workers = `ps -e -www -o pid,rss,command | grep '[u]nicorn_rails worker'`
      workers.split("\n").each do |line|
        parts = line.split(' ')
        if parts[1].to_i > unicorn_worker_memory_limit
          # tell the worker to die after it finishes serving its request
          ::Process.kill('QUIT', parts[0].to_i)
    rescue Object
      # don't die ever once we've tested this

    sleep 30



On Mar 1, 2012, at 5:52 PM, Ben Somers wrote:

> Two ideas, one more controversial than the other.
> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since
> the auto-scaling meant that bloated workers got killed periodically.
> In a perfect world, we'd find and patch all of the leaks, but in the
> meantime (and as a safety net) I'd like to get the bloated workers
> auto-killed. It looks like it'd be simple to add in a bloated-worker
> check at the same point when we check for timeout violations, and it
> could be hidden behind a config setting. Alternately, I could write
> this in a separate script.
> Pros: might be a useful built-in feature, looks easy to implement the killing
> Cons: Getting the memory usage might actually be surprisingly
> difficult. Comparing to passenger's memory management code, where they
> actually use platform-specific system calls, and we might get a
> sizeable quantity of code that we don't want dirtying up the unicorn
> internals. Also, some methods of checking appear to have performance
> risks.
> Second: in my use case, I have webservers running as VMs, sharing a
> physical box with backend utility servers. The util servers run lots
> of very CPU- and memory-hungry jobs, mostly at night; the webservers
> handle requests, mostly in the daytime. Currently, most of these
> webservers are running passenger, which is very polite about not using
> more resources than it needs to handle requests. Unicorn, by contrast
> (and by design) is very resource-greedy, what with the "scale to what
> you can theoretically handle" strategy. If I spin down my number of
> unicorn workers when they're not needed, I free up resources for my
> util servers, which is important. TTOU and TTIN signals give me a
> (very nice) means to write an auto-scaling module outside of unicorn,
> but it might be nice to have it included as an optional component. (I
> expect this will get voted down, as I expect the dev team is not
> interested in it).
> Happy to work on implementing these myself, just wanted to poll to see
> if it'd be worth developing them as part of unicorn proper rather than
> standalone scripts.
> -ben
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at
> Do not quote signatures (like this one) or top post when replying

More information about the mongrel-unicorn mailing list