Maintaining capacity during deploys

seth.cousins at seth.cousins at
Fri Nov 30 04:48:16 UTC 2012

In my experience high loads and contention are a common issue when restarting the unicorn master process. In a previous project we dealt with this by 1) performing some warmup requests in the master before starting to fork workers; 2) replacing old workers slowly by having each new worker send a TTOU to the old master in after_fork and having the new master sleep for a couple of seconds between spawning workers.

It was a couple of years ago so the details are not fresh but iirc before tuning a restart took 5-10 seconds followed by load climbing to 10-20 (on a 4 proc machine) with a 2-5 minute slow recovery of long request times. In particularly pathological cases requests can start timing out which results in workers being killed and new workers needing to warm up on and already overloaded system. After tuning the rolling restart took 30-40 seconds but the load barely budged and the request processing times stayed constant.


On Nov 29, 2012, at 5:24 PM, Eric Wong <normalperson at> wrote:

> Tony Arcieri <tony.arcieri at> wrote:
>> On Thu, Nov 29, 2012 at 3:34 PM, Lawrence Pit <lawrence.pit at> wrote:
>>> Perhaps it's possible to warm up the workers in the unicorn after_fork block?
>> Are people doing this in production (i.e. moving the termination of
>> the old master from before_fork to after_fork)? My worry is that
>> during this warming process you will have 2X the normal number of
>> Unicorn workers active at the same time, which could potentially lead
>> to exhausting of system resources (i.e. RAM)
> I haven't done any terminations in the *_fork hooks for a long time.
> I just let 2x the normal workers run for a bit before sending SIGQUIT.
> That said, I usually have plenty of RAM (and DB connections) to spare.
> Excessive CPU-bound loads are handled very well nowadays.
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at
> Do not quote signatures (like this one) or top post when replying

More information about the mongrel-unicorn mailing list