Fwd: Maintaining capacity during deploys
normalperson at yhbt.net
Thu Nov 29 23:32:08 UTC 2012
Tony Arcieri <tony.arcieri at gmail.com> wrote:
> We're using unicornctl restart with the default before/after hook
> behavior, which is to reap old Unicorn workers via SIGQUIT after the
> new one has finished booting.
> Unfortunately, while the new workers are forking and begin processing
> requests, we're still seeing significant spikes in our haproxy request
> queue. It seems as if after we restart, the unwarmed workers get
> swamped by the incoming requests. As far as I can tell, the momentary
> loss of capacity we experience translates fairly quickly into a
> thundering herd.
> We've experimented with rolling restarts at the server level but these
> do not resolve the problem.
So it's one haproxy -> multiple nginx/unicorn servers?
Do you mark the server down or lower the weight in haproxy when
deploying the Ruby app? (Perhaps using monitor-uri in haproxy)
If the server is down to haproxy, but still up, you can send some warmup
requests to the server before enabling the monitor-uri for haproxy.
More information about the mongrel-unicorn