Unicorn is killing our rainbows workers
samuel.kadolph at shopify.com
Tue Jul 31 14:09:08 UTC 2012
On Fri, Jul 27, 2012 at 4:40 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Samuel Kadolph <samuel.kadolph at shopify.com> wrote:
>> On Thu, Jul 26, 2012 at 8:11 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> >> Our ops guys have been busy so I don't have the output from lsof but
>> >> it didn't look like it was spawning any extra threads or opening any
>> >> unexplainable connections. But I think we should have been checking
>> >> the worker processes and not the master, right?
>> > Definitely check the master, too. It's the master that seems to
>> > believe it's suspended, so that makes me believe something is wrong
>> > with the master (and this is likely due to preload_app).
>> >> Haven't tried disabling preload_app yet but we have tried
>> I've got the output of lsof and ls at https://gist.github.com/3190171.
> Thanks, that's the output for the master? I don't see anything
> obviously wrong.
> I seem to recall the Ruby library responsible for the following log file
> also spawns its own background thread, but your "ls" only shows 2 tasks
> (instead of 3):
>> ruby 26564 root 9w REG 202,1 51221 529742 APP_PATH/shared/log/newrelic_agent.log
>> $ ls /proc/26564/task/
>> 26564 27052
> (While the Ruby code for the module responsible for that log file is
> technically "open", it's not Free, so I'm not comfortable looking at
> that code).
So 2 updates: yes that lsof output is from the master process and
using preload_app false solves the issue. No more killings and the
suspend/hibernation messages stopped as well. We lost newrelic data so
we're going to try putting preload_app back to true and removing the
More information about the rainbows-talk