Unicorn is killing our rainbows workers

Samuel Kadolph samuel.kadolph at shopify.com
Tue Jul 31 14:09:08 UTC 2012


On Fri, Jul 27, 2012 at 4:40 PM, Eric Wong <normalperson at yhbt.net> wrote:
> Samuel Kadolph <samuel.kadolph at shopify.com> wrote:
>> On Thu, Jul 26, 2012 at 8:11 PM, Eric Wong <normalperson at yhbt.net> wrote:
>> >> Our ops guys have been busy so I don't have the output from lsof but
>> >> it didn't look like it was spawning any extra threads or opening any
>> >> unexplainable connections. But I think we should have been checking
>> >> the worker processes and not the master, right?
>> >
>> > Definitely check the master, too.  It's the master that seems to
>> > believe it's suspended, so that makes me believe something is wrong
>> > with the master (and this is likely due to preload_app).
>> >
>> >> Haven't tried disabling preload_app yet but we have tried
>>
>> I've got the output of lsof and ls at https://gist.github.com/3190171.
>
> Thanks, that's the output for the master?  I don't see anything
> obviously wrong.
>
> I seem to recall the Ruby library responsible for the following log file
> also spawns its own background thread, but your "ls" only shows 2 tasks
> (instead of 3):
>
>> ruby    26564 root    9w   REG              202,1     51221   529742 APP_PATH/shared/log/newrelic_agent.log
>
>> $ ls /proc/26564/task/
>> 26564  27052
>
> (While the Ruby code for the module responsible for that log file is
>  technically "open", it's not Free, so I'm not comfortable looking at
>  that code).

So 2 updates: yes that lsof output is from the master process and
using preload_app false solves the issue. No more killings and the
suspend/hibernation messages stopped as well. We lost newrelic data so
we're going to try putting preload_app back to true and removing the
newrelic gem.


More information about the rainbows-talk mailing list