[Backgroundrb-devel] Repeatedly dying with "failed to find slave socket"

Mason Hale masonhale at gmail.com
Wed Feb 7 11:16:48 EST 2007


Hi Jon -

I have see similar errors like this in the past, but I don't have any
in my backgroundrb log now.

I didn't specifically track down and fix this error, but I made some
other changes that may have inadvertantly avoided this problem. Those
changes were:

1.) Avoid using the results feature at all. All status from workers is
stored in the db now.
2.) Call self.delete at the end of the worker doing its work
3.) Simplifying the workers themselves. They now just load arguments
and call other classes with those arguments. (This step also makes
testing easier).

I have a scheduled worker that is long running and runs once per
minute. That worker looks for pending jobs and spawns new workers to
handle those jobs as needed. Those spawner workers take from a few
minutes to several hours to do their work before they finish. We have
up to 10 workers running at a time.

We also have a cron job that checks every minute to see if the
scheduled worker has run in the last 3 minutes. If not, it restarts
the backgroundrb server and sends a notification email. It's been a
little over a week now since the backgroundrb server was
auto-restarted.

Mason

On 2/7/07, Jonathan del Strother <maillist at steelskies.com> wrote:
> I'm struggling to keep backgroundrb v2 running for more than 24
> hours.  It appears to be running fine for a while, then tries to fire
> off a worker (exactly the same worker it's been running for the past
> few hours), and it suddenly dies.  I haven't found any pattern for the
> cause of this.
>
> In backgroundrb_server.log, the final entry is this :
>
> 20070206-10:41:45 (5189) Schedule triggered: #<struct #<Class:
> 0x2afb643b56c0> job=#<Proc:0x00002afb63ed83c8 at .../vendor/plugins/
> backgroundrb/server/lib/backgroundrb/middleman.rb:355>,
> trigger=#<BackgrounDRb::Trigger:0x2afb6546ccc0 @repeat_interval=600,
> @end_time=nil, @start_time=Tue Feb 06 03:41:45 -0800 2007>,
> earliest=Tue Feb 06 10:41:45 -0800 2007, last=Tue Feb 06 10:41:45
> -0800 2007>
> 20070206-10:41:45 (5189) failed to find slave socket - (RuntimeError)
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/slave-1.2.0/lib/
> slave.rb:435:in `initialize'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/middleman.rb:210:in `new'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/middleman.rb:210:in `new_worker'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/thread_pool.rb:36:in `dispatch'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/thread_pool.rb:22:in `initialize'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/thread_pool.rb:22:in `new'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/thread_pool.rb:22:in `dispatch'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/middleman.rb:199:in `new_worker'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/middleman.rb:357:in `schedule_worker'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:46:in `call'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:46:in `run'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:40:in `each'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:40:in `run'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:35:in `loop'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:35:in `run'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:95:in `run'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:94:in `initialize'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:94:in `new'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/scheduler.rb:94:in `run'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb/middleman.rb:106:in `setup'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb_server.rb:306:in `run'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/application.rb:186:in `call'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/application.rb:186:in `start_proc'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/daemonize.rb:192:in `call'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/daemonize.rb:192:in `call_as_daemon'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/application.rb:190:in `start_proc'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/application.rb:226:in `start'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/controller.rb:69:in `run'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons.rb:182:in `run_proc'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/cmdline.rb:94:in `call'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons/cmdline.rb:94:in `catch_exceptions'
> 20070206-10:41:45 (5189) /usr/lib64/ruby/gems/1.8/gems/daemons-1.0.4/
> lib/daemons.rb:181:in `run_proc'
> 20070206-10:41:45 (5189) .../vendor/plugins/backgroundrb/server/lib/
> backgroundrb_server.rb:301:in `run'
> 20070206-10:41:45 (5189) .../script/backgroundrb:29
>
>
>
> This happens both on my EngineYard server, and running locally on
> intel OS X 10.4.8.
>
>
> Any suggestions?  Anyone else experiencing this?
>
> Jon
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


More information about the Backgroundrb-devel mailing list