[Backgroundrb-devel] struggling with stability

Jodi Showers jodi at nnovation.ca
Sat Jun 30 13:03:56 EDT 2007


Greetings Tieg,

On 27-Jun-07, at 11:48 AM, Tieg Zaharia wrote:

> Re: Jodi's post,
>
> I'm having the same problem just today (don't think I've had this  
> before), and only when I feed it a custom background.yml file.  
> Here's the error:
>
> 20070627-11:36:28 (1804) Starting BackgrounDRb Server
> 20070627-11:36:28 (1804) config: RAILS_ROOT/config/backgroundrb.yml
> 20070627-11:36:28 (1804) rails_env: development
> 20070627-11:36:28 (1804) socket_dir: RAILS_ROOT/tmp/backgroundrb.1804
> 20070627-11:36:28 (1804) pool_size: 5
> 20070627-11:36:28 (1804) host: localhost
> 20070627-11:36:28 (1804) acl: orderdenyallowdenyallallowlocalhost  
> 127.0.0.1
> 20070627-11:36:28 (1804) temp_dir: RAILS_ROOT/tmp
> 20070627-11:36:28 (1804) port: 2000
> 20070627-11:36:28 (1804) worker_dir: RAILS_ROOT/lib/workers
> 20070627-11:36:28 (1804) protocol: drbunix
> 20070627-11:36:28 (1804) uri: drbunix:///RAILS_ROOT/tmp/ 
> backgroundrbunix_localhost_2000
> 20070627-11:36:29 (1806) Starting worker:  
> BackgrounDRb::Worker::WorkerLogger backgroundrb_logger  
> (backgroundrb_logger) ()
> 20070627-11:36:29 (1806) failed to find slave socket - (RuntimeError)
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> slave-1.2.1/lib/slave.rb:435:in `initialize'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/middleman.rb:210:in `new'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/middleman.rb:210:in `new_worker'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/thread_pool.rb:36:in `dispatch'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/thread_pool.rb:22:in `initialize'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/thread_pool.rb:22:in `new'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/thread_pool.rb:22:in `dispatch'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/middleman.rb:199:in `new_worker'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb/middleman.rb:103:in `setup'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb_server.rb:306:in `run'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/application.rb:193:in `call'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/application.rb:193:in `start_proc'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/daemonize.rb:192:in `call_as_daemon'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/application.rb:197:in `start_proc'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/application.rb:233:in `start'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/controller.rb:69:in `run'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons.rb:185:in `run_proc'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/cmdline.rb:105:in `call'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons/cmdline.rb:105:in `catch_exceptions'
> 20070627-11:36:29 (1806) /usr/local/lib/ruby/gems/1.8/gems/ 
> daemons-1.0.6/lib/daemons.rb:184:in `run_proc'
> 20070627-11:36:29 (1806) RAILS_ROOT/vendor/plugins/backgroundrb/ 
> server/lib/backgroundrb_server.rb:301:in `run'
> 20070627-11:36:29 (1806) script/backgroundrb:29
> 20070627-11:36:29 (1806) Loading Worker Class File: RAILS_ROOT/lib/ 
> workers/my_worker.rb
>
> And I only used a minimal config file to change the tmp directory:
> ---
> :temp_dir: "RAILS_ROOT/tmp"
>
> (the RAILS_ROOT in all of this is just a replacement for the paths  
> to my app)
> I just noticed this problem and haven't even started to dig into it  
> yet, but it looks like feeding it a custom temp dir messes up the  
> slave somehow? I tried replacing the temp_dir via the command line  
> args and it still happens.
>
> environment:
> - OSX 10.4.10
> - ruby 1.8.4 (2005-12-24) [x86_64-linux]
> - ruby 1.8.5 (2006-12-25 patchlevel 12) [i686-darwin8.8.5]
> - rails EDGE
> - slave (1.2.1)
> - daemons (1.0.6)
> - mysql 5.0.27
>

I have resorted to using Monit to keep backgroundrb up. Last week the  
app went 4 days without downtime - woop!

I'm quite convinced it's not an app specific problem - my tests are  
quite complete. And dev, staging and production show the same  
symptoms - unpredictable failures.

The only guess I have so far is that it maybe a resource contention  
issue - as long as I keep. My brb tasks ratchet up the ram usage to  
+80%.

Funny (not so actually) just checked my prod box and cp was at +95%,  
and 3 separate backgroundrb tasks were running (the dangers of using  
monit to heal processes!)

good luck
Jodi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070630/6d791709/attachment.html 


More information about the Backgroundrb-devel mailing list