[Backgroundrb-devel] Deploying to a staging server using Capistrano: how to start up BackgrounDRb?

Bobby Santiago bobby.santiago at gmail.com
Mon May 5 20:27:42 EDT 2008


Hey, Stevie,

When I changed the port number in backgroundrb.yml to match the  
mongrel ports in my conf file, the connection error went away. I  
looked into /var/www/apps/my_app/shared/pids, and  
backgroundrb_9201.pid is there. In the /var/www/apps/my_app/shared/log  
dir, I have
backgroundrb_9201.log
backgroundrb_9201_debug.log
backgroundrb_server_9201.log

When I did  ps aux | grep thumbnail_generator_worker, I got:
deploy   12279  0.0  0.1   1816   496 pts/0    R+   20:07   0:00 grep  
thumbnail_generator_worker
So I guess it's not running.

After setting :log: foreground in backgroundrb.yml and deploying, I  
tailed the backgroundrb_9201.log, backgroundrb_9201_debug.log, and  
backgroundrb_server_9201.log files. No output there.

In development, the backgroundrb_1106.log file has the correct output:
BillingWorker#create called...
billing_worker started
Schedules for worker loaded
ThumbnailGeneratorWorker#create called...
thumbnail_generator_worker started
Schedules for worker loaded
generate 37

There is no feedback from the ask_status method, I get nil when I  
inspect the variable in staging.log.

I'm running out of ideas.

Bobby

On May 6, 2008, at 12:14 AM, Stevie Clifton wrote:

> Hey Bobby,
>
> I think you're dealing with two separate issues here.  The first error
> you're seeing with capistrano is b/c capistrano doesn't auto-link the
> /RAILS_ROOT/tmp/pids folder, so backgroundrb is unable to create a pid
> file there.  I do something like the following (untested):
>
> after "deploy:update_code", "deploy:symlink_pids"
>
> namespace :deploy
>  desc "Symlink tmp/pids folder"
>  task :symlink_pids do
>    run "mkdir -p #{release_path}/tmp"
>    run "ln -nfs #{shared_path}/pids #{release_path}/tmp/pids"
>  end
> end
>
> For the second issue, it's a little difficult knowing why rails can't
> see your worker without seeing some bdrb log output.  Have you
> verified that the worker is actually running? (ps aux | grep
> <worker_name>).  Also, have you checked your backgroundrb_server.log
> to see if it's dying on instantiation because of some other
> dependencies?  I'd recommend launching backgroundrb manually on
> staging and setting :log: foreground in backgroundrb.yml so you can
> see what's going on with your worker.
>
> stevie
>
>
> On Sun, May 4, 2008 at 11:02 PM, Bobby Santiago
> <bobby.santiago at gmail.com> wrote:
>> Hi.
>>
>> I am using BackgrounDRb to process thumbnails and upload to S3 -  
>> things are
>> hunky-dory in development (thumbs are generated, these are uploaded  
>> to S3,
>> the metadata is saved to trhe DB, and I get a nice status page  
>> updated by
>> periodic calls via ask_status), but when I tried to deploy to our  
>> staging
>> server and stop/start BackgrounDRb via Capistrano, things blew up -  
>> well,
>> not exactly, but when the app tried to hand off the thumbnail  
>> generation, I
>> got a "Could not connect to the BackgrounDRb server" error.
>>
>> Initially, when I didn't start the BDRb server manually, I got:
>>
>> * executing `after_update_code'
>> * executing `set_env_staging'
>> ENV['RAILS_ENV'] = staging
>> * executing `restart_backgroundrb'
>> * executing `stop_backgroundrb'
>> * executing "cd /var/www/apps/my_app/releases/20080501133415 &&
>> ./script/backgroundrb -e staging stop"
>>  servers: ["staging.my_app.com"]
>>  [staging.my_app.com] executing command
>> ** [out :: staging.my_app.com] ./script/backgroundrb:46:in  
>> `initialize'
>> ** [out :: staging.my_app.com] :
>> ** [out :: staging.my_app.com] No such file or directory -
>> /var/www/apps/my_app/releases/20080501133415/tmp/pids/ 
>> backgroundrb_22222.pid
>> ** [out :: staging.my_app.com] (
>> ** [out :: staging.my_app.com] Errno::ENOENT
>> ** [out :: staging.my_app.com] )
>> ** [out :: staging.my_app.com] from ./script/backgroundrb:46:in  
>> `open'
>> ** [out :: staging.my_app.com] from ./script/backgroundrb:46
>>  command finished
>> *** [deploy:update_code] rolling back
>> * executing "rm -rf /var/www/apps/my_app/releases/20080501133415;  
>> true"
>>  servers: ["staging.my_app.com"]
>>  [staging.my_app.com] executing command
>>  command finished
>>
>> So I tried manually starting backgroundrb on the staging server,,  
>> and the
>> deployment went through successfully. But when I tried generating  
>> thumbs
>> with a background worker, I got the "Could not connect to the  
>> BackgrounDRb
>> server" error.
>>
>> I check out from GitHub, and use Capistrano-ext for multistage  
>> deployments:
>>
>> cap staging deploy
>>
>> Before I deploy, I SSH into the VPS and do
>> script/backgroundrb -e staging start
>>
>> in my config/deploy/staging.rb file:
>> ----------------
>> set :rails_env, 'staging'
>> ...
>>
>> task :after_update_code do
>>  set_env_staging
>>  copy_mongrel_cluster_config
>>  restart_backgroundrb
>> end
>>
>> desc "Sets the environment variable RAILS_ENV='staging'."
>> task :set_env_staging do
>>  ENV['RAILS_ENV'] = 'staging'
>>  puts "ENV['RAILS_ENV'] = #{ENV['RAILS_ENV']}"
>> end
>>
>> desc "Copying the right mongrel cluster config for the current stage
>> environment."
>> task :copy_mongrel_cluster_config do
>>  run "cp -f #{release_path}/config/deploy/staging/mongrel_cluster.yml
>> #{release_path}/config/mongrel_cluster.yml"
>> end
>>
>> desc "Start the backgroundrb server"
>>  task :start_backgroundrb , :roles => :app do
>>   run "cd #{current_path} && nohup ./script/backgroundrb start -- -r
>> staging > #{current_path}/log/backgroundrb-cap.log 2>&1"
>>  end
>>
>>
>> desc "Stop the backgroundrb server"
>> task :stop_backgroundrb, :roles => :app do
>>  run "cd #{release_path} && ./script/backgroundrb -e staging stop"
>> end
>>
>> desc "Restart the backgroundrb server"
>> task :restart_backgroundrb, :roles => :app do
>>  stop_backgroundrb
>>  start_backgroundrb
>> end
>>
>> -------
>> Here's the exception_notifier backtrace:
>>
>>  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:81:in
>> `dump_object'
>>  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:98:in
>> `new_worker'
>>  [RAILS_ROOT]/app/models/uploader.rb:54:in `start_thumbnail_worker'
>>  [RAILS_ROOT]/app/controllers/photos_controller.rb:49:in `create'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> mime_responds.rb:106:in
>> `call'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> mime_responds.rb:106:in
>> `respond_to'
>>  [RAILS_ROOT]/app/controllers/photos_controller.rb:47:in `create'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:1158:in
>> `send'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:1158:in
>> `perform_action_without_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:697:in
>> `call_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:689:in
>> `perform_action_without_benchmark'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> benchmarking.rb:68:in
>> `perform_action_without_rescue'
>>  /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> benchmarking.rb:68:in
>> `perform_action_without_rescue'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> rescue.rb:199:in
>> `perform_action_without_caching'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> caching.rb:678:in
>> `perform_action'
>>
>> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> connection_adapters/abstract/query_cache.rb:33:in
>> `cache'
>>
>> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> query_cache.rb:8:in
>> `cache'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> caching.rb:677:in
>> `perform_action'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:524:in
>> `send'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:524:in
>> `process_without_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:685:in
>> `process_without_session_management_support'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> session_management.rb:123:in
>> `sass_old_process'
>>  /usr/lib/ruby/gems/1.8/gems/haml-1.8.1/lib/sass/plugin/rails.rb: 
>> 15:in
>> `process'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:388:in
>> `process'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:171:in
>> `handle_request'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:115:in
>> `dispatch'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:126:in
>> `dispatch_cgi'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:9:in
>> `dispatch'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in
>> `process'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
>> `synchronize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
>> `process'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in
>> `process_client'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in  
>> `each'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in
>> `process_client'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in
>> `initialize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `new'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in
>> `initialize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in  
>> `new'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in  
>> `run'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:282:in
>> `run'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:281:in
>> `each'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:281:in
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb: 
>> 212:in
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
>>  /usr/bin/mongrel_rails:19:in `load'
>>  /usr/bin/mongrel_rails:19
>>
>> My backgroundrb.yml file:
>> :backgroundrb:
>>  :ip: 0.0.0.0
>>
>> :development:
>>  :backgroundrb:
>>    :port: 11006     # use port 11006
>>    :log: foreground # foreground mode,print log messages on console
>>    :debug_log: true
>>
>> :staging:
>>  :backgroundrb:
>>    :port: 22222      # use port 22222
>>    :lazy_load: true  # do not load models eagerly
>>    :debug_log: false # disable log workers and other logging
>>
>> :production:
>>  :backgroundrb:
>>    :port: 33333      # use port 33333
>>    :lazy_load: true  # do not load models eagerly
>>    :debug_log: false # disable log workers and other logging
>>
>> My worker:
>> class ThumbnailGeneratorWorker < BackgrounDRb::MetaWorker
>>  set_worker_name :thumbnail_generator_worker
>>  set_no_auto_load(true)
>>
>>  def create(args = nil)
>>    register_status(:percent_complete => 0)
>>    args.each_with_index do |uploader_id, index|
>>      @uploader = Uploader.find(uploader_id)
>>      @uploader.generate_thumbnails
>>      percent_complete = ((index + 1) * 100) / args.length
>>      logger.info "Thumbnail generation is #{percent_complete}%  
>> complete..."
>>      register_status(:percent_complete => percent_complete)
>>    end
>>    exit
>>
>>  end
>> end
>>
>> Any hints/help/ideas would be much appreciated.
>>
>> Bobby
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>



More information about the Backgroundrb-devel mailing list