From jasonlee9 at gmail.com  Thu May  1 10:29:50 2008
From: jasonlee9 at gmail.com (Jason Lee)
Date: Thu, 1 May 2008 07:29:50 -0700
Subject: [Backgroundrb-devel] strange deployment issue
In-Reply-To: <972417190804301422s4a84b3a8m650ba880377a07e8@mail.gmail.com>
References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com>
	<109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws>
	<972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com>
	<a426f29f0804300815j34cbe11dm854ceb6ea448d3dd@mail.gmail.com>
	<972417190804300949s527c5fadlb7dc3d043de23bcf@mail.gmail.com>
	<AF45A0A1-E974-4CAD-AC4B-C34972649032@thewilliams.ws>
	<972417190804301422s4a84b3a8m650ba880377a07e8@mail.gmail.com>
Message-ID: <972417190805010729j6de1f405g419b0c2e8adde0b4@mail.gmail.com>

Solved.

So I basically followed Adam's advice and have my deploy copying a
different backgroundrb.yml file out based on the deploy.

One thing that was a little puzzling was the whole worker thread
process and how it picked up the environment. I noticed a few things,
but I'm not sure if I'm totally correct about them as I haven't really
delved into the codebase.

1. Seems each worker process/run picks up the config. At first glance
I thought "hmm, why re-load the config every time?". I'll probably be
running multiple workers - maybe half a dozen - doing various things
and the config reload seems like an unnecessary hit. *However*, I
didn't write BRb and I'm sure there's obviously a reason why this is
happening and I just don't know it yet.

2. The fact that the config is picked up on each run, overrides any
env I pass in from the command line or from any automated
rake/execution task. I put debug in various places, kicked off the
server with the rails env set and saw that the first run picked up the
env. Every run after, defaulted to development.

So anyway, this isn't a critique, just observations. I'll be putting
my system into a real production env in a few weeks and I'm glad BRb
exists at all - keeps me from having to write yet another
component/service. :)

Thanks for the replies, it's been very helpful!

- jason

On Wed, Apr 30, 2008 at 2:22 PM, Jason Lee <jasonlee9 at gmail.com> wrote:
> Adam,
>
>  Thanks for the reply. I'll have to see if I can play around with the
>  yml file. I definitely cannot use the scheduler definition as it was
>  very unreliable for the tasks I needed to launch. If I can just get
>  the environments to come up without any of the other config, I'll be
>  good.
>
>  I'll post back my solution when I find it.
>
>  Thanks again,
>
>  - jason
>
>
>
>  On Wed, Apr 30, 2008 at 10:35 AM, Adam Williams <adam at thewilliams.ws> wrote:
>  > So, there is some serious confusion about this in the code. I now know of
>  > two other folks (one you, Jason) who struggled with this besides me.
>  >
>  >  I solved it by having a shared/config/backgroundrb.yml on the server, which
>  > we link into config of the app on deploy (capistrano). It looks like this:
>  >
>  >  :backgroundrb:
>  >   :port: 11006
>  >
>  >   :ip: 0.0.0.0
>  >   :environment: staging
>  >   :lazy_load: true
>  >   :debug_log: true
>  >
>  >  :schedules:
>  >   :notification:
>  >     :deliver_event_reminders:
>  >       :trigger_args: 0 */3 * * * * *
>  >
>  >  As you can see, we have the environment in the config. The code in
>  > meta_worker obviously disrespects -e. Now, you ALSO need to pass -e staging,
>  > so that when Rails boots, it has the correct environment. Here is our init.d
>  > line:
>  >
>  >   /var/www/apps/railsapplication/current/script/backgroundrb "$1" -e staging
>  > >> /var/log/backgroundrb
>  >
>  >  I may patch this at some point, but it's already eaten a bit too much time
>  > (not you, but solving it originally).
>  >
>  >    aiwilliams
>  >
>  >
>  >
>  >  On Apr 30, 2008, at 12:49 PM, Jason Lee wrote:
>  >
>  >
>  > >
>  > >
>  > >
>  > > So I tried this and my debug still says the env is development..
>  > >
>  > > Guess I'll have to start hacking to see why this is..
>  > >
>  > > - jason
>  > >
>  > > On Wed, Apr 30, 2008 at 8:15 AM, Stevie Clifton <stevie at slowbicycle.com>
>  > wrote:
>  > >
>  > > > Hey Jason,
>  > > >
>  > > > To get the environment passed in correctly, I use:
>  > > > RAILS_ENV=staging ./script/backgroundrb start
>  > > >
>  > > > And just to make sure, you're aren't actually calling your staging
>  > > > environment "stage" are you?  Or if you are, you have an environment
>  > > > intentionally named that, right?  Just had to ask :).
>  > > >
>  > > > stevie
>  > > >
>  > >
>  > > _______________________________________________
>  > > Backgroundrb-devel mailing list
>  > > Backgroundrb-devel at rubyforge.org
>  > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>  > >
>  >
>  >
>

From joshea at nooked.com  Fri May  2 14:23:11 2008
From: joshea at nooked.com (John O'Shea)
Date: Fri, 2 May 2008 19:23:11 +0100
Subject: [Backgroundrb-devel] Storing worker driven rails logging in worker
	log file?
Message-ID: <A7541678-747B-4B4C-A09B-E75AA75EC6FF@nooked.com>

Hi all,
	We have a few workers that kick off at scheduled intervals or to do  
some user driven long running tasks asynchronously.  At the moment  
when the workers start working with AR model objects, all of the Rails  
logging ends up in the default rails environment log file  
(develolopment/prodiction.log).
	Does anyone have any tips for getting rails to append to a separate  
worker log file?  We'd like to keep the rails logging in a separate  
file so that we can easily find it later - mixed into the main rails  
log it can be difficult to figure out which log entries are as a  
result of controller actions doing their work and which are being  
driven by workers.

Thanks for any tips,

John.


From kapelner at gmail.com  Fri May  2 22:20:16 2008
From: kapelner at gmail.com (Adam Kapelner)
Date: Fri, 2 May 2008 19:20:16 -0700
Subject: [Backgroundrb-devel] backgroundrb / actionmailer / sendmail
Message-ID: <b2baff6b0805021920q5936b215td4f5b30d707552d6@mail.gmail.com>

Hello,

I'm trying to send emails from a backgroundrb worker and the email doesn't
send. I get no log messages in the sendmail log or the system log, nothing
in the backgroundrb logs at all, nothing in the production log.

Below is my code for each component

worker:

class QuizReviewerWorker < BackgrounDRb::MetaWorker
  set_worker_name :quiz_review_sender

  def create(args = nil)
  end

  def send_review_emails
    mail = Notifier.create_test_email
    status = Notifier.deliver(mail)
    logger.info 'test mail:' + mail.body
  end
end

mailer:

class Notifier < ActionMailer::Base

  def test_email
    recipients 'kapelner at gmail.com'
    subject 'test email'
    body 'body of email'
    content_type "text/plain"
  end
end

backgroundrb configuration:

---
:backgroundrb:
  :ip: 0.0.0.0
  :port: 11006
  :environment: production
  :debug_log: true
:schedules:
  :quiz_review_sender:
    :send_review_emails:
      :trigger_args:
        :start: <%= Time.now + 30.seconds %>
        :end: <%= Time.now + 10.years %>
        :repeat_interval: <%= 20.seconds %>

environment.rb configuration:

ActionMailer::Base.delivery_method = :sendmail
ActionMailer::Base.raise_delivery_errors = true
ActionMailer::Base.default_content_type = 'text/html'


Now, when doing the request from rails itself, no problem, I get the email
instantly. Here's the controller code:

class MiscellanyController < ApplicationController
  def test_email
    mail = Notifier.create_test_email
    Notifier.deliver(mail)
    render :text => mail.body
  end
end


What could be the issue? I'm using Ruby 1.8.6, Rails 1.2.5, on Fedora Core
6, Backgroundrb (last updated 2008-02-28)

thanks so much,
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080502/262655b2/attachment-0001.html>

From jasonlee9 at gmail.com  Sat May  3 11:42:32 2008
From: jasonlee9 at gmail.com (Jason Lee)
Date: Sat, 3 May 2008 08:42:32 -0700
Subject: [Backgroundrb-devel] backgroundrb / actionmailer / sendmail
In-Reply-To: <b2baff6b0805021920q5936b215td4f5b30d707552d6@mail.gmail.com>
References: <b2baff6b0805021920q5936b215td4f5b30d707552d6@mail.gmail.com>
Message-ID: <972417190805030842h624ce759o543d5eca9a561933@mail.gmail.com>

Adam,

Not sure if it's a Rails 1.2.5 issue (I'm on 2.0.2). I've used the
scheduler before and found it not as reliable as Unix cron, so I moved
the scheduling code into the worker itself. I have BRb doing
background scheduled email tasks and it works quite well. Here's what
mine looks like:

class EmailWorker < BackgrounDRb::MetaWorker

  set_worker_name :Email_worker

  def create(args = nil)
    # time argument is in seconds
    add_periodic_timer(10) { process_orders }
  end

  def process
     # do some work
  end

end


My config looks like so (for development):

---
:backgroundrb:
    :ip: 0.0.0.0
    :port: 11006
    :environment: development


I removed all my scheduler code out of the config and it works well,
so maybe that will help.

- jason


On Fri, May 2, 2008 at 7:20 PM, Adam Kapelner <kapelner at gmail.com> wrote:
> Hello,
>
> I'm trying to send emails from a backgroundrb worker and the email doesn't
> send. I get no log messages in the sendmail log or the system log, nothing
> in the backgroundrb logs at all, nothing in the production log.
>
> Below is my code for each component
>
> worker:
>
> class QuizReviewerWorker < BackgrounDRb::MetaWorker
>    set_worker_name :quiz_review_sender
>
>    def create(args = nil)
>   end
>
>   def send_review_emails
>     mail = Notifier.create_test_email
>      status = Notifier.deliver(mail)
>     logger.info 'test mail:' + mail.body
>    end
> end
>
> mailer:
>
>  class Notifier < ActionMailer::Base
>
>   def test_email
>      recipients 'kapelner at gmail.com'
>     subject 'test email'
>      body 'body of email'
>     content_type "text/plain"
>    end
> end
>
> backgroundrb configuration:
>
>  ---
> :backgroundrb:
>    :ip: 0.0.0.0
>   :port: 11006
>    :environment: production
>   :debug_log: true
>  :schedules:
>   :quiz_review_sender:
>      :send_review_emails:
>       :trigger_args:
>          :start: <%= Time.now + 30.seconds %>
>         :end: <%= Time.now + 10.years %>
>          :repeat_interval: <%= 20.seconds %>
>
> environment.rb configuration:
>
> ActionMailer::Base.delivery_method = :sendmail
>  ActionMailer::Base.raise_delivery_errors = true
> ActionMailer::Base.default_content_type = 'text/html'
>
>
> Now, when doing the request from rails itself, no problem, I get the email
> instantly. Here's the controller code:
>
> class MiscellanyController < ApplicationController
>    def test_email
>     mail = Notifier.create_test_email
>      Notifier.deliver(mail)
>     render :text => mail.body
>    end
> end
>
>
> What could be the issue? I'm using Ruby 1.8.6, Rails 1.2.5, on Fedora Core
> 6, Backgroundrb (last updated 2008-02-28)
>
> thanks so much,
> Adam
>
>
>
>
>
>
> _______________________________________________
>  Backgroundrb-devel mailing list
>  Backgroundrb-devel at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From jasonlee9 at gmail.com  Sat May  3 11:43:58 2008
From: jasonlee9 at gmail.com (Jason Lee)
Date: Sat, 3 May 2008 08:43:58 -0700
Subject: [Backgroundrb-devel] backgroundrb / actionmailer / sendmail
In-Reply-To: <972417190805030842h624ce759o543d5eca9a561933@mail.gmail.com>
References: <b2baff6b0805021920q5936b215td4f5b30d707552d6@mail.gmail.com>
	<972417190805030842h624ce759o543d5eca9a561933@mail.gmail.com>
Message-ID: <972417190805030843n27e59af9te1e60463ae46a84e@mail.gmail.com>

Oops! My method is called

process_orders

and not 'process'. Sorry :)

- jason

On Sat, May 3, 2008 at 8:42 AM, Jason Lee <jasonlee9 at gmail.com> wrote:
> Adam,
>
>  Not sure if it's a Rails 1.2.5 issue (I'm on 2.0.2). I've used the
>  scheduler before and found it not as reliable as Unix cron, so I moved
>  the scheduling code into the worker itself. I have BRb doing
>  background scheduled email tasks and it works quite well. Here's what
>  mine looks like:
>
>  class EmailWorker < BackgrounDRb::MetaWorker
>
>   set_worker_name :Email_worker
>
>   def create(args = nil)
>     # time argument is in seconds
>     add_periodic_timer(10) { process_orders }
>   end
>
>   def process
>      # do some work
>   end
>
>  end
>
>
>  My config looks like so (for development):
>
>  ---
>
> :backgroundrb:
>     :ip: 0.0.0.0
>     :port: 11006
>     :environment: development
>
>
>  I removed all my scheduler code out of the config and it works well,
>  so maybe that will help.
>
>  - jason
>
>
>
>
>  On Fri, May 2, 2008 at 7:20 PM, Adam Kapelner <kapelner at gmail.com> wrote:
>  > Hello,
>  >
>  > I'm trying to send emails from a backgroundrb worker and the email doesn't
>  > send. I get no log messages in the sendmail log or the system log, nothing
>  > in the backgroundrb logs at all, nothing in the production log.
>  >
>  > Below is my code for each component
>  >
>  > worker:
>  >
>  > class QuizReviewerWorker < BackgrounDRb::MetaWorker
>  >    set_worker_name :quiz_review_sender
>  >
>  >    def create(args = nil)
>  >   end
>  >
>  >   def send_review_emails
>  >     mail = Notifier.create_test_email
>  >      status = Notifier.deliver(mail)
>  >     logger.info 'test mail:' + mail.body
>  >    end
>  > end
>  >
>  > mailer:
>  >
>  >  class Notifier < ActionMailer::Base
>  >
>  >   def test_email
>  >      recipients 'kapelner at gmail.com'
>  >     subject 'test email'
>  >      body 'body of email'
>  >     content_type "text/plain"
>  >    end
>  > end
>  >
>  > backgroundrb configuration:
>  >
>  >  ---
>  > :backgroundrb:
>  >    :ip: 0.0.0.0
>  >   :port: 11006
>  >    :environment: production
>  >   :debug_log: true
>  >  :schedules:
>  >   :quiz_review_sender:
>  >      :send_review_emails:
>  >       :trigger_args:
>  >          :start: <%= Time.now + 30.seconds %>
>  >         :end: <%= Time.now + 10.years %>
>  >          :repeat_interval: <%= 20.seconds %>
>  >
>  > environment.rb configuration:
>  >
>  > ActionMailer::Base.delivery_method = :sendmail
>  >  ActionMailer::Base.raise_delivery_errors = true
>  > ActionMailer::Base.default_content_type = 'text/html'
>  >
>  >
>  > Now, when doing the request from rails itself, no problem, I get the email
>  > instantly. Here's the controller code:
>  >
>  > class MiscellanyController < ApplicationController
>  >    def test_email
>  >     mail = Notifier.create_test_email
>  >      Notifier.deliver(mail)
>  >     render :text => mail.body
>  >    end
>  > end
>  >
>  >
>  > What could be the issue? I'm using Ruby 1.8.6, Rails 1.2.5, on Fedora Core
>  > 6, Backgroundrb (last updated 2008-02-28)
>  >
>  > thanks so much,
>  > Adam
>  >
>  >
>  >
>  >
>  >
>  >
>  > _______________________________________________
>  >  Backgroundrb-devel mailing list
>  >  Backgroundrb-devel at rubyforge.org
>  >  http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>  >
>

From sohara at sohara.com  Sun May  4 12:56:15 2008
From: sohara at sohara.com (Sean O'Hara)
Date: Sun, 4 May 2008 12:56:15 -0400
Subject: [Backgroundrb-devel] best approach to managing workers and getting
	status
Message-ID: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>

Hi,

I am using backgroundrb to process audio files from a rails  
controller. Currently a new worker gets created every time the method  
is called on the worker, using this code:

@job_key = MiddleMan.new_worker(:worker  
=> :audio_file_worker, :job_key => Time.now.to_i)
MiddleMan.worker(:audio_file_worker,  
@job_key).make_new_audio_file(params[:release_id])

I need to create the new worker each time in order to be able to get  
the status from the worker using the job key (this info is returned to  
the client using ajax requests). But this means that I end up with  
many workers eating up memory, and just hanging around after their  
jobs are complete.

I am planning to just kill them from the controller each time the  
status returns that they are complete. This will prevent the extra  
processes from hanging around and using memory, but I guess it still  
entails some 'costs' in starting up the new worker each time, since it  
contains a rails instance?

Is there a better approach, e.g. just having one worker, and sending  
the jobs to the same worker? If so, how do we keep track of statuses  
of unique jobs in this case? Since the job_key is created when  
creating a new worker, isn't in effect a worker key, rather than a job  
key?

I did read about this different approach discussed here (using  
thread_pool.defer) but it doesn't seem to allow for getting status of  
the unique threads, as far as I can tell: http://rubyforge.org/pipermail/backgroundrb-devel/2008-February/001532.html

Any guidance is appreciated.

Thanks,
Sean




From scott at butlerpress.com  Sun May  4 19:09:30 2008
From: scott at butlerpress.com (Scott Willson)
Date: Sun, 4 May 2008 16:09:30 -0700
Subject: [Backgroundrb-devel] Rails Configuration in Workers
Message-ID: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>

I override some Rails' configuration options in my project, like:
     config.database_configuration_file = "#{RAILS_ROOT}/local/config/ 
database.yml"

But they aren't picked up in BackgroundRB because Rails ActiveRecord  
is just initialized directly in MasterWorker#load_rails_env:
       db_config_file = YAML.load(ERB.new(IO.read("#{RAILS_HOME}/ 
config/database.yml")).result)

Is this an intentional design decision, or just simplest approach  
tha's "good enough?" Personally, I'd prefer that my Rails' environment  
was initialized the same way in my client application, BackgroundRB  
server and the workers?and I'm happy to contribute a patch to do it.  
If it makes sense. Is this a dead end that's been discussed before, or  
has just no one needed it before?

Thanks for any info.

Scott


From bobby.santiago at gmail.com  Sun May  4 23:02:30 2008
From: bobby.santiago at gmail.com (Bobby Santiago)
Date: Mon, 5 May 2008 11:02:30 +0800
Subject: [Backgroundrb-devel] Deploying to a staging server using
	Capistrano: how to start up BackgrounDRb?
Message-ID: <63FAEBEE-2B68-4EC2-B345-9B212A9BF3FE@gmail.com>

Hi.

I am using BackgrounDRb to process thumbnails and upload to S3 -  
things are hunky-dory in development (thumbs are generated, these are  
uploaded to S3, the metadata is saved to trhe DB, and I get a nice  
status page updated by periodic calls via ask_status), but when I  
tried to deploy to our staging server and stop/start BackgrounDRb via  
Capistrano, things blew up - well, not exactly, but when the app tried  
to hand off the thumbnail generation, I got a "Could not connect to  
the BackgrounDRb server" error.

Initially, when I didn't start the BDRb server manually, I got:

* executing `after_update_code'
* executing `set_env_staging'
ENV['RAILS_ENV'] = staging
* executing `restart_backgroundrb'
* executing `stop_backgroundrb'
* executing "cd /var/www/apps/my_app/releases/20080501133415 && ./ 
script/backgroundrb -e staging stop"
  servers: ["staging.my_app.com"]
  [staging.my_app.com] executing command
** [out :: staging.my_app.com] ./script/backgroundrb:46:in `initialize'
** [out :: staging.my_app.com] :
** [out :: staging.my_app.com] No such file or directory - /var/www/ 
apps/my_app/releases/20080501133415/tmp/pids/backgroundrb_22222.pid
** [out :: staging.my_app.com] (
** [out :: staging.my_app.com] Errno::ENOENT
** [out :: staging.my_app.com] )
** [out :: staging.my_app.com] from ./script/backgroundrb:46:in `open'
** [out :: staging.my_app.com] from ./script/backgroundrb:46
  command finished
*** [deploy:update_code] rolling back
* executing "rm -rf /var/www/apps/my_app/releases/20080501133415; true"
  servers: ["staging.my_app.com"]
  [staging.my_app.com] executing command
  command finished

So I tried manually starting backgroundrb on the staging server,, and  
the deployment went through successfully. But when I tried generating  
thumbs with a background worker, I got the "Could not connect to the  
BackgrounDRb server" error.

I check out from GitHub, and use Capistrano-ext for multistage  
deployments:

cap staging deploy

Before I deploy, I SSH into the VPS and do
script/backgroundrb -e staging start

in my config/deploy/staging.rb file:
----------------
set :rails_env, 'staging'
...

task :after_update_code do
  set_env_staging
  copy_mongrel_cluster_config
  restart_backgroundrb
end

desc "Sets the environment variable RAILS_ENV='staging'."
task :set_env_staging do
  ENV['RAILS_ENV'] = 'staging'
  puts "ENV['RAILS_ENV'] = #{ENV['RAILS_ENV']}"
end

desc "Copying the right mongrel cluster config for the current stage  
environment."
task :copy_mongrel_cluster_config do
  run "cp -f #{release_path}/config/deploy/staging/mongrel_cluster.yml  
#{release_path}/config/mongrel_cluster.yml"
end

desc "Start the backgroundrb server"
  task :start_backgroundrb , :roles => :app do
    run "cd #{current_path} && nohup ./script/backgroundrb start -- -r  
staging > #{current_path}/log/backgroundrb-cap.log 2>&1"
  end


desc "Stop the backgroundrb server"
task :stop_backgroundrb, :roles => :app do
  run "cd #{release_path} && ./script/backgroundrb -e staging stop"
end

desc "Restart the backgroundrb server"
task :restart_backgroundrb, :roles => :app do
  stop_backgroundrb
  start_backgroundrb
end

-------
Here's the exception_notifier backtrace:

  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:81:in  
`dump_object'
  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:98:in  
`new_worker'
  [RAILS_ROOT]/app/models/uploader.rb:54:in `start_thumbnail_worker'
  [RAILS_ROOT]/app/controllers/photos_controller.rb:49:in `create'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
mime_responds.rb:106:in `call'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
mime_responds.rb:106:in `respond_to'
  [RAILS_ROOT]/app/controllers/photos_controller.rb:47:in `create'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
base.rb:1158:in `send'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
base.rb:1158:in `perform_action_without_filters'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
filters.rb:697:in `call_filters'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
filters.rb:689:in `perform_action_without_benchmark'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
benchmarking.rb:68:in `perform_action_without_rescue'
  /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
benchmarking.rb:68:in `perform_action_without_rescue'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
rescue.rb:199:in `perform_action_without_caching'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
caching.rb:678:in `perform_action'
  /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
connection_adapters/abstract/query_cache.rb:33:in `cache'
  /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
query_cache.rb:8:in `cache'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
caching.rb:677:in `perform_action'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
base.rb:524:in `send'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
base.rb:524:in `process_without_filters'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
filters.rb:685:in `process_without_session_management_support'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
session_management.rb:123:in `sass_old_process'
  /usr/lib/ruby/gems/1.8/gems/haml-1.8.1/lib/sass/plugin/rails.rb: 
15:in `process'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
base.rb:388:in `process'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
dispatcher.rb:171:in `handle_request'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
dispatcher.rb:115:in `dispatch'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
dispatcher.rb:126:in `dispatch_cgi'
  /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
dispatcher.rb:9:in `dispatch'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in  
`process'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in  
`synchronize'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in  
`process'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in  
`process_client'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `each'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in  
`process_client'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
`initialize'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `new'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in  
`initialize'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `new'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
configurator.rb:282:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
configurator.rb:281:in `each'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
configurator.rb:281:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in  
`run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb: 
212:in `run'
  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
  /usr/bin/mongrel_rails:19:in `load'
  /usr/bin/mongrel_rails:19

My backgroundrb.yml file:
:backgroundrb:
   :ip: 0.0.0.0

:development:
   :backgroundrb:
     :port: 11006     # use port 11006
     :log: foreground # foreground mode,print log messages on console
     :debug_log: true

:staging:
   :backgroundrb:
     :port: 22222      # use port 22222
     :lazy_load: true  # do not load models eagerly
     :debug_log: false # disable log workers and other logging

:production:
   :backgroundrb:
     :port: 33333      # use port 33333
     :lazy_load: true  # do not load models eagerly
     :debug_log: false # disable log workers and other logging

My worker:
class ThumbnailGeneratorWorker < BackgrounDRb::MetaWorker
   set_worker_name :thumbnail_generator_worker
   set_no_auto_load(true)

   def create(args = nil)
     register_status(:percent_complete => 0)
     args.each_with_index do |uploader_id, index|
       @uploader = Uploader.find(uploader_id)
       @uploader.generate_thumbnails
       percent_complete = ((index + 1) * 100) / args.length
       logger.info "Thumbnail generation is #{percent_complete}%  
complete..."
       register_status(:percent_complete => percent_complete)
     end
     exit

   end
end

Any hints/help/ideas would be much appreciated.

Bobby

From jnutting at gmail.com  Mon May  5 04:35:51 2008
From: jnutting at gmail.com (Jack Nutting)
Date: Mon, 5 May 2008 10:35:51 +0200
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
Message-ID: <f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>

The way that I (and perhaps many others here?) handle this is to use
 the database to hold a "work queue", and have a long-running worker
 that polls the database periodically and handles any pending requests.
  In your case for example, you could have an AudioFileWork model,
 containing fields for "release_id" and "pending";  Your app would
 create a new instance of that with pending=true, and your background
 worker would poll for rows where pending=true, then mark them as false
 when they are complete.  For your post-creation interactions (to show
 the status with ajax) you'd use the id of the created AudioFileWork
 instead of a job key.

 There are many advantages to this approach:

 - you always have a known number of workers (the number you configure
 and start), so you won't have uncontrollable memory usage explosions
 if your site gets busy, just slower response times
 - you can check the status of a request by querying the database
 - there is very little in-memory data that is lost in unhappy events
 (a crash or unhandled exception)
 - you have a historical record in the databse of all work that is
 completed by a background worker
 - you get a very loose coupling to backgroundrb.  In my case, I simply
 specify in a model class that it needs processing, and it just
 happens.  My app doesn't "talk" directly with backgroundrb at all,
 except for an admin page where I can make sure it's up and running.



-- 
// jack
// http://www.nuthole.com

From adam at thewilliams.ws  Mon May  5 07:56:17 2008
From: adam at thewilliams.ws (Adam Williams)
Date: Mon, 5 May 2008 07:56:17 -0400
Subject: [Backgroundrb-devel] Storing worker driven rails logging in
	worker log file?
In-Reply-To: <A7541678-747B-4B4C-A09B-E75AA75EC6FF@nooked.com>
References: <A7541678-747B-4B4C-A09B-E75AA75EC6FF@nooked.com>
Message-ID: <0E8D5F73-AD82-4885-A953-18CFCABEBB5D@thewilliams.ws>

I don't know the answer to this, but I'd second it and ask, Is this  
why there is a 150M logging worker???

On May 2, 2008, at 2:23 PM, John O'Shea wrote:

> Hi all,
> 	We have a few workers that kick off at scheduled intervals or to do  
> some user driven long running tasks asynchronously.  At the moment  
> when the workers start working with AR model objects, all of the  
> Rails logging ends up in the default rails environment log file  
> (develolopment/prodiction.log).
> 	Does anyone have any tips for getting rails to append to a separate  
> worker log file?  We'd like to keep the rails logging in a separate  
> file so that we can easily find it later - mixed into the main rails  
> log it can be difficult to figure out which log entries are as a  
> result of controller actions doing their work and which are being  
> driven by workers.
>
> Thanks for any tips,
>
> John.
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From adam at thewilliams.ws  Mon May  5 08:13:30 2008
From: adam at thewilliams.ws (Adam Williams)
Date: Mon, 5 May 2008 08:13:30 -0400
Subject: [Backgroundrb-devel] Rails Configuration in Workers
In-Reply-To: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>
References: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>
Message-ID: <BCF33C74-8FB7-40F9-B020-0DBC382ABA25@thewilliams.ws>

I'm not a dev of backgroundrb, but in the two weeks I've been using it  
- this is definitely in need of repair. Look at this simple Google  
search of the list archive: http://tinyurl.com/6xk4g2  Intentional or  
not, folks are confused by it.

BackgrounDRb developers - Thanks for all the hard work. I only want to  
encourage Scott to help you make it better ;)

   aiwilliams

On May 4, 2008, at 7:09 PM, Scott Willson wrote:

> I override some Rails' configuration options in my project, like:
>    config.database_configuration_file = "#{RAILS_ROOT}/local/config/ 
> database.yml"
>
> But they aren't picked up in BackgroundRB because Rails ActiveRecord  
> is just initialized directly in MasterWorker#load_rails_env:
>      db_config_file = YAML.load(ERB.new(IO.read("#{RAILS_HOME}/ 
> config/database.yml")).result)
>
> Is this an intentional design decision, or just simplest approach  
> tha's "good enough?" Personally, I'd prefer that my Rails'  
> environment was initialized the same way in my client application,  
> BackgroundRB server and the workers?and I'm happy to contribute a  
> patch to do it. If it makes sense. Is this a dead end that's been  
> discussed before, or has just no one needed it before?
>
> Thanks for any info.
>
> Scott
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From stevie at slowbicycle.com  Mon May  5 12:14:00 2008
From: stevie at slowbicycle.com (Stevie Clifton)
Date: Mon, 5 May 2008 12:14:00 -0400
Subject: [Backgroundrb-devel] Deploying to a staging server using
	Capistrano: how to start up BackgrounDRb?
In-Reply-To: <63FAEBEE-2B68-4EC2-B345-9B212A9BF3FE@gmail.com>
References: <63FAEBEE-2B68-4EC2-B345-9B212A9BF3FE@gmail.com>
Message-ID: <a426f29f0805050914x480d5edcu40f91688d2a8cca3@mail.gmail.com>

Hey Bobby,

I think you're dealing with two separate issues here.  The first error
you're seeing with capistrano is b/c capistrano doesn't auto-link the
/RAILS_ROOT/tmp/pids folder, so backgroundrb is unable to create a pid
file there.  I do something like the following (untested):

after "deploy:update_code", "deploy:symlink_pids"

namespace :deploy
  desc "Symlink tmp/pids folder"
  task :symlink_pids do
    run "mkdir -p #{release_path}/tmp"
    run "ln -nfs #{shared_path}/pids #{release_path}/tmp/pids"
  end
end

For the second issue, it's a little difficult knowing why rails can't
see your worker without seeing some bdrb log output.  Have you
verified that the worker is actually running? (ps aux | grep
<worker_name>).  Also, have you checked your backgroundrb_server.log
to see if it's dying on instantiation because of some other
dependencies?  I'd recommend launching backgroundrb manually on
staging and setting :log: foreground in backgroundrb.yml so you can
see what's going on with your worker.

stevie


On Sun, May 4, 2008 at 11:02 PM, Bobby Santiago
<bobby.santiago at gmail.com> wrote:
> Hi.
>
>  I am using BackgrounDRb to process thumbnails and upload to S3 - things are
> hunky-dory in development (thumbs are generated, these are uploaded to S3,
> the metadata is saved to trhe DB, and I get a nice status page updated by
> periodic calls via ask_status), but when I tried to deploy to our staging
> server and stop/start BackgrounDRb via Capistrano, things blew up - well,
> not exactly, but when the app tried to hand off the thumbnail generation, I
> got a "Could not connect to the BackgrounDRb server" error.
>
>  Initially, when I didn't start the BDRb server manually, I got:
>
>  * executing `after_update_code'
>  * executing `set_env_staging'
>  ENV['RAILS_ENV'] = staging
>  * executing `restart_backgroundrb'
>  * executing `stop_backgroundrb'
>  * executing "cd /var/www/apps/my_app/releases/20080501133415 &&
> ./script/backgroundrb -e staging stop"
>   servers: ["staging.my_app.com"]
>   [staging.my_app.com] executing command
>  ** [out :: staging.my_app.com] ./script/backgroundrb:46:in `initialize'
>  ** [out :: staging.my_app.com] :
>  ** [out :: staging.my_app.com] No such file or directory -
> /var/www/apps/my_app/releases/20080501133415/tmp/pids/backgroundrb_22222.pid
>  ** [out :: staging.my_app.com] (
>  ** [out :: staging.my_app.com] Errno::ENOENT
>  ** [out :: staging.my_app.com] )
>  ** [out :: staging.my_app.com] from ./script/backgroundrb:46:in `open'
>  ** [out :: staging.my_app.com] from ./script/backgroundrb:46
>   command finished
>  *** [deploy:update_code] rolling back
>  * executing "rm -rf /var/www/apps/my_app/releases/20080501133415; true"
>   servers: ["staging.my_app.com"]
>   [staging.my_app.com] executing command
>   command finished
>
>  So I tried manually starting backgroundrb on the staging server,, and the
> deployment went through successfully. But when I tried generating thumbs
> with a background worker, I got the "Could not connect to the BackgrounDRb
> server" error.
>
>  I check out from GitHub, and use Capistrano-ext for multistage deployments:
>
>  cap staging deploy
>
>  Before I deploy, I SSH into the VPS and do
>  script/backgroundrb -e staging start
>
>  in my config/deploy/staging.rb file:
>  ----------------
>  set :rails_env, 'staging'
>  ...
>
>  task :after_update_code do
>   set_env_staging
>   copy_mongrel_cluster_config
>   restart_backgroundrb
>  end
>
>  desc "Sets the environment variable RAILS_ENV='staging'."
>  task :set_env_staging do
>   ENV['RAILS_ENV'] = 'staging'
>   puts "ENV['RAILS_ENV'] = #{ENV['RAILS_ENV']}"
>  end
>
>  desc "Copying the right mongrel cluster config for the current stage
> environment."
>  task :copy_mongrel_cluster_config do
>   run "cp -f #{release_path}/config/deploy/staging/mongrel_cluster.yml
> #{release_path}/config/mongrel_cluster.yml"
>  end
>
>  desc "Start the backgroundrb server"
>   task :start_backgroundrb , :roles => :app do
>    run "cd #{current_path} && nohup ./script/backgroundrb start -- -r
> staging > #{current_path}/log/backgroundrb-cap.log 2>&1"
>   end
>
>
>  desc "Stop the backgroundrb server"
>  task :stop_backgroundrb, :roles => :app do
>   run "cd #{release_path} && ./script/backgroundrb -e staging stop"
>  end
>
>  desc "Restart the backgroundrb server"
>  task :restart_backgroundrb, :roles => :app do
>   stop_backgroundrb
>   start_backgroundrb
>  end
>
>  -------
>  Here's the exception_notifier backtrace:
>
>   [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:81:in
> `dump_object'
>   [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:98:in
> `new_worker'
>   [RAILS_ROOT]/app/models/uploader.rb:54:in `start_thumbnail_worker'
>   [RAILS_ROOT]/app/controllers/photos_controller.rb:49:in `create'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/mime_responds.rb:106:in
> `call'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/mime_responds.rb:106:in
> `respond_to'
>   [RAILS_ROOT]/app/controllers/photos_controller.rb:47:in `create'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in
> `send'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in
> `perform_action_without_filters'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:697:in
> `call_filters'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:689:in
> `perform_action_without_benchmark'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in
> `perform_action_without_rescue'
>   /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in
> `perform_action_without_rescue'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/rescue.rb:199:in
> `perform_action_without_caching'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:678:in
> `perform_action'
>
> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in
> `cache'
>
> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/query_cache.rb:8:in
> `cache'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:677:in
> `perform_action'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in
> `send'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in
> `process_without_filters'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:685:in
> `process_without_session_management_support'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/session_management.rb:123:in
> `sass_old_process'
>   /usr/lib/ruby/gems/1.8/gems/haml-1.8.1/lib/sass/plugin/rails.rb:15:in
> `process'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:388:in
> `process'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:171:in
> `handle_request'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:115:in
> `dispatch'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:126:in
> `dispatch_cgi'
>
> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:9:in
> `dispatch'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in
> `process'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
> `synchronize'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
> `process'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in
> `process_client'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `each'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in
> `process_client'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in
> `initialize'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `new'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in
> `initialize'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `new'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `run'
>
> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:282:in
> `run'
>
> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in
> `each'
>
> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in
> `run'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in `run'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb:212:in
> `run'
>   /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
>   /usr/bin/mongrel_rails:19:in `load'
>   /usr/bin/mongrel_rails:19
>
>  My backgroundrb.yml file:
>  :backgroundrb:
>   :ip: 0.0.0.0
>
>  :development:
>   :backgroundrb:
>     :port: 11006     # use port 11006
>     :log: foreground # foreground mode,print log messages on console
>     :debug_log: true
>
>  :staging:
>   :backgroundrb:
>     :port: 22222      # use port 22222
>     :lazy_load: true  # do not load models eagerly
>     :debug_log: false # disable log workers and other logging
>
>  :production:
>   :backgroundrb:
>     :port: 33333      # use port 33333
>     :lazy_load: true  # do not load models eagerly
>     :debug_log: false # disable log workers and other logging
>
>  My worker:
>  class ThumbnailGeneratorWorker < BackgrounDRb::MetaWorker
>   set_worker_name :thumbnail_generator_worker
>   set_no_auto_load(true)
>
>   def create(args = nil)
>     register_status(:percent_complete => 0)
>     args.each_with_index do |uploader_id, index|
>       @uploader = Uploader.find(uploader_id)
>       @uploader.generate_thumbnails
>       percent_complete = ((index + 1) * 100) / args.length
>       logger.info "Thumbnail generation is #{percent_complete}% complete..."
>       register_status(:percent_complete => percent_complete)
>     end
>     exit
>
>   end
>  end
>
>  Any hints/help/ideas would be much appreciated.
>
>  Bobby
>  _______________________________________________
>  Backgroundrb-devel mailing list
>  Backgroundrb-devel at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From jnutting at gmail.com  Mon May  5 12:33:47 2008
From: jnutting at gmail.com (Jack Nutting)
Date: Mon, 5 May 2008 18:33:47 +0200
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <D6C4853F-DDBE-4B13-81ED-DBD310025FDB@sohara.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<D6C4853F-DDBE-4B13-81ED-DBD310025FDB@sohara.com>
Message-ID: <f84841020805050933j43e1aebas58508118b35a7381@mail.gmail.com>

On Mon, May 5, 2008 at 4:25 PM, Sean O'Hara <sohara at sohara.com> wrote:
> Hi Jack,
>
>  That approach sounds very good, and makes a lot of sense for this kind of
> job. Although, it doesn't give me the satisfaction of watching the ajax
> progress bar :)
>
>  But could we do for a background job that really does require providing
> feedback to the user, such as processing a credit card transaction for an
> order in real time. I would like to offload the job to backgroundrb so that
> the user is getting some feedback, and isn't tempted to submit the order
> twice out of impatience, but I also don't want extra workers hanging around,
> or to be incurring the extra memory usage associated with starting them up.
> Is there a way to have a single worker act as the transaction processor, but
> still be able to give back the status of unique jobs to rails?

How about an extension of the approach I mentioned earlier:  Instead
of just a simple boolean "pending" flag, you could have a field of any
kind you like:  An integer or a float to indicate percentage complete,
or a string for arbitrary text.  Then, while your backgroundrb worker
is doing its thing, it can update the field with the current status
(percentage complete, or "authenticating card number..." etc), and
your ajax method could grab that from the database instead of asking
the worker for it.

-- 
// jack
// http://www.nuthole.com

From bobby.santiago at gmail.com  Mon May  5 20:27:42 2008
From: bobby.santiago at gmail.com (Bobby Santiago)
Date: Tue, 6 May 2008 08:27:42 +0800
Subject: [Backgroundrb-devel] Deploying to a staging server using
	Capistrano: how to start up BackgrounDRb?
In-Reply-To: <a426f29f0805050914x480d5edcu40f91688d2a8cca3@mail.gmail.com>
References: <63FAEBEE-2B68-4EC2-B345-9B212A9BF3FE@gmail.com>
	<a426f29f0805050914x480d5edcu40f91688d2a8cca3@mail.gmail.com>
Message-ID: <5E66D556-D9BF-43F9-A3A9-033479FC1F5B@gmail.com>

Hey, Stevie,

When I changed the port number in backgroundrb.yml to match the  
mongrel ports in my conf file, the connection error went away. I  
looked into /var/www/apps/my_app/shared/pids, and  
backgroundrb_9201.pid is there. In the /var/www/apps/my_app/shared/log  
dir, I have
backgroundrb_9201.log
backgroundrb_9201_debug.log
backgroundrb_server_9201.log

When I did  ps aux | grep thumbnail_generator_worker, I got:
deploy   12279  0.0  0.1   1816   496 pts/0    R+   20:07   0:00 grep  
thumbnail_generator_worker
So I guess it's not running.

After setting :log: foreground in backgroundrb.yml and deploying, I  
tailed the backgroundrb_9201.log, backgroundrb_9201_debug.log, and  
backgroundrb_server_9201.log files. No output there.

In development, the backgroundrb_1106.log file has the correct output:
BillingWorker#create called...
billing_worker started
Schedules for worker loaded
ThumbnailGeneratorWorker#create called...
thumbnail_generator_worker started
Schedules for worker loaded
generate 37

There is no feedback from the ask_status method, I get nil when I  
inspect the variable in staging.log.

I'm running out of ideas.

Bobby

On May 6, 2008, at 12:14 AM, Stevie Clifton wrote:

> Hey Bobby,
>
> I think you're dealing with two separate issues here.  The first error
> you're seeing with capistrano is b/c capistrano doesn't auto-link the
> /RAILS_ROOT/tmp/pids folder, so backgroundrb is unable to create a pid
> file there.  I do something like the following (untested):
>
> after "deploy:update_code", "deploy:symlink_pids"
>
> namespace :deploy
>  desc "Symlink tmp/pids folder"
>  task :symlink_pids do
>    run "mkdir -p #{release_path}/tmp"
>    run "ln -nfs #{shared_path}/pids #{release_path}/tmp/pids"
>  end
> end
>
> For the second issue, it's a little difficult knowing why rails can't
> see your worker without seeing some bdrb log output.  Have you
> verified that the worker is actually running? (ps aux | grep
> <worker_name>).  Also, have you checked your backgroundrb_server.log
> to see if it's dying on instantiation because of some other
> dependencies?  I'd recommend launching backgroundrb manually on
> staging and setting :log: foreground in backgroundrb.yml so you can
> see what's going on with your worker.
>
> stevie
>
>
> On Sun, May 4, 2008 at 11:02 PM, Bobby Santiago
> <bobby.santiago at gmail.com> wrote:
>> Hi.
>>
>> I am using BackgrounDRb to process thumbnails and upload to S3 -  
>> things are
>> hunky-dory in development (thumbs are generated, these are uploaded  
>> to S3,
>> the metadata is saved to trhe DB, and I get a nice status page  
>> updated by
>> periodic calls via ask_status), but when I tried to deploy to our  
>> staging
>> server and stop/start BackgrounDRb via Capistrano, things blew up -  
>> well,
>> not exactly, but when the app tried to hand off the thumbnail  
>> generation, I
>> got a "Could not connect to the BackgrounDRb server" error.
>>
>> Initially, when I didn't start the BDRb server manually, I got:
>>
>> * executing `after_update_code'
>> * executing `set_env_staging'
>> ENV['RAILS_ENV'] = staging
>> * executing `restart_backgroundrb'
>> * executing `stop_backgroundrb'
>> * executing "cd /var/www/apps/my_app/releases/20080501133415 &&
>> ./script/backgroundrb -e staging stop"
>>  servers: ["staging.my_app.com"]
>>  [staging.my_app.com] executing command
>> ** [out :: staging.my_app.com] ./script/backgroundrb:46:in  
>> `initialize'
>> ** [out :: staging.my_app.com] :
>> ** [out :: staging.my_app.com] No such file or directory -
>> /var/www/apps/my_app/releases/20080501133415/tmp/pids/ 
>> backgroundrb_22222.pid
>> ** [out :: staging.my_app.com] (
>> ** [out :: staging.my_app.com] Errno::ENOENT
>> ** [out :: staging.my_app.com] )
>> ** [out :: staging.my_app.com] from ./script/backgroundrb:46:in  
>> `open'
>> ** [out :: staging.my_app.com] from ./script/backgroundrb:46
>>  command finished
>> *** [deploy:update_code] rolling back
>> * executing "rm -rf /var/www/apps/my_app/releases/20080501133415;  
>> true"
>>  servers: ["staging.my_app.com"]
>>  [staging.my_app.com] executing command
>>  command finished
>>
>> So I tried manually starting backgroundrb on the staging server,,  
>> and the
>> deployment went through successfully. But when I tried generating  
>> thumbs
>> with a background worker, I got the "Could not connect to the  
>> BackgrounDRb
>> server" error.
>>
>> I check out from GitHub, and use Capistrano-ext for multistage  
>> deployments:
>>
>> cap staging deploy
>>
>> Before I deploy, I SSH into the VPS and do
>> script/backgroundrb -e staging start
>>
>> in my config/deploy/staging.rb file:
>> ----------------
>> set :rails_env, 'staging'
>> ...
>>
>> task :after_update_code do
>>  set_env_staging
>>  copy_mongrel_cluster_config
>>  restart_backgroundrb
>> end
>>
>> desc "Sets the environment variable RAILS_ENV='staging'."
>> task :set_env_staging do
>>  ENV['RAILS_ENV'] = 'staging'
>>  puts "ENV['RAILS_ENV'] = #{ENV['RAILS_ENV']}"
>> end
>>
>> desc "Copying the right mongrel cluster config for the current stage
>> environment."
>> task :copy_mongrel_cluster_config do
>>  run "cp -f #{release_path}/config/deploy/staging/mongrel_cluster.yml
>> #{release_path}/config/mongrel_cluster.yml"
>> end
>>
>> desc "Start the backgroundrb server"
>>  task :start_backgroundrb , :roles => :app do
>>   run "cd #{current_path} && nohup ./script/backgroundrb start -- -r
>> staging > #{current_path}/log/backgroundrb-cap.log 2>&1"
>>  end
>>
>>
>> desc "Stop the backgroundrb server"
>> task :stop_backgroundrb, :roles => :app do
>>  run "cd #{release_path} && ./script/backgroundrb -e staging stop"
>> end
>>
>> desc "Restart the backgroundrb server"
>> task :restart_backgroundrb, :roles => :app do
>>  stop_backgroundrb
>>  start_backgroundrb
>> end
>>
>> -------
>> Here's the exception_notifier backtrace:
>>
>>  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:81:in
>> `dump_object'
>>  [RAILS_ROOT]/vendor/plugins/backgroundrb/lib/backgroundrb.rb:98:in
>> `new_worker'
>>  [RAILS_ROOT]/app/models/uploader.rb:54:in `start_thumbnail_worker'
>>  [RAILS_ROOT]/app/controllers/photos_controller.rb:49:in `create'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> mime_responds.rb:106:in
>> `call'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> mime_responds.rb:106:in
>> `respond_to'
>>  [RAILS_ROOT]/app/controllers/photos_controller.rb:47:in `create'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:1158:in
>> `send'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:1158:in
>> `perform_action_without_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:697:in
>> `call_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:689:in
>> `perform_action_without_benchmark'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> benchmarking.rb:68:in
>> `perform_action_without_rescue'
>>  /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> benchmarking.rb:68:in
>> `perform_action_without_rescue'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> rescue.rb:199:in
>> `perform_action_without_caching'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> caching.rb:678:in
>> `perform_action'
>>
>> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> connection_adapters/abstract/query_cache.rb:33:in
>> `cache'
>>
>> /usr/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> query_cache.rb:8:in
>> `cache'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> caching.rb:677:in
>> `perform_action'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:524:in
>> `send'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:524:in
>> `process_without_filters'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> filters.rb:685:in
>> `process_without_session_management_support'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> session_management.rb:123:in
>> `sass_old_process'
>>  /usr/lib/ruby/gems/1.8/gems/haml-1.8.1/lib/sass/plugin/rails.rb: 
>> 15:in
>> `process'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> base.rb:388:in
>> `process'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:171:in
>> `handle_request'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:115:in
>> `dispatch'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:126:in
>> `dispatch_cgi'
>>
>> /usr/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/ 
>> dispatcher.rb:9:in
>> `dispatch'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in
>> `process'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
>> `synchronize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in
>> `process'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in
>> `process_client'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in  
>> `each'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in
>> `process_client'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in
>> `initialize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `new'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in
>> `initialize'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in  
>> `new'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in  
>> `run'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:282:in
>> `run'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:281:in
>> `each'
>>
>> /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/ 
>> configurator.rb:281:in
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in  
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb: 
>> 212:in
>> `run'
>>  /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
>>  /usr/bin/mongrel_rails:19:in `load'
>>  /usr/bin/mongrel_rails:19
>>
>> My backgroundrb.yml file:
>> :backgroundrb:
>>  :ip: 0.0.0.0
>>
>> :development:
>>  :backgroundrb:
>>    :port: 11006     # use port 11006
>>    :log: foreground # foreground mode,print log messages on console
>>    :debug_log: true
>>
>> :staging:
>>  :backgroundrb:
>>    :port: 22222      # use port 22222
>>    :lazy_load: true  # do not load models eagerly
>>    :debug_log: false # disable log workers and other logging
>>
>> :production:
>>  :backgroundrb:
>>    :port: 33333      # use port 33333
>>    :lazy_load: true  # do not load models eagerly
>>    :debug_log: false # disable log workers and other logging
>>
>> My worker:
>> class ThumbnailGeneratorWorker < BackgrounDRb::MetaWorker
>>  set_worker_name :thumbnail_generator_worker
>>  set_no_auto_load(true)
>>
>>  def create(args = nil)
>>    register_status(:percent_complete => 0)
>>    args.each_with_index do |uploader_id, index|
>>      @uploader = Uploader.find(uploader_id)
>>      @uploader.generate_thumbnails
>>      percent_complete = ((index + 1) * 100) / args.length
>>      logger.info "Thumbnail generation is #{percent_complete}%  
>> complete..."
>>      register_status(:percent_complete => percent_complete)
>>    end
>>    exit
>>
>>  end
>> end
>>
>> Any hints/help/ideas would be much appreciated.
>>
>> Bobby
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>


From rogboone at yahoo.com  Fri May  9 09:15:57 2008
From: rogboone at yahoo.com (Roggie Boone)
Date: Fri, 9 May 2008 06:15:57 -0700 (PDT)
Subject: [Backgroundrb-devel] register_status for excess thread_pool?
Message-ID: <70978.29424.qm@web32507.mail.mud.yahoo.com>

Hi,

Newbie here.  I've got a worker (for generating PDF reports) that
uses the "thread_pool" to allow processing multiple reports 
simultaneously and queue up any requests that exceed the thread
pool (pool_size = 10 currently).

  def process_pdf(user)
      thread_pool.defer(user) do |user|
        makepdf(user)
      end
  end

My question is:  I use a mutex to handle synchronizing the
register_status.  This works fine:

def makepdf(user)
      txt = user.to_s + " started"
      save_status(user, :progress, txt)
      do_report(user)      
      txt = user.to_s + " ended"
      save_status(user, :progress, txt) 
  end

  def save_status(user_id,key,data)
    @status_mutex.synchronize do
      @worker_status[user_id] = { :key => key, :data => data}
    end
    register_status(@worker_status)
  end

However, if more than 10 requests are submitted to the worker
at a time, those in excess of 10 are queued and thus never
get assigned a status in register_status.  So, in my Rails/ajax
view, I have no "status" to display while the user waits.
I read that one shouldn't use the register_status inside the
thread_pool.defer block.  Is there an alternate way to 
show a "pending" status to requests that have been deferred
due to exceeding the thread pool?

Thanks,
Rog

       
---------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile.  Try it now.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080509/6ac6a838/attachment.html>

From stevie at slowbicycle.com  Fri May  9 09:31:28 2008
From: stevie at slowbicycle.com (Stevie Clifton)
Date: Fri, 9 May 2008 09:31:28 -0400
Subject: [Backgroundrb-devel] register_status for excess thread_pool?
In-Reply-To: <70978.29424.qm@web32507.mail.mud.yahoo.com>
References: <70978.29424.qm@web32507.mail.mud.yahoo.com>
Message-ID: <a426f29f0805090631q5572e62dy1591181c1720dba7@mail.gmail.com>

Hey Roggie,

If I'm reading your code correctly, the simplest way would just be to
call your save_status method before thread_pool.  That way you would
guarantee that each pdf job would start out with a "pending" state
that would change only when there's a free thread to handle that job.

def process_pdf(user)
  save_status(user, :progress, "#{user} pending")
  thread_pool.defer(user) do |user|
    makepdf(user)
  end
end


>

On Fri, May 9, 2008 at 9:15 AM, Roggie Boone <rogboone at yahoo.com> wrote:
> Hi,
>
> Newbie here.  I've got a worker (for generating PDF reports) that
> uses the "thread_pool" to allow processing multiple reports
> simultaneously and queue up any requests that exceed the thread
> pool (pool_size = 10 currently).
>
>   def process_pdf(user)
>       thread_pool.defer(user) do |user|
>         makepdf(user)
>       end
>   end
>
> My question is:  I use a mutex to handle synchronizing the
> register_status.  This works fine:
>
> def makepdf(user)
>       txt = user.to_s + " started"
>       save_status(user, :progress, txt)
>       do_report(user)
>       txt = user.to_s + " ended"
>       save_status(user, :progress, txt)
>   end
>
>   def save_status(user_id,key,data)
>     @status_mutex.synchronize do
>       @worker_status[user_id] = { :key => key, :data => data}
>     end
>     register_status(@worker_status)
>   end
>
> However, if more than 10 requests are submitted to the worker
> at a time, those in excess of 10 are queued and thus never
> get assigned a status in register_status.  So, in my Rails/ajax
> view, I have no "status" to display while the user waits.
> I read that one shouldn't use the register_status inside the
> thread_pool.defer block.  Is there an alternate way to
> show a "pending" status to requests that have been deferred
> due to exceeding the thread pool?
>
> Thanks,
> Rog
>
> ________________________________
> Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it
> now.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From cariquez at gmail.com  Thu May 15 10:58:37 2008
From: cariquez at gmail.com (Luis Ferreira)
Date: Thu, 15 May 2008 15:58:37 +0100
Subject: [Backgroundrb-devel] scheduling jobs
Message-ID: <4ecd06df0805150758u6d176f52nd67ab0af50df5a89@mail.gmail.com>

Hi,
I need to schedule several jobs (defined by the user on a graphical
interface). Each job must have an initial date to start, an end and a
periodic call (every 5 seconds or every minute).

I also need to unschedule jobs even if they have already started.

I know i can do this using backgroundrb, but i can't find the documentation
anywhere. Can anyone help me?

Best Regards,
Luis Ferreira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080515/6eb1d8b9/attachment.html>

From raghu.srinivasan at gmail.com  Fri May 16 21:18:59 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 16 May 2008 18:18:59 -0700
Subject: [Backgroundrb-devel] Can someone help with this error?
Message-ID: <6482c06a0805161818ue775d4chfbb640b5dfcddf3d@mail.gmail.com>

Hello all,

I get this intermittent error saying OpenURI::HTTPError. At the bottom of
the email is what is reported in my backgroundrb_server_<port>.log file. The
process_netflix_reviews method is parsing (in the background)  Netflix's RSS
feeds. This error also pretty much brings down the BDRb process and to
recover, I am forced to do a

./script/backgroundrb stop
./script/backgroundrb start

Once I've done this, it chugs along fine - even for the same URI it crashed
on last. Until at some point in the future it barfs on another URI and I
need to restart BDRb... This has been happening on and off for several days
now.

Any pointers on what I am doing wrong?

Much appreciated - thanks!

Raghu

======================================================
/usr/lib/ruby/1.8/open-uri.rb:287:in `open_http': 500 Internal Server Error
(OpenURI::HTTPError)
        from /usr/lib/ruby/1.8/open-uri.rb:626:in `buffer_open'
        from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
        from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
        from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
        from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
        from /usr/lib/ruby/1.8/open-uri.rb:528:in `open'
        from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
        from /home/raghus/public_html/feedflix.com/ff/lib/netflix.rb:467:in`process_netflix_reviews'
         ... 21 levels...
        from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
        from /home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
        from ./script/backgroundrb:42:in `new'
        from ./script/backgroundrb:42
~
======================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080516/1827f18b/attachment.html>

From stevie at slowbicycle.com  Sat May 17 08:34:52 2008
From: stevie at slowbicycle.com (Stevie Clifton)
Date: Sat, 17 May 2008 08:34:52 -0400
Subject: [Backgroundrb-devel] Can someone help with this error?
In-Reply-To: <6482c06a0805161818ue775d4chfbb640b5dfcddf3d@mail.gmail.com>
References: <6482c06a0805161818ue775d4chfbb640b5dfcddf3d@mail.gmail.com>
Message-ID: <a426f29f0805170534q1ede6351n5e2ac028c2ba7306@mail.gmail.com>

Hey Raghu,

Without seeing more of what's going on in your code it's hard to
troubleshoot.  Seeing a 500/503 is a pretty normal part of
communicating with an external service, so you need to make sure you
are rescuing that error.  And what do you mean by "pretty much"
bringing down the bdrb process?  If you don't rescue that error I
would expect that your worker would die (disappear from your process
list) but that your bdrb master worker would keep on running.  Please
post your worker code (or at least your process_netflix_reviews
method) if you want troubleshooting help.

stevie

On Fri, May 16, 2008 at 9:18 PM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> Hello all,
>
> I get this intermittent error saying OpenURI::HTTPError. At the bottom of
> the email is what is reported in my backgroundrb_server_<port>.log file. The
> process_netflix_reviews method is parsing (in the background)  Netflix's RSS
> feeds. This error also pretty much brings down the BDRb process and to
> recover, I am forced to do a
>
> ./script/backgroundrb stop
> ./script/backgroundrb start
>
> Once I've done this, it chugs along fine - even for the same URI it crashed
> on last. Until at some point in the future it barfs on another URI and I
> need to restart BDRb... This has been happening on and off for several days
> now.
>
> Any pointers on what I am doing wrong?
>
> Much appreciated - thanks!
>
> Raghu
>
> ======================================================
> /usr/lib/ruby/1.8/open-uri.rb:287:in `open_http': 500 Internal Server Error
> (OpenURI::HTTPError)
>         from /usr/lib/ruby/1.8/open-uri.rb:626:in `buffer_open'
>         from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
>         from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
>         from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
>         from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
>         from /usr/lib/ruby/1.8/open-uri.rb:528:in `open'
>         from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
>         from /home/raghus/public_html/feedflix.com/ff/lib/netflix.rb:467:in
> `process_netflix_reviews'
>          ... 21 levels...
>         from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> `run'
>         from
> /home/raghus/public_html/feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
> `initialize'
>         from ./script/backgroundrb:42:in `new'
>         from ./script/backgroundrb:42
> ~
> ======================================================
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From atul.veer at betterlabs.net  Mon May 19 03:00:05 2008
From: atul.veer at betterlabs.net (Atul Veer - BetterLabs)
Date: Mon, 19 May 2008 12:30:05 +0530
Subject: [Backgroundrb-devel] uninitialized constant
	BackgrounDRb::MetaWorker (NameError)
Message-ID: <14c35d650805190000o4d69c1a7w53e0e3d30584b8f4@mail.gmail.com>

Hi All,

I am using backgroundrb to receive emails in my rails app. But I am unable
to start the Backgroundrb server. After doing this:

$ ./script/backgroundrb/start

I am getting following error:

???????????????????????????????
vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:263:in
`load_missing_constant': uninitialized constant BackgrounDRb::MetaWorker
(NameError)
from
/home/shashank/atul/live_apps/7_qlubb/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:452:in
`const_missing'
from
./script/backgroundrb/../../config/../lib/workers/mailreceiver_worker.rb:3

??????????????????????????-

Here is my worker: mailreceiver_worker.rb

-------------------------------------------------------------------------------------------

require 'rubygems'
require 'pop_ssl'

# Error is on following line
class MailreceiverWorker < BackgrounDRb::MetaWorker

set_worker_name :mailreceiver_worker

def create(args = nil)
# this method is called, when worker is loaded for the first time

end

def receive_emails

# I am receiving emails here. and using models here.
end
??????????????????????????????

I tried running some examples, and found that worker with do_work are
starting but those derived from MetaWorker are not.

I am using latest back-drb release.

Can anyone pl help me out of this?

Thanks in advance.

Atul.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080519/3f5453e5/attachment.html>

From raghu.srinivasan at gmail.com  Tue May 20 14:50:29 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Tue, 20 May 2008 11:50:29 -0700
Subject: [Backgroundrb-devel] Couple questions on BDRb and concurrent
	processing
Message-ID: <6482c06a0805201150p6a059779p40ee8796591e0e1d@mail.gmail.com>

My Rails site uses BackgroundDRb and I have a couple of questions:

1. Can someone point me to some sample code/examples for how to use
thread_pool? The website doc says to add a line

======
pool_size 10
======
in my Worker class which seems straightforward.

I wasn't able to understand this part though:
=========
thread_pool.defer(wiki_scrap_url) { |wiki_url| scrap_wikipedia(wiki_url) }
=========

Can someone explain this code please? And where does this code go? In my
Rails method that's actually kicking off the b/g process? Right now my
method looks likke this

======
....
            worker = MiddleMan.worker(:netflix_worker)
            result = worker.parse_netflix_feeds(passed_netflix_rss_code)
....
======
where parse_netflix_feeds is the method in my /lib/workers/netflix_worker.rb

2. My background process is initiated in two ways - one via a live user on
the web and another as a batch process on a scheduled basis. How do I
separate the two so that they don't wait for each other to complete? The b/g
process itself calls out to an RSS feed and inserts/updates data into a
MySQL DB. I don't want a live user's b/g job to wait for a batch process
that might be going through dozens of feeds.

Thanks!

Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080520/8984ab37/attachment-0001.html>

From mike at metaswitch.com  Tue May 20 15:30:51 2008
From: mike at metaswitch.com (Mike Evans)
Date: Tue, 20 May 2008 20:30:51 +0100
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
Message-ID: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>

I'm working on an application that does extensive database searching.
These searches can take a long time, so we have been working on moving
the searches to a backgroundrb worker task so we can provide a sexy AJAX
progress bar, and populate the search results as they are available.
All of this seems to work fine until the size of the search results gets
sufficiently large, when we start to hit exceptions in backgroundrb
(most likely in the packet layer).  We are using packet-0.5.1 and
backgroundrb from the latest svn mirror.

We have found and fixed one problem in the packet sender.  This is
triggered when the non-blocking send in NbioHelper::send_once cannot
send the entire buffer, resulting in an exception in the line 

      write_scheduled[fileno] ||= connections[fileno].instance

in Core::schedule_write because connections[fileno] is nil.  I can't
claim to fully understand the code, but I think there are two problems
here.

The main issue seems to be that when Core::handle_write_event calls
write_and_schedule to schedule the write, it doesn't clear out
internal_scheduled_write[fileno].  It looks like the code is expecting
the cancel_write call at the end of write_and_schedule to clear it out,
but this doesn't happen if there is enough queued data to cause the
non-blocking write to only partially succeed again.  In this case,
Core::schedule_write is called again, and because
internal_schedule_write[fileno] has not been cleared out, the code drops
through to the second if test, then hits the above exception.  We fixed
this by adding the line

	internal_scheduled_write.delete(fileno)

immediately before the call to write_and_schedule in
Core::handle_write_event.

The secondary issue is that the connections[fileno] structure is not
getting populated for this connection - I'm guessing because it is an
internal socket rather than a network socket, but I couldn't be sure.
We changed the second if test in Core::schedule_write to

      elsif write_scheduled[fileno].nil? && !connections[fileno].nil?

to firewall against this, but we are not sure if this is the right fix.

We are now hitting problems in the Packet::MetaPimp module receiving the
data, usually an exception in the Marshal.load call in
MetaPimp::receive_data.  We suspect this is caused by the packet code
corrupting the data somewhere, probably because we are sending such
large arrays of results (the repro I am working on at the moment is
trying to marshal over 200k of data).  We've been trying to put extra
diagnostics in the code so we can see what is happening, but if we edit
puts statements into the code we only seem to get output from the end of
the connection that hits an exception and so far our attempts to make
logger objects available throughout the code have failed.  We therefore
thought we would ask for help - either to see whether this is a known
problem, or whether there is a recommended way to add diagnostics to the
packet code.

I'm also open to ideas as to better ways to solve the problem!

Thanks in advance,

Mike



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080520/aad353ae/attachment.html>

From gethemant at gmail.com  Wed May 21 00:12:23 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 21 May 2008 09:42:23 +0530
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
In-Reply-To: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
Message-ID: <fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>

On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com> wrote:
> I'm working on an application that does extensive database searching.  These
> searches can take a long time, so we have been working on moving the
> searches to a backgroundrb worker task so we can provide a sexy AJAX
> progress bar, and populate the search results as they are available.  All of
> this seems to work fine until the size of the search results gets
> sufficiently large, when we start to hit exceptions in backgroundrb (most
> likely in the packet layer).  We are using packet-0.5.1 and backgroundrb
> from the latest svn mirror.
>
> We have found and fixed one problem in the packet sender.  This is triggered
> when the non-blocking send in NbioHelper::send_once cannot send the entire
> buffer, resulting in an exception in the line
>
>       write_scheduled[fileno] ||= connections[fileno].instance
>
> in Core::schedule_write because connections[fileno] is nil.  I can't claim
> to fully understand the code, but I think there are two problems here.
>
> The main issue seems to be that when Core::handle_write_event calls
> write_and_schedule to schedule the write, it doesn't clear out
> internal_scheduled_write[fileno].  It looks like the code is expecting the
> cancel_write call at the end of write_and_schedule to clear it out, but this
> doesn't happen if there is enough queued data to cause the non-blocking
> write to only partially succeed again.  In this case, Core::schedule_write
> is called again, and because internal_schedule_write[fileno] has not been
> cleared out, the code drops through to the second if test, then hits the
> above exception.  We fixed this by adding the line
>
> internal_scheduled_write.delete(fileno)
>
> immediately before the call to write_and_schedule in
> Core::handle_write_event.
>
> The secondary issue is that the connections[fileno] structure is not getting
> populated for this connection - I'm guessing because it is an internal
> socket rather than a network socket, but I couldn't be sure.  We changed the
> second if test in Core::schedule_write to
>
>       elsif write_scheduled[fileno].nil? && !connections[fileno].nil?
>
> to firewall against this, but we are not sure if this is the right fix.

Thats was surely a bug and I fixed it like this:

      def schedule_write(t_sock,internal_instance = nil)
        fileno = t_sock.fileno
        if UNIXSocket === t_sock && internal_scheduled_write[fileno].nil?
          write_ios << t_sock
          internal_scheduled_write[t_sock.fileno] ||= internal_instance
        elsif write_scheduled[fileno].nil? && !(t_sock.is_a?(UNIXSocket))
          write_ios << t_sock
          write_scheduled[fileno] ||= connections[fileno].instance
        end
      end

Also, I fixed issue with marshalling larger data across the channel.
Thanks for reporting this. I have been terribly busy with things in
office and personal life and hence my work on BackgrounDRb has been in
hiatus for a while. Unfortunately, you can't use trunk packet code
which is available from:

git clone git://github.com/gnufied/packet.git

directly with svn mirror of backgroundrb, since packet now uses fork
and exec to run workers and hence reducing memory usage of workers.
However in a day or two I will update git repository of BackgrounDRb
which makes use of latest packet version. In the meanwhile, you can
try backporting relevant packet changes to version you are using and
see if it fixes your problem.

>
> We are now hitting problems in the Packet::MetaPimp module receiving the
> data, usually an exception in the Marshal.load call in
> MetaPimp::receive_data.  We suspect this is caused by the packet code
> corrupting the data somewhere, probably because we are sending such large
> arrays of results (the repro I am working on at the moment is trying to
> marshal over 200k of data).  We've been trying to put extra diagnostics in
> the code so we can see what is happening, but if we edit puts statements
> into the code we only seem to get output from the end of the connection that
> hits an exception and so far our attempts to make logger objects available
> throughout the code have failed.  We therefore thought we would ask for help
> - either to see whether this is a known problem, or whether there is a
> recommended way to add diagnostics to the packet code.
>
> I'm also open to ideas as to better ways to solve the problem!
>
> Thanks in advance,
>
> Mike
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed May 21 00:36:00 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 21 May 2008 10:06:00 +0530
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
In-Reply-To: <fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
	<fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
Message-ID: <fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>

You can test git version of backgroundrb with git version of packet
(which incorporates latest changes). The procedure is as follows:

clone the packet git repo:

git clone git://github.com/gnufied/packet.git
cd packet;rake gem
cd pkg; sudo gem install --local packet-0.1.6.gem

Go to your vendor directory of your rails directory and remove or
backup older version of backgroundrb plugin and backup related config
file as well.

from vendor directory:

git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb
cd RAILS_ROOT
<<assuming older script and config file has been backed up>>
rake backgroundrb:setup
<<modify config/backgroundrb.yml according to your needs>>
./script/backgroundrb start
<<Let me know, how it goes and if this fixes your problem>>


On Wed, May 21, 2008 at 9:42 AM, hemant <gethemant at gmail.com> wrote:
> On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com> wrote:
>> I'm working on an application that does extensive database searching.  These
>> searches can take a long time, so we have been working on moving the
>> searches to a backgroundrb worker task so we can provide a sexy AJAX
>> progress bar, and populate the search results as they are available.  All of
>> this seems to work fine until the size of the search results gets
>> sufficiently large, when we start to hit exceptions in backgroundrb (most
>> likely in the packet layer).  We are using packet-0.5.1 and backgroundrb
>> from the latest svn mirror.
>>
>> We have found and fixed one problem in the packet sender.  This is triggered
>> when the non-blocking send in NbioHelper::send_once cannot send the entire
>> buffer, resulting in an exception in the line
>>
>>       write_scheduled[fileno] ||= connections[fileno].instance
>>
>> in Core::schedule_write because connections[fileno] is nil.  I can't claim
>> to fully understand the code, but I think there are two problems here.
>>
>> The main issue seems to be that when Core::handle_write_event calls
>> write_and_schedule to schedule the write, it doesn't clear out
>> internal_scheduled_write[fileno].  It looks like the code is expecting the
>> cancel_write call at the end of write_and_schedule to clear it out, but this
>> doesn't happen if there is enough queued data to cause the non-blocking
>> write to only partially succeed again.  In this case, Core::schedule_write
>> is called again, and because internal_schedule_write[fileno] has not been
>> cleared out, the code drops through to the second if test, then hits the
>> above exception.  We fixed this by adding the line
>>
>> internal_scheduled_write.delete(fileno)
>>
>> immediately before the call to write_and_schedule in
>> Core::handle_write_event.
>>
>> The secondary issue is that the connections[fileno] structure is not getting
>> populated for this connection - I'm guessing because it is an internal
>> socket rather than a network socket, but I couldn't be sure.  We changed the
>> second if test in Core::schedule_write to
>>
>>       elsif write_scheduled[fileno].nil? && !connections[fileno].nil?
>>
>> to firewall against this, but we are not sure if this is the right fix.
>
> Thats was surely a bug and I fixed it like this:
>
>      def schedule_write(t_sock,internal_instance = nil)
>        fileno = t_sock.fileno
>        if UNIXSocket === t_sock && internal_scheduled_write[fileno].nil?
>          write_ios << t_sock
>          internal_scheduled_write[t_sock.fileno] ||= internal_instance
>        elsif write_scheduled[fileno].nil? && !(t_sock.is_a?(UNIXSocket))
>          write_ios << t_sock
>          write_scheduled[fileno] ||= connections[fileno].instance
>        end
>      end
>
> Also, I fixed issue with marshalling larger data across the channel.
> Thanks for reporting this. I have been terribly busy with things in
> office and personal life and hence my work on BackgrounDRb has been in
> hiatus for a while. Unfortunately, you can't use trunk packet code
> which is available from:
>
> git clone git://github.com/gnufied/packet.git
>
> directly with svn mirror of backgroundrb, since packet now uses fork
> and exec to run workers and hence reducing memory usage of workers.
> However in a day or two I will update git repository of BackgrounDRb
> which makes use of latest packet version. In the meanwhile, you can
> try backporting relevant packet changes to version you are using and
> see if it fixes your problem.
>
>>
>> We are now hitting problems in the Packet::MetaPimp module receiving the
>> data, usually an exception in the Marshal.load call in
>> MetaPimp::receive_data.  We suspect this is caused by the packet code
>> corrupting the data somewhere, probably because we are sending such large
>> arrays of results (the repro I am working on at the moment is trying to
>> marshal over 200k of data).  We've been trying to put extra diagnostics in
>> the code so we can see what is happening, but if we edit puts statements
>> into the code we only seem to get output from the end of the connection that
>> hits an exception and so far our attempts to make logger objects available
>> throughout the code have failed.  We therefore thought we would ask for help
>> - either to see whether this is a known problem, or whether there is a
>> recommended way to add diagnostics to the packet code.
>>
>> I'm also open to ideas as to better ways to solve the problem!
>>
>> Thanks in advance,
>>
>> Mike
>>
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From saratkongara at gmail.com  Wed May 21 00:37:30 2008
From: saratkongara at gmail.com (Sarat Kongara)
Date: Wed, 21 May 2008 10:07:30 +0530
Subject: [Backgroundrb-devel] How to know the environment
	(development/test/production) inside a worker
Message-ID: <4c251b0d0805202137k1a5fe3cfr2b86cd67f69bc49c@mail.gmail.com>

Hi Guys,You can start backgroundRb in different environments using -e
option.

./script/backgroundrb -e production

I like to run a rake task from my worker - something like rake
RAILS_ENV=proper_environment thinking_sphinx:start

I am not sure how to figure out the environment with which the backgroundRb
is started from inside a worker task. Any help is appreciated. Thanks for
your time.

Regards
Sarat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/942a121b/attachment.html>

From mike at metaswitch.com  Wed May 21 03:14:18 2008
From: mike at metaswitch.com (Mike Evans)
Date: Wed, 21 May 2008 08:14:18 +0100
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
In-Reply-To: <fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
	<fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
	<fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>
Message-ID: <52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk>

Hemant

I got to the bottom of the other problem last night.  The issue was with
the NbioHelper::write_and_schedule method deleting entries from the
outbound_data array while iterating through it.  This can end up with
data getting out of order.  I fixed it by changing the
outbound_data.delete_at(index) statement to outbound_data[index] = nil,
and then compacting the array at the end of the iteration.

    # write the data in socket buffer and schedule the thing
    def write_and_schedule sock
      outbound_data.each_with_index do |t_data,index|
        leftover = write_once(t_data,sock)
        if leftover.empty?
          outbound_data[index] = nil
        else
          outbound_data[index] = leftover
          reactor.schedule_write(sock)
          break
        end
      end
      outbound_data.compact!
      reactor.cancel_write(sock) if outbound_data.empty?
    end

Mike 

-----Original Message-----
From: hemant [mailto:gethemant at gmail.com] 
Sent: 21 May 2008 05:36
To: Mike Evans
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Problems sending large results with
backgroundrb

You can test git version of backgroundrb with git version of packet
(which incorporates latest changes). The procedure is as follows:

clone the packet git repo:

git clone git://github.com/gnufied/packet.git
cd packet;rake gem
cd pkg; sudo gem install --local packet-0.1.6.gem

Go to your vendor directory of your rails directory and remove or backup
older version of backgroundrb plugin and backup related config file as
well.

from vendor directory:

git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb cd
RAILS_ROOT <<assuming older script and config file has been backed up>>
rake backgroundrb:setup <<modify config/backgroundrb.yml according to
your needs>> ./script/backgroundrb start <<Let me know, how it goes and
if this fixes your problem>>


On Wed, May 21, 2008 at 9:42 AM, hemant <gethemant at gmail.com> wrote:
> On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com>
wrote:
>> I'm working on an application that does extensive database searching.

>> These searches can take a long time, so we have been working on 
>> moving the searches to a backgroundrb worker task so we can provide a

>> sexy AJAX progress bar, and populate the search results as they are 
>> available.  All of this seems to work fine until the size of the 
>> search results gets sufficiently large, when we start to hit 
>> exceptions in backgroundrb (most likely in the packet layer).  We are

>> using packet-0.5.1 and backgroundrb from the latest svn mirror.
>>
>> We have found and fixed one problem in the packet sender.  This is 
>> triggered when the non-blocking send in NbioHelper::send_once cannot 
>> send the entire buffer, resulting in an exception in the line
>>
>>       write_scheduled[fileno] ||= connections[fileno].instance
>>
>> in Core::schedule_write because connections[fileno] is nil.  I can't 
>> claim to fully understand the code, but I think there are two
problems here.
>>
>> The main issue seems to be that when Core::handle_write_event calls 
>> write_and_schedule to schedule the write, it doesn't clear out 
>> internal_scheduled_write[fileno].  It looks like the code is 
>> expecting the cancel_write call at the end of write_and_schedule to 
>> clear it out, but this doesn't happen if there is enough queued data 
>> to cause the non-blocking write to only partially succeed again.  In 
>> this case, Core::schedule_write is called again, and because 
>> internal_schedule_write[fileno] has not been cleared out, the code 
>> drops through to the second if test, then hits the above exception.  
>> We fixed this by adding the line
>>
>> internal_scheduled_write.delete(fileno)
>>
>> immediately before the call to write_and_schedule in 
>> Core::handle_write_event.
>>
>> The secondary issue is that the connections[fileno] structure is not 
>> getting populated for this connection - I'm guessing because it is an

>> internal socket rather than a network socket, but I couldn't be sure.

>> We changed the second if test in Core::schedule_write to
>>
>>       elsif write_scheduled[fileno].nil? && !connections[fileno].nil?
>>
>> to firewall against this, but we are not sure if this is the right
fix.
>
> Thats was surely a bug and I fixed it like this:
>
>      def schedule_write(t_sock,internal_instance = nil)
>        fileno = t_sock.fileno
>        if UNIXSocket === t_sock &&
internal_scheduled_write[fileno].nil?
>          write_ios << t_sock
>          internal_scheduled_write[t_sock.fileno] ||= internal_instance
>        elsif write_scheduled[fileno].nil? &&
!(t_sock.is_a?(UNIXSocket))
>          write_ios << t_sock
>          write_scheduled[fileno] ||= connections[fileno].instance
>        end
>      end
>
> Also, I fixed issue with marshalling larger data across the channel.
> Thanks for reporting this. I have been terribly busy with things in 
> office and personal life and hence my work on BackgrounDRb has been in

> hiatus for a while. Unfortunately, you can't use trunk packet code 
> which is available from:
>
> git clone git://github.com/gnufied/packet.git
>
> directly with svn mirror of backgroundrb, since packet now uses fork 
> and exec to run workers and hence reducing memory usage of workers.
> However in a day or two I will update git repository of BackgrounDRb 
> which makes use of latest packet version. In the meanwhile, you can 
> try backporting relevant packet changes to version you are using and 
> see if it fixes your problem.
>
>>
>> We are now hitting problems in the Packet::MetaPimp module receiving 
>> the data, usually an exception in the Marshal.load call in 
>> MetaPimp::receive_data.  We suspect this is caused by the packet code

>> corrupting the data somewhere, probably because we are sending such 
>> large arrays of results (the repro I am working on at the moment is 
>> trying to marshal over 200k of data).  We've been trying to put extra

>> diagnostics in the code so we can see what is happening, but if we 
>> edit puts statements into the code we only seem to get output from 
>> the end of the connection that hits an exception and so far our 
>> attempts to make logger objects available throughout the code have 
>> failed.  We therefore thought we would ask for help
>> - either to see whether this is a known problem, or whether there is 
>> a recommended way to add diagnostics to the packet code.
>>
>> I'm also open to ideas as to better ways to solve the problem!
>>
>> Thanks in advance,
>>
>> Mike
>>
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting 
> conservatories; give me the privilege of making my own summer with my 
> own coals.
>
> http://gnufied.org
>



--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed May 21 03:56:16 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 21 May 2008 13:26:16 +0530
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
In-Reply-To: <52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
	<fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
	<fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>
	<52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk>
Message-ID: <fb283d5e0805210056y5792f134u7f3a55769ddd3e1d@mail.gmail.com>

Yeah that too. But I wonder, how did you solve following two problems:

Take a look at this code:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && internal_scheduled_write[fileno]
# we have a problem here
            write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno]
# I was drunk while writing following line
            handler_instance.write_scheduled(sock_fd)
          end
        end
      end

The problem is, as you said say in a MetaPimp some data is left
unwritten, it won't get written in subsequent writes because
outbound_data belongs to MetaPimp class not main reactor class and
hence, it should be:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && (internal_instance =
internal_scheduled_write[fileno])
            internal_instance.write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno]
            handler_instance.write_and_schedule(sock_fd)
          end
        end
      end

Also, I have included your changes in packet git. So, if you can give
backgroundrb git a shot, I will appreciate that ( Please backup your
older plugin and config files)

On Wed, May 21, 2008 at 12:44 PM, Mike Evans <mike at metaswitch.com> wrote:
> Hemant
>
> I got to the bottom of the other problem last night.  The issue was with
> the NbioHelper::write_and_schedule method deleting entries from the
> outbound_data array while iterating through it.  This can end up with
> data getting out of order.  I fixed it by changing the
> outbound_data.delete_at(index) statement to outbound_data[index] = nil,
> and then compacting the array at the end of the iteration.
>
>    # write the data in socket buffer and schedule the thing
>    def write_and_schedule sock
>      outbound_data.each_with_index do |t_data,index|
>        leftover = write_once(t_data,sock)
>        if leftover.empty?
>          outbound_data[index] = nil
>        else
>          outbound_data[index] = leftover
>          reactor.schedule_write(sock)
>          break
>        end
>      end
>      outbound_data.compact!
>      reactor.cancel_write(sock) if outbound_data.empty?
>    end
>
> Mike
>
> -----Original Message-----
> From: hemant [mailto:gethemant at gmail.com]
> Sent: 21 May 2008 05:36
> To: Mike Evans
> Cc: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Problems sending large results with
> backgroundrb
>
> You can test git version of backgroundrb with git version of packet
> (which incorporates latest changes). The procedure is as follows:
>
> clone the packet git repo:
>
> git clone git://github.com/gnufied/packet.git
> cd packet;rake gem
> cd pkg; sudo gem install --local packet-0.1.6.gem
>
> Go to your vendor directory of your rails directory and remove or backup
> older version of backgroundrb plugin and backup related config file as
> well.
>
> from vendor directory:
>
> git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb cd
> RAILS_ROOT <<assuming older script and config file has been backed up>>
> rake backgroundrb:setup <<modify config/backgroundrb.yml according to
> your needs>> ./script/backgroundrb start <<Let me know, how it goes and
> if this fixes your problem>>
>
>
> On Wed, May 21, 2008 at 9:42 AM, hemant <gethemant at gmail.com> wrote:
>> On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com>
> wrote:
>>> I'm working on an application that does extensive database searching.
>
>>> These searches can take a long time, so we have been working on
>>> moving the searches to a backgroundrb worker task so we can provide a
>
>>> sexy AJAX progress bar, and populate the search results as they are
>>> available.  All of this seems to work fine until the size of the
>>> search results gets sufficiently large, when we start to hit
>>> exceptions in backgroundrb (most likely in the packet layer).  We are
>
>>> using packet-0.5.1 and backgroundrb from the latest svn mirror.
>>>
>>> We have found and fixed one problem in the packet sender.  This is
>>> triggered when the non-blocking send in NbioHelper::send_once cannot
>>> send the entire buffer, resulting in an exception in the line
>>>
>>>       write_scheduled[fileno] ||= connections[fileno].instance
>>>
>>> in Core::schedule_write because connections[fileno] is nil.  I can't
>>> claim to fully understand the code, but I think there are two
> problems here.
>>>
>>> The main issue seems to be that when Core::handle_write_event calls
>>> write_and_schedule to schedule the write, it doesn't clear out
>>> internal_scheduled_write[fileno].  It looks like the code is
>>> expecting the cancel_write call at the end of write_and_schedule to
>>> clear it out, but this doesn't happen if there is enough queued data
>>> to cause the non-blocking write to only partially succeed again.  In
>>> this case, Core::schedule_write is called again, and because
>>> internal_schedule_write[fileno] has not been cleared out, the code
>>> drops through to the second if test, then hits the above exception.
>>> We fixed this by adding the line
>>>
>>> internal_scheduled_write.delete(fileno)
>>>
>>> immediately before the call to write_and_schedule in
>>> Core::handle_write_event.
>>>
>>> The secondary issue is that the connections[fileno] structure is not
>>> getting populated for this connection - I'm guessing because it is an
>
>>> internal socket rather than a network socket, but I couldn't be sure.
>
>>> We changed the second if test in Core::schedule_write to
>>>
>>>       elsif write_scheduled[fileno].nil? && !connections[fileno].nil?
>>>
>>> to firewall against this, but we are not sure if this is the right
> fix.
>>
>> Thats was surely a bug and I fixed it like this:
>>
>>      def schedule_write(t_sock,internal_instance = nil)
>>        fileno = t_sock.fileno
>>        if UNIXSocket === t_sock &&
> internal_scheduled_write[fileno].nil?
>>          write_ios << t_sock
>>          internal_scheduled_write[t_sock.fileno] ||= internal_instance
>>        elsif write_scheduled[fileno].nil? &&
> !(t_sock.is_a?(UNIXSocket))
>>          write_ios << t_sock
>>          write_scheduled[fileno] ||= connections[fileno].instance
>>        end
>>      end
>>
>> Also, I fixed issue with marshalling larger data across the channel.
>> Thanks for reporting this. I have been terribly busy with things in
>> office and personal life and hence my work on BackgrounDRb has been in
>
>> hiatus for a while. Unfortunately, you can't use trunk packet code
>> which is available from:
>>
>> git clone git://github.com/gnufied/packet.git
>>
>> directly with svn mirror of backgroundrb, since packet now uses fork
>> and exec to run workers and hence reducing memory usage of workers.
>> However in a day or two I will update git repository of BackgrounDRb
>> which makes use of latest packet version. In the meanwhile, you can
>> try backporting relevant packet changes to version you are using and
>> see if it fixes your problem.
>>
>>>
>>> We are now hitting problems in the Packet::MetaPimp module receiving
>>> the data, usually an exception in the Marshal.load call in
>>> MetaPimp::receive_data.  We suspect this is caused by the packet code
>
>>> corrupting the data somewhere, probably because we are sending such
>>> large arrays of results (the repro I am working on at the moment is
>>> trying to marshal over 200k of data).  We've been trying to put extra
>
>>> diagnostics in the code so we can see what is happening, but if we
>>> edit puts statements into the code we only seem to get output from
>>> the end of the connection that hits an exception and so far our
>>> attempts to make logger objects available throughout the code have
>>> failed.  We therefore thought we would ask for help
>>> - either to see whether this is a known problem, or whether there is
>>> a recommended way to add diagnostics to the packet code.
>>>
>>> I'm also open to ideas as to better ways to solve the problem!
>>>
>>> Thanks in advance,
>>>
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>>
>> --
>> Let them talk of their oriental summer climes of everlasting
>> conservatories; give me the privilege of making my own summer with my
>> own coals.
>>
>> http://gnufied.org
>>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed May 21 04:01:12 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 21 May 2008 13:31:12 +0530
Subject: [Backgroundrb-devel] Couple questions on BDRb and concurrent
	processing
In-Reply-To: <6482c06a0805201150p6a059779p40ee8796591e0e1d@mail.gmail.com>
References: <6482c06a0805201150p6a059779p40ee8796591e0e1d@mail.gmail.com>
Message-ID: <fb283d5e0805210101p5704dcd9wc978a34734ba6b78@mail.gmail.com>

On Wed, May 21, 2008 at 12:20 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> My Rails site uses BackgroundDRb and I have a couple of questions:
>
> 1. Can someone point me to some sample code/examples for how to use
> thread_pool? The website doc says to add a line
>
> ======
> pool_size 10
> ======
> in my Worker class which seems straightforward.
>
> I wasn't able to understand this part though:
> =========
> thread_pool.defer(wiki_scrap_url) { |wiki_url| scrap_wikipedia(wiki_url) }
> =========
>
> Can someone explain this code please? And where does this code go? In my
> Rails method that's actually kicking off the b/g process? Right now my
> method looks likke this

Above code goes in your worker, if you don't want worker to be tied up
while doing some processing.

>
> ======
> ....
>             worker = MiddleMan.worker(:netflix_worker)
>             result = worker.parse_netflix_feeds(passed_netflix_rss_code)
> ....
> ======
> where parse_netflix_feeds is the method in my /lib/workers/netflix_worker.rb
>
> 2. My background process is initiated in two ways - one via a live user on
> the web and another as a batch process on a scheduled basis. How do I
> separate the two so that they don't wait for each other to complete? The b/g
> process itself calls out to an RSS feed and inserts/updates data into a
> MySQL DB. I don't want a live user's b/g job to wait for a batch process
> that might be going through dozens of feeds.
>

One solution is to use one worker for running scheduled task and
another worker for running direct requests. Another solution is to
use, thread_pool for direct requests.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From glenn.gillen at gmail.com  Wed May 21 06:59:11 2008
From: glenn.gillen at gmail.com (Glenn Gillen)
Date: Wed, 21 May 2008 11:59:11 +0100
Subject: [Backgroundrb-devel] Queuing in backgroundrb, is it possible?
Message-ID: <69b2ac8a0805210359g7001b5e1x4f7e7bf325725cc5@mail.gmail.com>

Hey all,

I've been playing with backgroundrb for a while but I've never had to
worry about this particular scenario.

We have a process that can take a while (<30secs) to return results so
it needs to be moved off of the standard rails stack and into
backgroundrb land (easy, and essentially done). However, we can't
tolerate the possibility that any more than 5 concurrent requests are
sent to this service, is there a way to have backgroundrb throttle
concurrency and queue and back up any further jobs until a slot is
available?

Thanks,

-- 
Glenn

From gethemant at gmail.com  Wed May 21 07:13:59 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 21 May 2008 16:43:59 +0530
Subject: [Backgroundrb-devel] Queuing in backgroundrb, is it possible?
In-Reply-To: <69b2ac8a0805210359g7001b5e1x4f7e7bf325725cc5@mail.gmail.com>
References: <69b2ac8a0805210359g7001b5e1x4f7e7bf325725cc5@mail.gmail.com>
Message-ID: <fb283d5e0805210413w36bc955k985f2657ebb830e9@mail.gmail.com>

On Wed, May 21, 2008 at 4:29 PM, Glenn Gillen <glenn.gillen at gmail.com> wrote:
> Hey all,
>
> I've been playing with backgroundrb for a while but I've never had to
> worry about this particular scenario.
>
> We have a process that can take a while (<30secs) to return results so
> it needs to be moved off of the standard rails stack and into
> backgroundrb land (easy, and essentially done). However, we can't
> tolerate the possibility that any more than 5 concurrent requests are
> sent to this service, is there a way to have backgroundrb throttle
> concurrency and queue and back up any further jobs until a slot is
> available?
>

If you are using inbuilt thread pool, you can pretty much easily do
that. To avoid 'throttling' you can reduce thread_pool size.

From joel.chippindale at gmail.com  Wed May 21 11:49:22 2008
From: joel.chippindale at gmail.com (Joel Chippindale)
Date: Wed, 21 May 2008 16:49:22 +0100
Subject: [Backgroundrb-devel]  Schedule write errors?
Message-ID: <4c647b520805210849x87de511p64b4978a3641965c@mail.gmail.com>

My BackgrounDRb server intermittently bails out with the following exception
in the middle of a long running task running on a Worker sub classed
from BackgrounDRb::MetaWorker (the point at which it raises this exception
is different each time it runs)
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
`schedule_write': You have a nil object when you didn't expect it!
(NoMethodError)
The error occurred while evaluating nil.instance from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
`write_and_schedule'
from
/Users/joel/git/deathstar/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/whiny_nil.rb:35:in
`each_with_index'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each_with_index'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`write_and_schedule'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
`dump_object'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:31:in
`send_data'
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:40:in
`send_request'
 ... 23 levels...
from
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
from
/Users/joel/git/deathstar/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
`initialize'
from script/backgroundrb:60:in `new'
from script/backgroundrb:60

I am running BackgrounDRb (r 324 from the subversion repository), with
packet v 0.1.5 on OS X.

Any suggestions for resolving this problem are welcomed.

J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/23f0b11c/attachment.html>

From gethemant at gmail.com  Wed May 21 15:16:56 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 22 May 2008 00:46:56 +0530
Subject: [Backgroundrb-devel] Schedule write errors?
In-Reply-To: <4c647b520805210849x87de511p64b4978a3641965c@mail.gmail.com>
References: <4c647b520805210849x87de511p64b4978a3641965c@mail.gmail.com>
Message-ID: <fb283d5e0805211216v5e52763dr7da46697e263c5f0@mail.gmail.com>

Look in the mail previous mail by Mike also see my response. You can
try out git versions of packet and backgroundrb and see if this fixes
your problem.

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/


On Wed, May 21, 2008 at 9:19 PM, Joel Chippindale
<joel.chippindale at gmail.com> wrote:
> My BackgrounDRb server intermittently bails out with the following exception
> in the middle of a long running task running on a Worker sub classed
> from BackgrounDRb::MetaWorker (the point at which it raises this exception
> is different each time it runs)
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
> `schedule_write': You have a nil object when you didn't expect it!
> (NoMethodError)
> The error occurred while evaluating nil.instance from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
> `write_and_schedule'
> from
> /Users/joel/git/deathstar/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/whiny_nil.rb:35:in
> `each_with_index'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> `each'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> `each_with_index'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> `write_and_schedule'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
> `dump_object'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:31:in
> `send_data'
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:40:in
> `send_request'
> ... 23 levels...
> from
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> `run'
> from
> /Users/joel/git/deathstar/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
> `initialize'
> from script/backgroundrb:60:in `new'
> from script/backgroundrb:60
> I am running BackgrounDRb (r 324 from the subversion repository), with
> packet v 0.1.5 on OS X.
> Any suggestions for resolving this problem are welcomed.
> J.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From salinas.j at gmail.com  Wed May 21 18:18:54 2008
From: salinas.j at gmail.com (Jim Salinas)
Date: Wed, 21 May 2008 17:18:54 -0500
Subject: [Backgroundrb-devel] Does backgroundrb support SSL?
Message-ID: <834fa2620805211518k568156cj3b3975538c9bd8d6@mail.gmail.com>

Does backgroundrb support SSL like drb? I was unable to find any sample
configurations including SSL. Any help would be appreciated, thanks. -jim
salinas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/9588825a/attachment.html>

From salinas.j at gmail.com  Wed May 21 18:58:16 2008
From: salinas.j at gmail.com (Jim Salinas)
Date: Wed, 21 May 2008 17:58:16 -0500
Subject: [Backgroundrb-devel] Does backgroundrb support SSL?
In-Reply-To: <1abf6d300805211530r48905a43n38a8a472817bfa92@mail.gmail.com>
References: <834fa2620805211518k568156cj3b3975538c9bd8d6@mail.gmail.com>
	<1abf6d300805211530r48905a43n38a8a472817bfa92@mail.gmail.com>
Message-ID: <834fa2620805211558x7c441bclbc358e3a4f51f544@mail.gmail.com>

Thanks for replay David, I just did a search and only found a mention of drb
in a unit test and from digging around backgroundrb seems to have been
rewritten recently without drb. If this incorrect, someone please correct
me. :-)

On Wed, May 21, 2008 at 5:30 PM, David Masover <dave at 3mix.com> wrote:

> I asked this question almost a month ago, and got no answer. backgroundrb
> does use drb as a backend, but I see no way to configure SSL, or Unix
> sockets, even though drb supports both.
>
> On Wed, May 21, 2008 at 5:18 PM, Jim Salinas <salinas.j at gmail.com> wrote:
>
>> Does backgroundrb support SSL like drb? I was unable to find any sample
>> configurations including SSL. Any help would be appreciated, thanks. -jim
>> salinas
>>
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/723b3696/attachment.html>

From leavengood at gmail.com  Thu May 22 01:05:23 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Thu, 22 May 2008 01:05:23 -0400
Subject: [Backgroundrb-devel] How to know the environment
	(development/test/production) inside a worker
In-Reply-To: <4c251b0d0805202137k1a5fe3cfr2b86cd67f69bc49c@mail.gmail.com>
References: <4c251b0d0805202137k1a5fe3cfr2b86cd67f69bc49c@mail.gmail.com>
Message-ID: <a4855ac50805212205l4c337ffdl19eb0594bcd3c731@mail.gmail.com>

On 5/21/08, Sarat Kongara <saratkongara at gmail.com> wrote:
>
> I am not sure how to figure out the environment with which the backgroundRb
> is started from inside a worker task. Any help is appreciated. Thanks for
> your time.

ENV["RAILS_ENV"] is defined properly inside the workers.

Ryan

From leavengood at gmail.com  Thu May 22 01:14:22 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Thu, 22 May 2008 01:14:22 -0400
Subject: [Backgroundrb-devel] Does backgroundrb support SSL?
In-Reply-To: <834fa2620805211558x7c441bclbc358e3a4f51f544@mail.gmail.com>
References: <834fa2620805211518k568156cj3b3975538c9bd8d6@mail.gmail.com>
	<1abf6d300805211530r48905a43n38a8a472817bfa92@mail.gmail.com>
	<834fa2620805211558x7c441bclbc358e3a4f51f544@mail.gmail.com>
Message-ID: <a4855ac50805212214j437094a4j9f24ba3382db90b0@mail.gmail.com>

On 5/21/08, Jim Salinas <salinas.j at gmail.com> wrote:
> Thanks for replay David, I just did a search and only found a mention of drb
> in a unit test and from digging around backgroundrb seems to have been
> rewritten recently without drb. If this incorrect, someone please correct
> me. :-)

BackgrounDRb has not used DRb for sometime, the name is just a bit
misleading now ;)

It was switched to use a event framework a while ago and more recently
has switched to the pure Ruby 'packet' library.

As far as I can tell packet does not support SSL, but I am far from an
expert on the subject. I assume you have some sensitive data you need
to pass to BackgrounDRb from Rails that you don't want in the clear?

Maybe you could just use the database to pass that information, only
passing database IDs or unique tokens to BackgrounDRb for it to load
the information. Of course depending on your database the information
may still get passed around in the clear...

Ryan

From joel.chippindale at gmail.com  Thu May 22 07:08:57 2008
From: joel.chippindale at gmail.com (Joel Chippindale)
Date: Thu, 22 May 2008 12:08:57 +0100
Subject: [Backgroundrb-devel] Schedule write errors?
In-Reply-To: <fb283d5e0805211216v5e52763dr7da46697e263c5f0@mail.gmail.com>
References: <4c647b520805210849x87de511p64b4978a3641965c@mail.gmail.com>
	<fb283d5e0805211216v5e52763dr7da46697e263c5f0@mail.gmail.com>
Message-ID: <4c647b520805220408x56fced59saab5bd325146f745@mail.gmail.com>

Thanks very much for this, it seems to have resolved the problem very
nicely.
J.
2008/5/21 hemant <gethemant at gmail.com>:

> Look in the mail previous mail by Mike also see my response. You can
> try out git versions of packet and backgroundrb and see if this fixes
> your problem.
>
>
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>
>
> On Wed, May 21, 2008 at 9:19 PM, Joel Chippindale
> <joel.chippindale at gmail.com> wrote:
> > My BackgrounDRb server intermittently bails out with the following
> exception
> > in the middle of a long running task running on a Worker sub classed
> > from BackgrounDRb::MetaWorker (the point at which it raises this
> exception
> > is different each time it runs)
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
> > `schedule_write': You have a nil object when you didn't expect it!
> > (NoMethodError)
> > The error occurred while evaluating nil.instance from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
> > `write_and_schedule'
> > from
> >
> /Users/joel/git/deathstar/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/whiny_nil.rb:35:in
> > `each_with_index'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> > `each'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> > `each_with_index'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
> > `write_and_schedule'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
> > `dump_object'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:31:in
> > `send_data'
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:40:in
> > `send_request'
> > ... 23 levels...
> > from
> >
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> > `run'
> > from
> >
> /Users/joel/git/deathstar/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
> > `initialize'
> > from script/backgroundrb:60:in `new'
> > from script/backgroundrb:60
> > I am running BackgrounDRb (r 324 from the subversion repository), with
> > packet v 0.1.5 on OS X.
> > Any suggestions for resolving this problem are welcomed.
> > J.
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080522/0de0774c/attachment.html>

From salinas.j at gmail.com  Thu May 22 09:34:37 2008
From: salinas.j at gmail.com (Jim Salinas)
Date: Thu, 22 May 2008 08:34:37 -0500
Subject: [Backgroundrb-devel] Does backgroundrb support SSL?
In-Reply-To: <a4855ac50805212214j437094a4j9f24ba3382db90b0@mail.gmail.com>
References: <834fa2620805211518k568156cj3b3975538c9bd8d6@mail.gmail.com>
	<1abf6d300805211530r48905a43n38a8a472817bfa92@mail.gmail.com>
	<834fa2620805211558x7c441bclbc358e3a4f51f544@mail.gmail.com>
	<a4855ac50805212214j437094a4j9f24ba3382db90b0@mail.gmail.com>
Message-ID: <834fa2620805220634q6f0284e2y9c4ad575868451c3@mail.gmail.com>

I planned on pushing only database record IDs to the server, but maybe I
could encrypt/decrypt those IDs when sending them over the wire. From
looking at the server source, it is utilizing TCPSocket, so there is an
opportunity I believe to layer SSL. Given some time, I'll investigate this
further, as this seems to be a better solution in the long run in my
opinion. -Jim


On Thu, May 22, 2008 at 12:14 AM, Ryan Leavengood <leavengood at gmail.com>
wrote:

> On 5/21/08, Jim Salinas <salinas.j at gmail.com> wrote:
> > Thanks for replay David, I just did a search and only found a mention of
> drb
> > in a unit test and from digging around backgroundrb seems to have been
> > rewritten recently without drb. If this incorrect, someone please correct
> > me. :-)
>
> BackgrounDRb has not used DRb for sometime, the name is just a bit
> misleading now ;)
>
> It was switched to use a event framework a while ago and more recently
> has switched to the pure Ruby 'packet' library.
>
> As far as I can tell packet does not support SSL, but I am far from an
> expert on the subject. I assume you have some sensitive data you need
> to pass to BackgrounDRb from Rails that you don't want in the clear?
>
> Maybe you could just use the database to pass that information, only
> passing database IDs or unique tokens to BackgrounDRb for it to load
> the information. Of course depending on your database the information
> may still get passed around in the clear...
>
> Ryan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080522/dc11844f/attachment-0001.html>

From bobby.santiago at gmail.com  Thu May 22 22:21:31 2008
From: bobby.santiago at gmail.com (Bobby Santiago)
Date: Fri, 23 May 2008 10:21:31 +0800
Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 24, Issue 7
In-Reply-To: <mailman.199.1211463279.13906.backgroundrb-devel@rubyforge.org>
References: <mailman.199.1211463279.13906.backgroundrb-devel@rubyforge.org>
Message-ID: <8930D793-F236-4FE3-933F-06937D727027@gmail.com>

Hi, all. Just wondering if anyone has tried using the Paperclip file  
upload plugin with Backgroundrb. My current implementation uses  
attachment_fu - I save the file initially to the file system without  
generating thumbs, then a bdrb process picks up that file, generates  
thumbs, and saves to S3. This works fine and dandy, the only hitch is  
I want to save image metadata in the same table as my model (User, in  
this case), which is how Paperclip does it. My question is, how would  
you pass off file uploading/thumbnail generation/S3 upload to a  
background process using Paperclip?

Bobby


On May 22, 2008, at 9:34 PM, backgroundrb-devel-request at rubyforge.org  
wrote:

> Send Backgroundrb-devel mailing list submissions to
> 	backgroundrb-devel at rubyforge.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> or, via email, send a message with subject or body 'help' to
> 	backgroundrb-devel-request at rubyforge.org
>
> You can reach the person managing the list at
> 	backgroundrb-devel-owner at rubyforge.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Backgroundrb-devel digest..."
>
>
> Today's Topics:
>
>   1. Does backgroundrb support SSL? (Jim Salinas)
>   2. Re: Does backgroundrb support SSL? (Jim Salinas)
>   3. Re: How to know the environment	(development/test/production)
>      inside a worker (Ryan Leavengood)
>   4. Re: Does backgroundrb support SSL? (Ryan Leavengood)
>   5. Re: Schedule write errors? (Joel Chippindale)
>   6. Re: Does backgroundrb support SSL? (Jim Salinas)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 21 May 2008 17:18:54 -0500
> From: "Jim Salinas" <salinas.j at gmail.com>
> Subject: [Backgroundrb-devel] Does backgroundrb support SSL?
> To: backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<834fa2620805211518k568156cj3b3975538c9bd8d6 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Does backgroundrb support SSL like drb? I was unable to find any  
> sample
> configurations including SSL. Any help would be appreciated, thanks.  
> -jim
> salinas
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/9588825a/attachment-0001.html 
> >
>
> ------------------------------
>
> Message: 2
> Date: Wed, 21 May 2008 17:58:16 -0500
> From: "Jim Salinas" <salinas.j at gmail.com>
> Subject: Re: [Backgroundrb-devel] Does backgroundrb support SSL?
> To: "David Masover" <dave at 3mix.com>, backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<834fa2620805211558x7c441bclbc358e3a4f51f544 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Thanks for replay David, I just did a search and only found a  
> mention of drb
> in a unit test and from digging around backgroundrb seems to have been
> rewritten recently without drb. If this incorrect, someone please  
> correct
> me. :-)
>
> On Wed, May 21, 2008 at 5:30 PM, David Masover <dave at 3mix.com> wrote:
>
>> I asked this question almost a month ago, and got no answer.  
>> backgroundrb
>> does use drb as a backend, but I see no way to configure SSL, or Unix
>> sockets, even though drb supports both.
>>
>> On Wed, May 21, 2008 at 5:18 PM, Jim Salinas <salinas.j at gmail.com>  
>> wrote:
>>
>>> Does backgroundrb support SSL like drb? I was unable to find any  
>>> sample
>>> configurations including SSL. Any help would be appreciated,  
>>> thanks. -jim
>>> salinas
>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080521/723b3696/attachment-0001.html 
> >
>
> ------------------------------
>
> Message: 3
> Date: Thu, 22 May 2008 01:05:23 -0400
> From: "Ryan Leavengood" <leavengood at gmail.com>
> Subject: Re: [Backgroundrb-devel] How to know the environment
> 	(development/test/production) inside a worker
> To: "Sarat Kongara" <saratkongara at gmail.com>
> Cc: backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<a4855ac50805212205l4c337ffdl19eb0594bcd3c731 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 5/21/08, Sarat Kongara <saratkongara at gmail.com> wrote:
>>
>> I am not sure how to figure out the environment with which the  
>> backgroundRb
>> is started from inside a worker task. Any help is appreciated.  
>> Thanks for
>> your time.
>
> ENV["RAILS_ENV"] is defined properly inside the workers.
>
> Ryan
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 22 May 2008 01:14:22 -0400
> From: "Ryan Leavengood" <leavengood at gmail.com>
> Subject: Re: [Backgroundrb-devel] Does backgroundrb support SSL?
> To: "Jim Salinas" <salinas.j at gmail.com>
> Cc: backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<a4855ac50805212214j437094a4j9f24ba3382db90b0 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On 5/21/08, Jim Salinas <salinas.j at gmail.com> wrote:
>> Thanks for replay David, I just did a search and only found a  
>> mention of drb
>> in a unit test and from digging around backgroundrb seems to have  
>> been
>> rewritten recently without drb. If this incorrect, someone please  
>> correct
>> me. :-)
>
> BackgrounDRb has not used DRb for sometime, the name is just a bit
> misleading now ;)
>
> It was switched to use a event framework a while ago and more recently
> has switched to the pure Ruby 'packet' library.
>
> As far as I can tell packet does not support SSL, but I am far from an
> expert on the subject. I assume you have some sensitive data you need
> to pass to BackgrounDRb from Rails that you don't want in the clear?
>
> Maybe you could just use the database to pass that information, only
> passing database IDs or unique tokens to BackgrounDRb for it to load
> the information. Of course depending on your database the information
> may still get passed around in the clear...
>
> Ryan
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 22 May 2008 12:08:57 +0100
> From: "Joel Chippindale" <joel.chippindale at gmail.com>
> Subject: Re: [Backgroundrb-devel] Schedule write errors?
> To: hemant <gethemant at gmail.com>
> Cc: backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<4c647b520805220408x56fced59saab5bd325146f745 at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Thanks very much for this, it seems to have resolved the problem very
> nicely.
> J.
> 2008/5/21 hemant <gethemant at gmail.com>:
>
>> Look in the mail previous mail by Mike also see my response. You can
>> try out git versions of packet and backgroundrb and see if this fixes
>> your problem.
>>
>>
>> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>>
>>
>> On Wed, May 21, 2008 at 9:19 PM, Joel Chippindale
>> <joel.chippindale at gmail.com> wrote:
>>> My BackgrounDRb server intermittently bails out with the following
>> exception
>>> in the middle of a long running task running on a Worker sub classed
>>> from BackgrounDRb::MetaWorker (the point at which it raises this
>> exception
>>> is different each time it runs)
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_core.rb:145:in
>>> `schedule_write': You have a nil object when you didn't expect it!
>>> (NoMethodError)
>>> The error occurred while evaluating nil.instance from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_nbio.rb:57:in
>>> `write_and_schedule'
>>> from
>>>
>> /Users/joel/git/deathstar/vendor/rails/activerecord/lib/../../ 
>> activesupport/lib/active_support/whiny_nil.rb:35:in
>>> `each_with_index'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_nbio.rb:51:in
>>> `each'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_nbio.rb:51:in
>>> `each_with_index'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_nbio.rb:51:in
>>> `write_and_schedule'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_nbio.rb:72:in
>>> `dump_object'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_worker.rb:31:in
>>> `send_data'
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_worker.rb:40:in
>>> `send_request'
>>> ... 23 levels...
>>> from
>>>
>> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ 
>> packet_master.rb:20:in
>>> `run'
>>> from
>>>
>> /Users/joel/git/deathstar/vendor/plugins/backgroundrb/server/lib/ 
>> master_worker.rb:166:in
>>> `initialize'
>>> from script/backgroundrb:60:in `new'
>>> from script/backgroundrb:60
>>> I am running BackgrounDRb (r 324 from the subversion repository),  
>>> with
>>> packet v 0.1.5 on OS X.
>>> Any suggestions for resolving this problem are welcomed.
>>> J.
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>>
>> --
>> Let them talk of their oriental summer climes of everlasting
>> conservatories; give me the privilege of making my own summer with my
>> own coals.
>>
>> http://gnufied.org
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080522/0de0774c/attachment-0001.html 
> >
>
> ------------------------------
>
> Message: 6
> Date: Thu, 22 May 2008 08:34:37 -0500
> From: "Jim Salinas" <salinas.j at gmail.com>
> Subject: Re: [Backgroundrb-devel] Does backgroundrb support SSL?
> To: "Ryan Leavengood" <leavengood at gmail.com>
> Cc: backgroundrb-devel at rubyforge.org
> Message-ID:
> 	<834fa2620805220634q6f0284e2y9c4ad575868451c3 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I planned on pushing only database record IDs to the server, but  
> maybe I
> could encrypt/decrypt those IDs when sending them over the wire. From
> looking at the server source, it is utilizing TCPSocket, so there is  
> an
> opportunity I believe to layer SSL. Given some time, I'll  
> investigate this
> further, as this seems to be a better solution in the long run in my
> opinion. -Jim
>
>
> On Thu, May 22, 2008 at 12:14 AM, Ryan Leavengood <leavengood at gmail.com 
> >
> wrote:
>
>> On 5/21/08, Jim Salinas <salinas.j at gmail.com> wrote:
>>> Thanks for replay David, I just did a search and only found a  
>>> mention of
>> drb
>>> in a unit test and from digging around backgroundrb seems to have  
>>> been
>>> rewritten recently without drb. If this incorrect, someone please  
>>> correct
>>> me. :-)
>>
>> BackgrounDRb has not used DRb for sometime, the name is just a bit
>> misleading now ;)
>>
>> It was switched to use a event framework a while ago and more  
>> recently
>> has switched to the pure Ruby 'packet' library.
>>
>> As far as I can tell packet does not support SSL, but I am far from  
>> an
>> expert on the subject. I assume you have some sensitive data you need
>> to pass to BackgrounDRb from Rails that you don't want in the clear?
>>
>> Maybe you could just use the database to pass that information, only
>> passing database IDs or unique tokens to BackgrounDRb for it to load
>> the information. Of course depending on your database the information
>> may still get passed around in the clear...
>>
>> Ryan
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080522/dc11844f/attachment.html 
> >
>
> ------------------------------
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> End of Backgroundrb-devel Digest, Vol 24, Issue 7
> *************************************************


From ruben.grunge84 at gmail.com  Fri May 23 17:00:39 2008
From: ruben.grunge84 at gmail.com (Ruben. D.)
Date: Fri, 23 May 2008 16:00:39 -0500
Subject: [Backgroundrb-devel] Strange problem with ActiveRecord#find in
	worker
Message-ID: <6bcc63d50805231400p470de508lea422d8c8d6ac133@mail.gmail.com>

Hi friends, I've a strange problem with ActiveRecord#find in a worker, this
sometimes work fine and somethimes return Nil but the SQL query is fine, I
don't know what happen, I was worked a complete day on debugging this, here
is my code:

http://pastie.caboo.se/201986

Very thanks for your help.
Regards.

-- 
Rub?n D?vila Santos.
http://rubenonrails.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080523/fa6b4459/attachment.html>

From iambenbryant at gmail.com  Fri May 23 17:07:25 2008
From: iambenbryant at gmail.com (Benjamin H. Bryant)
Date: Fri, 23 May 2008 17:07:25 -0400
Subject: [Backgroundrb-devel] Strange problem with ActiveRecord#find in
	worker
In-Reply-To: <6bcc63d50805231400p470de508lea422d8c8d6ac133@mail.gmail.com>
References: <6bcc63d50805231400p470de508lea422d8c8d6ac133@mail.gmail.com>
Message-ID: <8863AAB7-3CBA-4EBF-94EF-EC3518B7789B@gmail.com>

Ruben,

I had the same (very frustrating) problem.  Im not entirely sure what  
it's root is, but it seems to be due to mysql not having completely  
finished writing/updating before your worker attempts to find the  
record.  Try adding a sleep(1) (or potentially even less time) in your  
worker method before the find.  That worked for me.

Ben

On May 23, 2008, at 5:00 PM, Ruben. D. wrote:

> Hi friends, I've a strange problem with ActiveRecord#find in a  
> worker, this sometimes work fine and somethimes return Nil but the  
> SQL query is fine, I don't know what happen, I was worked a complete  
> day on debugging this, here is my code:
>
> http://pastie.caboo.se/201986
>
> Very thanks for your help.
> Regards.
>
> -- 
> Rub?n D?vila Santos.
> http://rubenonrails.com  
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080523/305e1ea9/attachment.html>

From mike at metaswitch.com  Sat May 24 09:51:43 2008
From: mike at metaswitch.com (Mike Evans)
Date: Sat, 24 May 2008 14:51:43 +0100
Subject: [Backgroundrb-devel] Problems sending large results with
	backgroundrb
In-Reply-To: <fb283d5e0805210056y5792f134u7f3a55769ddd3e1d@mail.gmail.com>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
	<fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
	<fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>
	<52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk>
	<fb283d5e0805210056y5792f134u7f3a55769ddd3e1d@mail.gmail.com>
Message-ID: <52A0FF47062B0B4C80862F2526E02409062F3B31@enfimail2.datcon.co.uk>

Hemant

I'm not sure why we didn't hit that problem in original testing, but we
have hit it in later testing.

I've tried upgrading to the latest packet and backgroundrb from git, but
I'm now having problems with the initial start_worker.  I'm trying to
start the worker passing it a Ruby object of type SearchDn (which is
declared in app/model/search_dn.rb), but I'm hitting the exception
below.  Previously I was running with :lazy_load set to false, but this
doesn't seem to make any difference - has this feature been retired in
this version of code?

Any thoughts?

Mike

/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- dn (MissingSourceFile)
        from
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:495:in `require'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:342:in `new_constants_in'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:495:in `require'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:60:in `load_data'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:32:in `receive_data'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_parser.
rb:30:in `call'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_parser.
rb:30:in `extract'
         ... 9 levels...
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_master.
rb:21:in `run'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:188:in `initialize'
        from ../script/backgroundrb:42:in `new'
        from ../script/backgroundrb:42
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
nbio.rb:25:in `read_data': Packet::DisconnectError
(Packet::DisconnectError)
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
worker.rb:49:in `handle_internal_messages'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:179:in `handle_read_event'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:177:in `each'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:177:in `handle_read_event'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:133:in `start_reactor'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:126:in `loop'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:126:in `start_reactor'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
worker.rb:21:in `start_worker'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
38:in `load_worker'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
26:in `initialize'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
47:in `new'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
47
        from /usr/local/bin/packet_worker_runner:16:in `load'
        from /usr/local/bin/packet_worker_runner:16




-----Original Message-----
From: hemant [mailto:gethemant at gmail.com] 
Sent: 21 May 2008 08:56
To: Mike Evans
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Problems sending large results with
backgroundrb

Yeah that too. But I wonder, how did you solve following two problems:

Take a look at this code:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && internal_scheduled_write[fileno]
# we have a problem here
            write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno] # I was drunk
while writing following line
            handler_instance.write_scheduled(sock_fd)
          end
        end
      end

The problem is, as you said say in a MetaPimp some data is left
unwritten, it won't get written in subsequent writes because
outbound_data belongs to MetaPimp class not main reactor class and
hence, it should be:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && (internal_instance =
internal_scheduled_write[fileno])
            internal_instance.write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno]
            handler_instance.write_and_schedule(sock_fd)
          end
        end
      end

Also, I have included your changes in packet git. So, if you can give
backgroundrb git a shot, I will appreciate that ( Please backup your
older plugin and config files)

On Wed, May 21, 2008 at 12:44 PM, Mike Evans <mike at metaswitch.com>
wrote:
> Hemant
>
> I got to the bottom of the other problem last night.  The issue was 
> with the NbioHelper::write_and_schedule method deleting entries from 
> the outbound_data array while iterating through it.  This can end up 
> with data getting out of order.  I fixed it by changing the
> outbound_data.delete_at(index) statement to outbound_data[index] = 
> nil, and then compacting the array at the end of the iteration.
>
>    # write the data in socket buffer and schedule the thing
>    def write_and_schedule sock
>      outbound_data.each_with_index do |t_data,index|
>        leftover = write_once(t_data,sock)
>        if leftover.empty?
>          outbound_data[index] = nil
>        else
>          outbound_data[index] = leftover
>          reactor.schedule_write(sock)
>          break
>        end
>      end
>      outbound_data.compact!
>      reactor.cancel_write(sock) if outbound_data.empty?
>    end
>
> Mike
>
> -----Original Message-----
> From: hemant [mailto:gethemant at gmail.com]
> Sent: 21 May 2008 05:36
> To: Mike Evans
> Cc: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Problems sending large results with 
> backgroundrb
>
> You can test git version of backgroundrb with git version of packet 
> (which incorporates latest changes). The procedure is as follows:
>
> clone the packet git repo:
>
> git clone git://github.com/gnufied/packet.git
> cd packet;rake gem
> cd pkg; sudo gem install --local packet-0.1.6.gem
>
> Go to your vendor directory of your rails directory and remove or 
> backup older version of backgroundrb plugin and backup related config 
> file as well.
>
> from vendor directory:
>
> git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb 
> cd RAILS_ROOT <<assuming older script and config file has been backed 
> up>> rake backgroundrb:setup <<modify config/backgroundrb.yml 
> according to your needs>> ./script/backgroundrb start <<Let me know, 
> how it goes and if this fixes your problem>>
>
>
> On Wed, May 21, 2008 at 9:42 AM, hemant <gethemant at gmail.com> wrote:
>> On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com>
> wrote:
>>> I'm working on an application that does extensive database
searching.
>
>>> These searches can take a long time, so we have been working on 
>>> moving the searches to a backgroundrb worker task so we can provide 
>>> a
>
>>> sexy AJAX progress bar, and populate the search results as they are 
>>> available.  All of this seems to work fine until the size of the 
>>> search results gets sufficiently large, when we start to hit 
>>> exceptions in backgroundrb (most likely in the packet layer).  We 
>>> are
>
>>> using packet-0.5.1 and backgroundrb from the latest svn mirror.
>>>
>>> We have found and fixed one problem in the packet sender.  This is 
>>> triggered when the non-blocking send in NbioHelper::send_once cannot

>>> send the entire buffer, resulting in an exception in the line
>>>
>>>       write_scheduled[fileno] ||= connections[fileno].instance
>>>
>>> in Core::schedule_write because connections[fileno] is nil.  I can't

>>> claim to fully understand the code, but I think there are two
> problems here.
>>>
>>> The main issue seems to be that when Core::handle_write_event calls 
>>> write_and_schedule to schedule the write, it doesn't clear out 
>>> internal_scheduled_write[fileno].  It looks like the code is 
>>> expecting the cancel_write call at the end of write_and_schedule to 
>>> clear it out, but this doesn't happen if there is enough queued data

>>> to cause the non-blocking write to only partially succeed again.  In

>>> this case, Core::schedule_write is called again, and because 
>>> internal_schedule_write[fileno] has not been cleared out, the code 
>>> drops through to the second if test, then hits the above exception.
>>> We fixed this by adding the line
>>>
>>> internal_scheduled_write.delete(fileno)
>>>
>>> immediately before the call to write_and_schedule in 
>>> Core::handle_write_event.
>>>
>>> The secondary issue is that the connections[fileno] structure is not

>>> getting populated for this connection - I'm guessing because it is 
>>> an
>
>>> internal socket rather than a network socket, but I couldn't be
sure.
>
>>> We changed the second if test in Core::schedule_write to
>>>
>>>       elsif write_scheduled[fileno].nil? &&
!connections[fileno].nil?
>>>
>>> to firewall against this, but we are not sure if this is the right
> fix.
>>
>> Thats was surely a bug and I fixed it like this:
>>
>>      def schedule_write(t_sock,internal_instance = nil)
>>        fileno = t_sock.fileno
>>        if UNIXSocket === t_sock &&
> internal_scheduled_write[fileno].nil?
>>          write_ios << t_sock
>>          internal_scheduled_write[t_sock.fileno] ||=
internal_instance
>>        elsif write_scheduled[fileno].nil? &&
> !(t_sock.is_a?(UNIXSocket))
>>          write_ios << t_sock
>>          write_scheduled[fileno] ||= connections[fileno].instance
>>        end
>>      end
>>
>> Also, I fixed issue with marshalling larger data across the channel.
>> Thanks for reporting this. I have been terribly busy with things in 
>> office and personal life and hence my work on BackgrounDRb has been 
>> in
>
>> hiatus for a while. Unfortunately, you can't use trunk packet code 
>> which is available from:
>>
>> git clone git://github.com/gnufied/packet.git
>>
>> directly with svn mirror of backgroundrb, since packet now uses fork 
>> and exec to run workers and hence reducing memory usage of workers.
>> However in a day or two I will update git repository of BackgrounDRb 
>> which makes use of latest packet version. In the meanwhile, you can 
>> try backporting relevant packet changes to version you are using and 
>> see if it fixes your problem.
>>
>>>
>>> We are now hitting problems in the Packet::MetaPimp module receiving

>>> the data, usually an exception in the Marshal.load call in 
>>> MetaPimp::receive_data.  We suspect this is caused by the packet 
>>> code
>
>>> corrupting the data somewhere, probably because we are sending such 
>>> large arrays of results (the repro I am working on at the moment is 
>>> trying to marshal over 200k of data).  We've been trying to put 
>>> extra
>
>>> diagnostics in the code so we can see what is happening, but if we 
>>> edit puts statements into the code we only seem to get output from 
>>> the end of the connection that hits an exception and so far our 
>>> attempts to make logger objects available throughout the code have 
>>> failed.  We therefore thought we would ask for help
>>> - either to see whether this is a known problem, or whether there is

>>> a recommended way to add diagnostics to the packet code.
>>>
>>> I'm also open to ideas as to better ways to solve the problem!
>>>
>>> Thanks in advance,
>>>
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>>
>> --
>> Let them talk of their oriental summer climes of everlasting 
>> conservatories; give me the privilege of making my own summer with my

>> own coals.
>>
>> http://gnufied.org
>>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting 
> conservatories; give me the privilege of making my own summer with my 
> own coals.
>
> http://gnufied.org
>



--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From mike at metaswitch.com  Sat May 24 10:49:48 2008
From: mike at metaswitch.com (Mike Evans)
Date: Sat, 24 May 2008 15:49:48 +0100
Subject: [Backgroundrb-devel] Problems sending large results
	withbackgroundrb
In-Reply-To: <52A0FF47062B0B4C80862F2526E02409062F3B31@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk><fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com><fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com><52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk><fb283d5e0805210056y5792f134u7f3a55769ddd3e1d@mail.gmail.com>
	<52A0FF47062B0B4C80862F2526E02409062F3B31@enfimail2.datcon.co.uk>
Message-ID: <52A0FF47062B0B4C80862F2526E02409062F3B32@enfimail2.datcon.co.uk>

Hemant

I fixed a minor bug that means the code is now getting the right file
name, but object file is still failing to load.

The fix is to change the regular expression used to process the
Marshal.load exception is MasterWorker::load_data from

      if error_msg =~ /^undefined.+([A-Z]\w+)/ 

to

      if error_msg =~ /^undefined.+ ([A-Z]\w+)/

The extra space forces it to take the whole of the last word in the
error message, not the just the last capital onward.

I suspect the issue I'm now seeing is because the MasterWorker class
doesn't load the Rails environment.  Any thoughts on how to fix this?

Mike

-----Original Message-----
From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of Mike
Evans
Sent: 24 May 2008 14:52
To: hemant
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Problems sending large results
withbackgroundrb

Hemant

I'm not sure why we didn't hit that problem in original testing, but we
have hit it in later testing.

I've tried upgrading to the latest packet and backgroundrb from git, but
I'm now having problems with the initial start_worker.  I'm trying to
start the worker passing it a Ruby object of type SearchDn (which is
declared in app/model/search_dn.rb), but I'm hitting the exception
below.  Previously I was running with :lazy_load set to false, but this
doesn't seem to make any difference - has this feature been retired in
this version of code?

Any thoughts?

Mike

/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- dn (MissingSourceFile)
        from
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:495:in `require'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:342:in `new_constants_in'
        from
/usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support
/dependencies.rb:495:in `require'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:60:in `load_data'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:32:in `receive_data'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_parser.
rb:30:in `call'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_parser.
rb:30:in `extract'
         ... 9 levels...
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/lib/packet/packet_master.
rb:21:in `run'
        from
/disk0.7/var/opt/MetaViewSAS/tview/vendor/plugins/backgroundrb/server/li
b/master_worker.rb:188:in `initialize'
        from ../script/backgroundrb:42:in `new'
        from ../script/backgroundrb:42
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
nbio.rb:25:in `read_data': Packet::DisconnectError
(Packet::DisconnectError)
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
worker.rb:49:in `handle_internal_messages'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:179:in `handle_read_event'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:177:in `each'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:177:in `handle_read_event'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:133:in `start_reactor'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:126:in `loop'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
core.rb:126:in `start_reactor'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/../lib/packet/packet_
worker.rb:21:in `start_worker'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
38:in `load_worker'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
26:in `initialize'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
47:in `new'
        from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.6/bin/packet_worker_runner:
47
        from /usr/local/bin/packet_worker_runner:16:in `load'
        from /usr/local/bin/packet_worker_runner:16




-----Original Message-----
From: hemant [mailto:gethemant at gmail.com]
Sent: 21 May 2008 08:56
To: Mike Evans
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Problems sending large results with
backgroundrb

Yeah that too. But I wonder, how did you solve following two problems:

Take a look at this code:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && internal_scheduled_write[fileno]
# we have a problem here
            write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno] # I was drunk
while writing following line
            handler_instance.write_scheduled(sock_fd)
          end
        end
      end

The problem is, as you said say in a MetaPimp some data is left
unwritten, it won't get written in subsequent writes because
outbound_data belongs to MetaPimp class not main reactor class and
hence, it should be:

      def handle_write_event(p_ready_fds)
        p_ready_fds.each do |sock_fd|
          fileno = sock_fd.fileno
          if UNIXSocket === sock_fd && (internal_instance =
internal_scheduled_write[fileno])
            internal_instance.write_and_schedule(sock_fd)
          elsif extern_opts = connection_completion_awaited[fileno]
            complete_connection(sock_fd,extern_opts)
          elsif handler_instance = write_scheduled[fileno]
            handler_instance.write_and_schedule(sock_fd)
          end
        end
      end

Also, I have included your changes in packet git. So, if you can give
backgroundrb git a shot, I will appreciate that ( Please backup your
older plugin and config files)

On Wed, May 21, 2008 at 12:44 PM, Mike Evans <mike at metaswitch.com>
wrote:
> Hemant
>
> I got to the bottom of the other problem last night.  The issue was 
> with the NbioHelper::write_and_schedule method deleting entries from 
> the outbound_data array while iterating through it.  This can end up 
> with data getting out of order.  I fixed it by changing the
> outbound_data.delete_at(index) statement to outbound_data[index] = 
> nil, and then compacting the array at the end of the iteration.
>
>    # write the data in socket buffer and schedule the thing
>    def write_and_schedule sock
>      outbound_data.each_with_index do |t_data,index|
>        leftover = write_once(t_data,sock)
>        if leftover.empty?
>          outbound_data[index] = nil
>        else
>          outbound_data[index] = leftover
>          reactor.schedule_write(sock)
>          break
>        end
>      end
>      outbound_data.compact!
>      reactor.cancel_write(sock) if outbound_data.empty?
>    end
>
> Mike
>
> -----Original Message-----
> From: hemant [mailto:gethemant at gmail.com]
> Sent: 21 May 2008 05:36
> To: Mike Evans
> Cc: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Problems sending large results with 
> backgroundrb
>
> You can test git version of backgroundrb with git version of packet 
> (which incorporates latest changes). The procedure is as follows:
>
> clone the packet git repo:
>
> git clone git://github.com/gnufied/packet.git
> cd packet;rake gem
> cd pkg; sudo gem install --local packet-0.1.6.gem
>
> Go to your vendor directory of your rails directory and remove or 
> backup older version of backgroundrb plugin and backup related config 
> file as well.
>
> from vendor directory:
>
> git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb 
> cd RAILS_ROOT <<assuming older script and config file has been backed
> up>> rake backgroundrb:setup <<modify config/backgroundrb.yml
> according to your needs>> ./script/backgroundrb start <<Let me know, 
> how it goes and if this fixes your problem>>
>
>
> On Wed, May 21, 2008 at 9:42 AM, hemant <gethemant at gmail.com> wrote:
>> On Wed, May 21, 2008 at 1:00 AM, Mike Evans <mike at metaswitch.com>
> wrote:
>>> I'm working on an application that does extensive database
searching.
>
>>> These searches can take a long time, so we have been working on 
>>> moving the searches to a backgroundrb worker task so we can provide 
>>> a
>
>>> sexy AJAX progress bar, and populate the search results as they are 
>>> available.  All of this seems to work fine until the size of the 
>>> search results gets sufficiently large, when we start to hit 
>>> exceptions in backgroundrb (most likely in the packet layer).  We 
>>> are
>
>>> using packet-0.5.1 and backgroundrb from the latest svn mirror.
>>>
>>> We have found and fixed one problem in the packet sender.  This is 
>>> triggered when the non-blocking send in NbioHelper::send_once cannot

>>> send the entire buffer, resulting in an exception in the line
>>>
>>>       write_scheduled[fileno] ||= connections[fileno].instance
>>>
>>> in Core::schedule_write because connections[fileno] is nil.  I can't

>>> claim to fully understand the code, but I think there are two
> problems here.
>>>
>>> The main issue seems to be that when Core::handle_write_event calls 
>>> write_and_schedule to schedule the write, it doesn't clear out 
>>> internal_scheduled_write[fileno].  It looks like the code is 
>>> expecting the cancel_write call at the end of write_and_schedule to 
>>> clear it out, but this doesn't happen if there is enough queued data

>>> to cause the non-blocking write to only partially succeed again.  In

>>> this case, Core::schedule_write is called again, and because 
>>> internal_schedule_write[fileno] has not been cleared out, the code 
>>> drops through to the second if test, then hits the above exception.
>>> We fixed this by adding the line
>>>
>>> internal_scheduled_write.delete(fileno)
>>>
>>> immediately before the call to write_and_schedule in 
>>> Core::handle_write_event.
>>>
>>> The secondary issue is that the connections[fileno] structure is not

>>> getting populated for this connection - I'm guessing because it is 
>>> an
>
>>> internal socket rather than a network socket, but I couldn't be
sure.
>
>>> We changed the second if test in Core::schedule_write to
>>>
>>>       elsif write_scheduled[fileno].nil? &&
!connections[fileno].nil?
>>>
>>> to firewall against this, but we are not sure if this is the right
> fix.
>>
>> Thats was surely a bug and I fixed it like this:
>>
>>      def schedule_write(t_sock,internal_instance = nil)
>>        fileno = t_sock.fileno
>>        if UNIXSocket === t_sock &&
> internal_scheduled_write[fileno].nil?
>>          write_ios << t_sock
>>          internal_scheduled_write[t_sock.fileno] ||=
internal_instance
>>        elsif write_scheduled[fileno].nil? &&
> !(t_sock.is_a?(UNIXSocket))
>>          write_ios << t_sock
>>          write_scheduled[fileno] ||= connections[fileno].instance
>>        end
>>      end
>>
>> Also, I fixed issue with marshalling larger data across the channel.
>> Thanks for reporting this. I have been terribly busy with things in 
>> office and personal life and hence my work on BackgrounDRb has been 
>> in
>
>> hiatus for a while. Unfortunately, you can't use trunk packet code 
>> which is available from:
>>
>> git clone git://github.com/gnufied/packet.git
>>
>> directly with svn mirror of backgroundrb, since packet now uses fork 
>> and exec to run workers and hence reducing memory usage of workers.
>> However in a day or two I will update git repository of BackgrounDRb 
>> which makes use of latest packet version. In the meanwhile, you can 
>> try backporting relevant packet changes to version you are using and 
>> see if it fixes your problem.
>>
>>>
>>> We are now hitting problems in the Packet::MetaPimp module receiving

>>> the data, usually an exception in the Marshal.load call in 
>>> MetaPimp::receive_data.  We suspect this is caused by the packet 
>>> code
>
>>> corrupting the data somewhere, probably because we are sending such 
>>> large arrays of results (the repro I am working on at the moment is 
>>> trying to marshal over 200k of data).  We've been trying to put 
>>> extra
>
>>> diagnostics in the code so we can see what is happening, but if we 
>>> edit puts statements into the code we only seem to get output from 
>>> the end of the connection that hits an exception and so far our 
>>> attempts to make logger objects available throughout the code have 
>>> failed.  We therefore thought we would ask for help
>>> - either to see whether this is a known problem, or whether there is

>>> a recommended way to add diagnostics to the packet code.
>>>
>>> I'm also open to ideas as to better ways to solve the problem!
>>>
>>> Thanks in advance,
>>>
>>> Mike
>>>
>>>
>>> _______________________________________________
>>> Backgroundrb-devel mailing list
>>> Backgroundrb-devel at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>>
>>
>>
>>
>> --
>> Let them talk of their oriental summer climes of everlasting 
>> conservatories; give me the privilege of making my own summer with my

>> own coals.
>>
>> http://gnufied.org
>>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting 
> conservatories; give me the privilege of making my own summer with my 
> own coals.
>
> http://gnufied.org
>



--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org
_______________________________________________
Backgroundrb-devel mailing list
Backgroundrb-devel at rubyforge.org
http://rubyforge.org/mailman/listinfo/backgroundrb-devel

From gethemant at gmail.com  Sat May 24 11:13:25 2008
From: gethemant at gmail.com (hemant kumar)
Date: Sat, 24 May 2008 20:43:25 +0530
Subject: [Backgroundrb-devel] Problems sending large
	results	withbackgroundrb
In-Reply-To: <52A0FF47062B0B4C80862F2526E02409062F3B32@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E024090629BA9A@enfimail2.datcon.co.uk>
	<fb283d5e0805202112v65a1e2dga24c1fd63ecab6a6@mail.gmail.com>
	<fb283d5e0805202136p25af9151wc9c658b93c6c7d6a@mail.gmail.com>
	<52A0FF47062B0B4C80862F2526E024090629BAE8@enfimail2.datcon.co.uk>
	<fb283d5e0805210056y5792f134u7f3a55769ddd3e1d@mail.gmail.com>
	<52A0FF47062B0B4C80862F2526E02409062F3B31@enfimail2.datcon.co.uk>
	<52A0FF47062B0B4C80862F2526E02409062F3B32@enfimail2.datcon.co.uk>
Message-ID: <1211642005.8142.4.camel@shire>

On Sat, 2008-05-24 at 15:49 +0100, Mike Evans wrote:
> Hemant
> 
> I fixed a minor bug that means the code is now getting the right file
> name, but object file is still failing to load.
> 
> The fix is to change the regular expression used to process the
> Marshal.load exception is MasterWorker::load_data from
> 
>       if error_msg =~ /^undefined.+([A-Z]\w+)/ 
> 
> to
> 
>       if error_msg =~ /^undefined.+ ([A-Z]\w+)/
> 
> The extra space forces it to take the whole of the last word in the
> error message, not the just the last capital onward.
> 
> I suspect the issue I'm now seeing is because the MasterWorker class
> doesn't load the Rails environment.  Any thoughts on how to fix this?
> 

Yeah, Mike, When I saw your mail I knew that there is some problem with
that piece of regexp. Now, rails environment IS getting loaded in master
worker, but somehow, autoloading of models is not working from master
class (although, it works from workers alright) and hence my own hand
rolled mechanism of autoloading models. I have no idea, from top of my
head, why thats happening, but expect a fix soon ( or a patch is more
than welcome ).



From andywatts at yahoo.com  Tue May 27 21:34:55 2008
From: andywatts at yahoo.com (Andy Watts)
Date: Tue, 27 May 2008 18:34:55 -0700 (PDT)
Subject: [Backgroundrb-devel] nil.send_request = Incorrect worker name?
Message-ID: <278118.80649.qm@web53102.mail.re2.yahoo.com>

Hi, 

Would greatly appreciate some help with this.
Is the following error caused by an incorrect worker name?

You have a nil object when you didn't expect it!
The error occurred while evaluating nil.send_request
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in `ask_worker'


Here's what I'm doing in my rails console.  

MiddleMan.new_worker(:worker => :foo_worker, :job_key => '1')
worker = MiddleMan.worker(:worker => :foo_worker, :job_key => '1')
worker.bar()



Querying all workers shows the foo_worker_1 is there..
>> pp MiddleMan.query_all_workers
{:type=>:all_worker_status}
{:log_worker=>nil,
 :foo_worker_1=>{:rand=>679}}

I'm using rails 2.02, Packet 0.1.5 and Chronic 0.2.3.


I tried the mainline git version but got this error.  It needs a newer version of Packet?
`load_missing_constant': uninitialized constant Packet::BinParser (NameError)

Many Thanks
Andy Watts

       
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080527/1b509fc6/attachment.html>

From gethemant at gmail.com  Tue May 27 22:30:55 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 28 May 2008 08:00:55 +0530
Subject: [Backgroundrb-devel] nil.send_request = Incorrect worker name?
In-Reply-To: <278118.80649.qm@web53102.mail.re2.yahoo.com>
References: <278118.80649.qm@web53102.mail.re2.yahoo.com>
Message-ID: <fb283d5e0805271930s7afe49abw148e9c2bb450503b@mail.gmail.com>

On Wed, May 28, 2008 at 7:04 AM, Andy Watts <andywatts at yahoo.com> wrote:
> Hi,
>
> Would greatly appreciate some help with this.
> Is the following error caused by an incorrect worker name?
>
> You have a nil object when you didn't expect it!
> The error occurred while evaluating nil.send_request
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in
> `ask_worker'
>
> Here's what I'm doing in my rails console.
>
> MiddleMan.new_worker(:worker => :foo_worker, :job_key => '1')
> worker = MiddleMan.worker(:worker => :foo_worker, :job_key => '1')
> worker.bar()
>
>
> Querying all workers shows the foo_worker_1 is there..
>>> pp MiddleMan.query_all_workers
> {:type=>:all_worker_status}
> {:log_worker=>nil,
>  :foo_worker_1=>{:rand=>679}}
>
> I'm using rails 2.02, Packet 0.1.5 and Chronic 0.2.3.
>
>
> I tried the mainline git version but got this error.  It needs a newer
> version of Packet?
> `load_missing_constant': uninitialized constant Packet::BinParser
> (NameError)
>

Yeah, it does need newer version of packet. Can you try this ?

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

From rubyonrails-ug at peuchert.de  Wed May 28 09:39:55 2008
From: rubyonrails-ug at peuchert.de (Alex)
Date: Wed, 28 May 2008 15:39:55 +0200
Subject: [Backgroundrb-devel] Has BackgroundRb been moved to github.com?
In-Reply-To: <fb283d5e0802210350g4e868242uf3a02603ca953118@mail.gmail.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>	
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>	
	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>	
	<fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>	
	<de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>	
	<47BD3DB9.1050507@peuchert.de>
	<fb283d5e0802210350g4e868242uf3a02603ca953118@mail.gmail.com>
Message-ID: <483D60AB.4050707@peuchert.de>

Hi BackgroundRb users!

I've been away for a while, now. And after having a look at 
backgroundrb, I found that there is a repository of backgroundrb at 
github.com. Is this the place to develop, now?

Regards, Alexander

From gethemant at gmail.com  Wed May 28 12:28:58 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 28 May 2008 21:58:58 +0530
Subject: [Backgroundrb-devel] Has BackgroundRb been moved to github.com?
In-Reply-To: <483D60AB.4050707@peuchert.de>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>
	<fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>
	<de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>
	<47BD3DB9.1050507@peuchert.de>
	<fb283d5e0802210350g4e868242uf3a02603ca953118@mail.gmail.com>
	<483D60AB.4050707@peuchert.de>
Message-ID: <fb283d5e0805280928y28433658j63fe795a8a8908ee@mail.gmail.com>

On Wed, May 28, 2008 at 7:09 PM, Alex <rubyonrails-ug at peuchert.de> wrote:
> Hi BackgroundRb users!
>
> I've been away for a while, now. And after having a look at backgroundrb, I
> found that there is a repository of backgroundrb at github.com. Is this the
> place to develop, now?
>

Hi Alex,

Yes, I have moved backgroundrb repo to github. I was having various
issues with gitorious and hence the reason. I will give you the commit
bits, hang on.

From solomon20723 at yahoo.com  Wed May 28 15:54:11 2008
From: solomon20723 at yahoo.com (A &amp; L)
Date: Wed, 28 May 2008 12:54:11 -0700 (PDT)
Subject: [Backgroundrb-devel] version yml file
Message-ID: <754815.32160.qm@web56613.mail.re3.yahoo.com>

Hi,
I am trying to get the scheduling to work with my RoR application.
How do I determine what version of backgroundrb I have configured?
When I try to invoke??? rake backgroundrb:setup? , it fails.
Thanks for the help.
Linda


      

From saratkongara at gmail.com  Thu May 29 06:05:23 2008
From: saratkongara at gmail.com (Sarat Kongara)
Date: Thu, 29 May 2008 15:35:23 +0530
Subject: [Backgroundrb-devel] How to know the environment
	(development/test/production) inside a worker
In-Reply-To: <a4855ac50805212205l4c337ffdl19eb0594bcd3c731@mail.gmail.com>
References: <4c251b0d0805202137k1a5fe3cfr2b86cd67f69bc49c@mail.gmail.com>
	<a4855ac50805212205l4c337ffdl19eb0594bcd3c731@mail.gmail.com>
Message-ID: <4c251b0d0805290305o371ef753s17fe0ba4c8f09bfc@mail.gmail.com>

Hi Ryan,For some reason even when the I start backgroundrb like this $
script/server -e production start
The RAILS_ENV is set to development. Any suggestions on how to fix this
issue. Thanks.

Regards
Sarat


On 5/22/08, Ryan Leavengood <leavengood at gmail.com> wrote:
>
> On 5/21/08, Sarat Kongara <saratkongara at gmail.com> wrote:
> >
> > I am not sure how to figure out the environment with which the
> backgroundRb
> > is started from inside a worker task. Any help is appreciated. Thanks for
> > your time.
>
>
> ENV["RAILS_ENV"] is defined properly inside the workers.
>
>
> Ryan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/4bae61fa/attachment.html>

From saratkongara at gmail.com  Thu May 29 08:06:17 2008
From: saratkongara at gmail.com (Sarat Kongara)
Date: Thu, 29 May 2008 17:36:17 +0530
Subject: [Backgroundrb-devel] How to know the environment
	(development/test/production) inside a worker - FIXED
Message-ID: <4c251b0d0805290506p16b5df9h882cef01a53ed4ab@mail.gmail.com>

Hi Guys,Looks like I have to do this in backgroundrb.yml

:backgroundrb:
  :ip: 0.0.0.0

:production:
  :backgroundrb:
    :port: 11008      # use port 11008
    :environment: production

That is, have an entry for "environment: production" under :production:,
This seems to have fixed the issue.

Regards
Sarat

On 5/29/08, Sarat Kongara <saratkongara at gmail.com> wrote:
>
> Hi Ryan,For some reason even when the I start backgroundrb like this $
> script/server -e production start
> The RAILS_ENV is set to development. Any suggestions on how to fix this
> issue. Thanks.
>
> Regards
> Sarat
>
>
>
> On 5/22/08, Ryan Leavengood <leavengood at gmail.com> wrote:
>>
>> On 5/21/08, Sarat Kongara <saratkongara at gmail.com> wrote:
>> >
>> > I am not sure how to figure out the environment with which the
>> backgroundRb
>> > is started from inside a worker task. Any help is appreciated. Thanks
>> for
>> > your time.
>>
>>
>> ENV["RAILS_ENV"] is defined properly inside the workers.
>>
>>
>> Ryan
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/a9ce8b09/attachment.html>

From raghu.srinivasan at gmail.com  Fri May 30 00:05:10 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Thu, 29 May 2008 21:05:10 -0700
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?
Message-ID: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>

I use BDRb to process RSS feeds for users on my site (http://feedflix.com).

I have a batch job that queries the DB for records that haven't been updated
in the last so many hours and kicks off a background job for each of them.
If N records are returned by the DB, N background jobs gets queued and get
done serially. As long as N is 255 or under, everything works like a charm.
I've noticed that whenever N is >= 256 (2 power 8), then at the 257th job
BDRb stops processing any more users. I can get around it by limiting the DB
query to return no more than 255 records and then all is fine. No problems
at all. But over that, I see this issue. Repeatedly.

Here's what I found in my log/backgroundrb_11007_debug.log file.
=========
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
./script/backgroundrb:42:in `new'
./script/backgroundrb:42
{:type=>:do_work, :worker=>:batch_worker, :data=>{:body=>"All Done!!!",
:subject=>"Completed processing for : 275 users"},
:worker_method=>:email_admin}
You have a nil object when you didn't expect it!
The error occurred while evaluating nil.instance
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
`schedule_write'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
`write_and_schedule'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in
`each_with_index'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each_with_index'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`write_and_schedule'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
`dump_object'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_pimp.rb:23:in
`send_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:76:in
`send_request'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in
`ask_worker'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:104:in`process_work'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:35:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`call'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`extract'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:42:in
`extract'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:31:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:212:in
`read_external_socket'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:204:in
`handle_external_messages'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
./script/backgroundrb:42:in `new'
./script/backgroundrb:42
=========
My log/backgroundrb_server_11007.log is clean - there are no error messages
at all there.
Is anyone else seeing this? Am I missing something obvious? Any solutions?
Thanks,
Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/dced6a98/attachment.html>

From hueyplong at gmail.com  Fri May 30 00:33:21 2008
From: hueyplong at gmail.com (Kirk Brown)
Date: Thu, 29 May 2008 23:33:21 -0500
Subject: [Backgroundrb-devel] Packet error on startup suddenly.
Message-ID: <c8c9893c0805292133h2e82581am9d8837aa1587a707@mail.gmail.com>

Hello, I've been using backgroundrb successfully for months now, but
suddenly I'm getting the error below when trying to start it up. I did
some searching and I see that someone posted about it last month:

http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001685.html

Anyone have any idea what could have happened? I made no changes to
the code or configs, but I did recently update the packages on the
Debian server that it's running on. Could some package be affecting
packet 0.1.5?

Here's the error, and thanks for any help you can give me!


/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:48:in
`process_request': undefined method `send_request' for nil:NilClass
(NoMethodError)
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:27:in
`handle_object'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:20:in
`receive_data'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in
`call'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in
`extract'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:18:in
`receive_data'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:64:in
`handle_internal_messages'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
        from /var/www/do512/releases/20080529045546/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
`start_worker'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
`fork_and_load'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
`load_workers'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`each'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`load_workers'
        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
        from /var/www/do512/releases/20080529045546/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42

From gethemant at gmail.com  Fri May 30 12:12:22 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 30 May 2008 21:42:22 +0530
Subject: [Backgroundrb-devel] version yml file
In-Reply-To: <754815.32160.qm@web56613.mail.re3.yahoo.com>
References: <754815.32160.qm@web56613.mail.re3.yahoo.com>
Message-ID: <fb283d5e0805300912h792fa526j831cfc6abc6834bb@mail.gmail.com>

you can use:
./script/backgroundrb --version


On Thu, May 29, 2008 at 1:24 AM, A &amp; L <solomon20723 at yahoo.com> wrote:
> Hi,
> I am trying to get the scheduling to work with my RoR application.
> How do I determine what version of backgroundrb I have configured?
> When I try to invoke    rake backgroundrb:setup  , it fails.
> Thanks for the help.
> Linda
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri May 30 12:17:20 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 30 May 2008 21:47:20 +0530
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued by
	BDRb?
In-Reply-To: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
References: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
Message-ID: <fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>

On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> I use BDRb to process RSS feeds for users on my site (http://feedflix.com).
>
> I have a batch job that queries the DB for records that haven't been updated
> in the last so many hours and kicks off a background job for each of them.
> If N records are returned by the DB, N background jobs gets queued and get
> done serially. As long as N is 255 or under, everything works like a charm.
> I've noticed that whenever N is >= 256 (2 power 8), then at the 257th job
> BDRb stops processing any more users. I can get around it by limiting the DB
> query to return no more than 255 records and then all is fine. No problems
> at all. But over that, I see this issue. Repeatedly.
>

How are you queuing the jobs? Are you using thread_pool? I am afraid,
it could be because of restriction in number of open file descriptors
open to 1024.

From gethemant at gmail.com  Fri May 30 12:21:03 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 30 May 2008 21:51:03 +0530
Subject: [Backgroundrb-devel] Packet error on startup suddenly.
In-Reply-To: <c8c9893c0805292133h2e82581am9d8837aa1587a707@mail.gmail.com>
References: <c8c9893c0805292133h2e82581am9d8837aa1587a707@mail.gmail.com>
Message-ID: <fb283d5e0805300921v54f211bfh5dd34c18b87bae40@mail.gmail.com>

Mike discovered a bug there and It could be because of that. If I am
not asking for too much, can you try git version of backgroundrb and
packet.

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

Git version has that fix. If you are not inclined in learning git
mumbo,jumbo you can just download tarball and try. BTW, i have the
fix, i will try to backport it to svn.

Thanks for patience.

On Fri, May 30, 2008 at 10:03 AM, Kirk Brown <hueyplong at gmail.com> wrote:
> Hello, I've been using backgroundrb successfully for months now, but
> suddenly I'm getting the error below when trying to start it up. I did
> some searching and I see that someone posted about it last month:
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001685.html
>
> Anyone have any idea what could have happened? I made no changes to
> the code or configs, but I did recently update the packages on the
> Debian server that it's running on. Could some package be affecting
> packet 0.1.5?
>
> Here's the error, and thanks for any help you can give me!
>
>
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:48:in
> `process_request': undefined method `send_request' for nil:NilClass
> (NoMethodError)
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:27:in
> `handle_object'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:20:in
> `receive_data'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in
> `call'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:37:in
> `extract'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:18:in
> `receive_data'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:64:in
> `handle_internal_messages'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
> `run'
>        from /var/www/do512/releases/20080529045546/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
> `handle_internal_messages'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
> `start_worker'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
> `fork_and_load'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
> `load_workers'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `each'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `load_workers'
>        from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> `run'
>        from /var/www/do512/releases/20080529045546/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From raghu.srinivasan at gmail.com  Fri May 30 13:25:40 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 30 May 2008 10:25:40 -0700
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued by
	BDRb?
In-Reply-To: <fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>
References: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
	<fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>
Message-ID: <6482c06a0805301025x36bf420bmc9c3e4a55aeb8133@mail.gmail.com>

Hi Hemant - no, I am not using thread_pool right now. I do have two separate
workers like you and Ryan Leavengood suggested - one for the batch process
and the other for the live/web user initiated process - which by the way
works out great, thanks!.

How are other folks handling 1000s of RSS refreshes? Via BDRb - or something
else? Is BDRb really the best tool for what I am trying to do? I'd really
appreciate if others could share their experiences.

Thanks in advance

On Fri, May 30, 2008 at 9:17 AM, hemant <gethemant at gmail.com> wrote:

> On Fri, May 30, 2008 at 9:35 AM, Raghu Srinivasan
> <raghu.srinivasan at gmail.com> wrote:
> > I use BDRb to process RSS feeds for users on my site (
> http://feedflix.com).
> >
> > I have a batch job that queries the DB for records that haven't been
> updated
> > in the last so many hours and kicks off a background job for each of
> them.
> > If N records are returned by the DB, N background jobs gets queued and
> get
> > done serially. As long as N is 255 or under, everything works like a
> charm.
> > I've noticed that whenever N is >= 256 (2 power 8), then at the 257th job
> > BDRb stops processing any more users. I can get around it by limiting the
> DB
> > query to return no more than 255 records and then all is fine. No
> problems
> > at all. But over that, I see this issue. Repeatedly.
> >
>
> How are you queuing the jobs? Are you using thread_pool? I am afraid,
> it could be because of restriction in number of open file descriptors
> open to 1024.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/3c0ccdad/attachment.html>

From solomon20723 at yahoo.com  Fri May 30 13:47:17 2008
From: solomon20723 at yahoo.com (A &amp; L)
Date: Fri, 30 May 2008 10:47:17 -0700 (PDT)
Subject: [Backgroundrb-devel] version yml file
Message-ID: <359195.71620.qm@web56603.mail.re3.yahoo.com>

Thanks for the response.
I don't think the backgroundrb is installed correctly.
Here is the response that I get:
script/backgroundrb: is a directory


----- Original Message ----
From: hemant <gethemant at gmail.com>
To: "A &, L" <solomon20723 at yahoo.com>
Cc: backgroundrb-devel at rubyforge.org
Sent: Friday, May 30, 2008 12:12:22 PM
Subject: Re: [Backgroundrb-devel] version yml file

you can use:
./script/backgroundrb --version


On Thu, May 29, 2008 at 1:24 AM, A &amp; L <solomon20723 at yahoo.com> wrote:
> Hi,
> I am trying to get the scheduling to work with my RoR application.
> How do I determine what version of backgroundrb I have configured?
> When I try to invoke? ? rake backgroundrb:setup? , it fails.
> Thanks for the help.
> Linda
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org



      

From gethemant at gmail.com  Fri May 30 15:01:09 2008
From: gethemant at gmail.com (hemant kumar)
Date: Sat, 31 May 2008 00:31:09 +0530
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued
	by	BDRb?
In-Reply-To: <6482c06a0805301025x36bf420bmc9c3e4a55aeb8133@mail.gmail.com>
References: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
	<fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>
	<6482c06a0805301025x36bf420bmc9c3e4a55aeb8133@mail.gmail.com>
Message-ID: <1212174069.12645.1.camel@shire>

On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:
> Hi Hemant - no, I am not using thread_pool right now. I do have two
> separate workers like you and Ryan Leavengood suggested - one for the
> batch process and the other for the live/web user initiated process -
> which by the way works out great, thanks!. 
> 
> How are other folks handling 1000s of RSS refreshes? Via BDRb - or
> something else? Is BDRb really the best tool for what I am trying to
> do? I'd really appreciate if others could share their experiences.
> 

okay, then it shouldn't be a problem. Can you try git version of
backgroundrb as suggest in following link and report us back?

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/




From gnegnegne at gmail.com  Fri May 30 15:20:49 2008
From: gnegnegne at gmail.com (Paolo Lancelli)
Date: Fri, 30 May 2008 21:20:49 +0200
Subject: [Backgroundrb-devel] Adding vendor folder to $load_path
Message-ID: <40376d8f0805301220s4ed865b6y6d1f30ffd4643c38@mail.gmail.com>

I'm using backgroundrb in a rails project, with packet and chronic unpacked
in the vendor folder, to make backgroundrb I've added the following line to
scribts/backgroundrb

$LOAD_PATH << File.join(RAILS_HOME, "vendor")

Do you believe it's a good idea? I don't like depend on the installed gems

ciao ciao
paolo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/6c24539c/attachment-0001.html>

From raghu.srinivasan at gmail.com  Fri May 30 15:38:39 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 30 May 2008 12:38:39 -0700
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued by
	BDRb?
In-Reply-To: <1212174069.12645.1.camel@shire>
References: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
	<fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>
	<6482c06a0805301025x36bf420bmc9c3e4a55aeb8133@mail.gmail.com>
	<1212174069.12645.1.camel@shire>
Message-ID: <6482c06a0805301238m259bd0a4o5a2a921d470e7ff5@mail.gmail.com>

On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at gmail.com> wrote:

> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:
> > Hi Hemant - no, I am not using thread_pool right now. I do have two
> > separate workers like you and Ryan Leavengood suggested - one for the
> > batch process and the other for the live/web user initiated process -
> > which by the way works out great, thanks!.
> >
> > How are other folks handling 1000s of RSS refreshes? Via BDRb - or
> > something else? Is BDRb really the best tool for what I am trying to
> > do? I'd really appreciate if others could share their experiences.
> >
>
> okay, then it shouldn't be a problem. Can you try git version of
> backgroundrb as suggest in following link and report us back?
>
>
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>
>
>
> Hemant - Thanks for your help. I'll try this first on development and then
report back. With this, will I be able to queue up 256+ or 1024+ b/g jobs?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080530/9b7f12cc/attachment.html>

From gethemant at gmail.com  Sat May 31 04:33:17 2008
From: gethemant at gmail.com (hemant kumar)
Date: Sat, 31 May 2008 14:03:17 +0530
Subject: [Backgroundrb-devel] Adding vendor folder to $load_path
In-Reply-To: <40376d8f0805301220s4ed865b6y6d1f30ffd4643c38@mail.gmail.com>
References: <40376d8f0805301220s4ed865b6y6d1f30ffd4643c38@mail.gmail.com>
Message-ID: <1212222797.17476.1.camel@shire>

On Fri, 2008-05-30 at 21:20 +0200, Paolo Lancelli wrote:
> I'm using backgroundrb in a rails project, with packet and chronic
> unpacked in the vendor folder, to make backgroundrb I've added the
> following line to scribts/backgroundrb
> 
> $LOAD_PATH << File.join(RAILS_HOME, "vendor")
> 
> Do you believe it's a good idea? I don't like depend on the installed
> gems
> 

yeah, that's perfectly alright. Also, it means that require "packet"
should go after require "enviroment.rb" so as use customization, hold
on.




From gethemant at gmail.com  Sat May 31 08:17:25 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 31 May 2008 17:47:25 +0530
Subject: [Backgroundrb-devel] Upper limit to number of jobs queued by
	BDRb?
In-Reply-To: <6482c06a0805301238m259bd0a4o5a2a921d470e7ff5@mail.gmail.com>
References: <6482c06a0805292105od5e2f96v283d407decbac60d@mail.gmail.com>
	<fb283d5e0805300917l736ae683hc3267cb370c4f84c@mail.gmail.com>
	<6482c06a0805301025x36bf420bmc9c3e4a55aeb8133@mail.gmail.com>
	<1212174069.12645.1.camel@shire>
	<6482c06a0805301238m259bd0a4o5a2a921d470e7ff5@mail.gmail.com>
Message-ID: <fb283d5e0805310517w607918d4gce0cbafda5d4b160@mail.gmail.com>

On Sat, May 31, 2008 at 1:08 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
>
>
> On Fri, May 30, 2008 at 12:01 PM, hemant kumar <gethemant at gmail.com> wrote:
>>
>> On Fri, 2008-05-30 at 10:25 -0700, Raghu Srinivasan wrote:
>> > Hi Hemant - no, I am not using thread_pool right now. I do have two
>> > separate workers like you and Ryan Leavengood suggested - one for the
>> > batch process and the other for the live/web user initiated process -
>> > which by the way works out great, thanks!.
>> >
>> > How are other folks handling 1000s of RSS refreshes? Via BDRb - or
>> > something else? Is BDRb really the best tool for what I am trying to
>> > do? I'd really appreciate if others could share their experiences.
>> >
>>
>> okay, then it shouldn't be a problem. Can you try git version of
>> backgroundrb as suggest in following link and report us back?
>>
>>
>> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>>
>>
>>
> Hemant - Thanks for your help. I'll try this first on development and then
> report back. With this, will I be able to queue up 256+ or 1024+ b/g jobs?
>

Why, the newer version will fix many of the problem is because, newer
version makes use of fork and exec, rather than just fork(). What it
means is, each of worker process is totally independent of each other.
So, even if you are opening lots of files and network resources in
your workers, they stay completely independent of each other. Now,
Ruby runtime is not copy on write friendly and hence with plain fork
it copies all the resources in forked processes  and hence causing
some memory issues.

-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org