From scott at butlerpress.com  Tue Jun  3 23:23:06 2008
From: scott at butlerpress.com (Scott Willson)
Date: Tue, 3 Jun 2008 20:23:06 -0700
Subject: [Backgroundrb-devel] Rails Configuration in Workers
In-Reply-To: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>
References: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>
Message-ID: <4CAE7B1F-590D-4621-A3E5-C70675ED4796@butlerpress.com>

On May 4, 2008, at 4:09 PM, Scott Willson wrote:

> I override some Rails' configuration options in my project, like:
>    config.database_configuration_file = "#{RAILS_ROOT}/local/config/ 
> database.yml"
>
> But they aren't picked up in BackgroundRB because Rails ActiveRecord  
> is just initialized directly in MasterWorker#load_rails_env:
>      db_config_file = YAML.load(ERB.new(IO.read("#{RAILS_HOME}/ 
> config/database.yml")).result)

Hey folks,

So this looks easy enough. You just need to change  
MetaWorker#load_rails_env to load the entire Rails environment rather  
than just read the database config + ActiveRecord. Patch attached (I  
apologize for not doing a nifty Git fork/patch).

I'm a bit leery of side effects from this patch?your mileage may vary? 
but it does fix my issue and seems to work just fine in my app.

Scott

-------------- next part --------------
A non-text attachment was scrubbed...
Name: meta_worker_rails_env.diff
Type: application/octet-stream
Size: 945 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080603/5f83758b/attachment.obj>
-------------- next part --------------


From gethemant at gmail.com  Wed Jun  4 02:41:50 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 4 Jun 2008 12:11:50 +0530
Subject: [Backgroundrb-devel] Rails Configuration in Workers
In-Reply-To: <4CAE7B1F-590D-4621-A3E5-C70675ED4796@butlerpress.com>
References: <B2EDABB2-E3AA-4E86-BEE1-8AC8824691E4@butlerpress.com>
	<4CAE7B1F-590D-4621-A3E5-C70675ED4796@butlerpress.com>
Message-ID: <fb283d5e0806032341n794e1093t94fb5b4bf063d692@mail.gmail.com>

On Wed, Jun 4, 2008 at 8:53 AM, Scott Willson <scott at butlerpress.com> wrote:
> On May 4, 2008, at 4:09 PM, Scott Willson wrote:
>
>> I override some Rails' configuration options in my project, like:
>>   config.database_configuration_file =
>> "#{RAILS_ROOT}/local/config/database.yml"
>>
>> But they aren't picked up in BackgroundRB because Rails ActiveRecord is
>> just initialized directly in MasterWorker#load_rails_env:
>>     db_config_file =
>> YAML.load(ERB.new(IO.read("#{RAILS_HOME}/config/database.yml")).result)
>
> Hey folks,
>
> So this looks easy enough. You just need to change MetaWorker#load_rails_env
> to load the entire Rails environment rather than just read the database
> config + ActiveRecord. Patch attached (I apologize for not doing a nifty Git
> fork/patch).
>
> I'm a bit leery of side effects from this patch?your mileage may vary?but it
> does fix my issue and seems to work just fine in my app.

Can you please create this patch against git version :

http://github.com/gnufied/backgroundrb/tree/master

Here is a small, summary of changes in git version:


http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

From ronan at mmspos.com  Wed Jun  4 08:34:22 2008
From: ronan at mmspos.com (Ronan)
Date: Wed, 04 Jun 2008 09:34:22 -0300
Subject: [Backgroundrb-devel] Large requests break BackgrounDRb
Message-ID: <1212582862.3934.2.camel@comp27.mmslan.mms>

Hello, I think I may have discovered a bug in BackgrounDRb.  It seems that 
when the data being requested is large (via send_request), say in the range of 
64k+, it crashes or at least incapacitates BackgrounDRb.

For example, take these two workers:

class SmallWorker < BackgrounDRb::MetaWorker
  set_worker_name :small_worker
  set_no_auto_load(true)

  attr_reader :payload

  def create(args)
    @payload = 's ' * 512
  end
end

class LargeWorker < BackgrounDRb::MetaWorker
  set_worker_name :large_worker
  set_no_auto_load(true)

  attr_reader :payload

  def create(args)
    @payload = 'L' * 65600
  end
end


I'll start with SmallWorker.  Created by:

    @job_key = rand 1000 # Nevermind the weak job keys for now.
    MiddleMan.new_worker(:worker => :small_worker, :data => 'not used',
        :job_key => @job_key)

then retrieved with:

    @ret = MiddleMan.send_request(:worker => :small_worker,
        :job_key => @job_key, :worker_method => :payload)

SmallWorker returns about 1k worth of data in the @ret hash; and it will work 
every time.

Now, do the same thing with LargeWorker, which has a payload of larger than 
64k.  @ret will be just nil and BackgrounDRb will no longer accept requests.

This was done in an isolated rails project, with only the bare minimum of 
files created for testing purposes.

If this is simply a matter of me not setting something up properly, great, 
just let me know and problem solved.  Otherwise, I think we have a bug here.  

Given that it has the issue around the 64k mark, it would seem likely we have 
a 16bit number tracking something that needs at least a 32.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080604/9e710825/attachment.html>

From aaron.pfeifer at gmail.com  Wed Jun  4 17:57:44 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Wed, 4 Jun 2008 17:57:44 -0400
Subject: [Backgroundrb-devel] Handling exceptions
Message-ID: <2c7df3ed0806041457n4c189dbu80e5c8710b5efc59@mail.gmail.com>

Is there a convention for handling exceptions within BackgrounDRb workers?
For example, I have a worker which does some asynchronous data processing
(*not* a scheduled worker) based on some things in the database.  However,
if an exception is raised within that worker, then the process for that
worker is killed off.  Ideally I would want that worker to just silently
fail and be optimistic that it was an edge case that won't happen every time
the worker is invoked.

As of now, I've just added some basic exception handling to log the error
for future inspection like so:

class EmailWorker < BackgrounDRb::MetaWorker
  set_worker_name :email_worker

  def run(data = nil)
    do_stuff
  rescue Exception => ex
    logger.info ex
  end
end

Is this generally the practice for doing exception handling within workers
or is there another approach?

Regards,

Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080604/14054adf/attachment-0001.html>

From tandaraho at gmail.com  Thu Jun  5 23:00:12 2008
From: tandaraho at gmail.com (tanda raho)
Date: Thu, 5 Jun 2008 20:00:12 -0700
Subject: [Backgroundrb-devel] Unexpected NameError when invoking task on
	worker
Message-ID: <db8ae9450806052000n7fdfa31aw2303ea94ce4ce84e@mail.gmail.com>

Hi, I'm brand new to BackgrounDRb and trying to figure out the setup.

I cloned the git repository today as listed on the BackgrounDRb website.

Starting out with a very basic worker, but BackgroundDRb is crashing
on a NameError. Any help to resolve this issue would be greatly
appreciated! Thank you in advance!!

I am also attaching a text file with the output below since it might
be easier to see it in that format.

config/backgroundrb.yml
:backgroundrb:
  :ip: 0.0.0.0
  :port: 11009
  :environment: development
  :log: foreground


lib/workers/basic_worker.rb
class BasicWorker < BackgrounDRb::MetaWorker
  set_worker_name :basic_worker
  def create(args = nil)
    # this method is called, when worker is loaded for the first time
  end

  def keep_count
      count = 1
  end
end


Then I am starting BackgrounDRb and using the console to invoke the
worker, but run into this problem, and not sure what is going on:

[d96440a6 at d96440a6 ~/testapp]$ script/backgroundrb start
[d96440a6 at d96440a6 ~/testapp]$ ruby script/console
Loading development environment (Rails 2.0.2)
>> MiddleMan
=> #<BackgrounDRb::WorkerProxy:0x8675b6c @mutex=#<Mutex:0x8675b44>>
>> worker = MiddleMan.worker(:basic_worker)
=> #<BackgrounDRb::RailsWorkerProxy:0x995f78c @job_key=nil,
@middle_man=#<BackgrounDRb::WorkerProxy:0x8675b6c
@mutex=#<Mutex:0x8675b44>>, @worker_name=:basic_worker>
>> worker.keep_count
=> nil
>> /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:266:in `load_missing_constant': uninitialized constant Packet::BinParser (NameError)
        from /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in
`const_missing'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:170:in
`post_init'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_connection.rb:21:in
`invoke_init'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:302:in
`decorate_handler'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:76:in
`accept_connection'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:202:in
`handle_external_messages'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
`start_worker'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
`fork_and_load'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
`load_workers'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`load_workers'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
`start_worker'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
`fork_and_load'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
`load_workers'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
`load_workers'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42
/opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
`start_worker'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
`fork_and_load'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:108:in
`start_worker'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:191:in
`initialize'
        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:19:in
`run'
        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
`initialize'
        from script/backgroundrb:42:in `new'
        from script/backgroundrb:42
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: output.txt
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080605/8dfeab53/attachment.txt>

From gethemant at gmail.com  Fri Jun  6 05:00:18 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 6 Jun 2008 14:30:18 +0530
Subject: [Backgroundrb-devel] Handling exceptions
In-Reply-To: <2c7df3ed0806041457n4c189dbu80e5c8710b5efc59@mail.gmail.com>
References: <2c7df3ed0806041457n4c189dbu80e5c8710b5efc59@mail.gmail.com>
Message-ID: <fb283d5e0806060200p32a97593o736604da207c781f@mail.gmail.com>

On Thu, Jun 5, 2008 at 3:27 AM, Aaron Pfeifer <aaron.pfeifer at gmail.com> wrote:
> Is there a convention for handling exceptions within BackgrounDRb workers?
> For example, I have a worker which does some asynchronous data processing
> (*not* a scheduled worker) based on some things in the database.  However,
> if an exception is raised within that worker, then the process for that
> worker is killed off.  Ideally I would want that worker to just silently
> fail and be optimistic that it was an edge case that won't happen every time
> the worker is invoked.
>
> As of now, I've just added some basic exception handling to log the error
> for future inspection like so:
>
> class EmailWorker < BackgrounDRb::MetaWorker
>   set_worker_name :email_worker
>
>   def run(data = nil)
>     do_stuff
>   rescue Exception => ex
>     logger.info ex
>   end
> end
>
> Is this generally the practice for doing exception handling within workers
> or is there another approach?
>

Currently, there is no other way, but I think, it would be a good idea
to have a callback ( on_exception :do_this) and continue with things.

From gethemant at gmail.com  Fri Jun  6 05:01:15 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 6 Jun 2008 14:31:15 +0530
Subject: [Backgroundrb-devel] Unexpected NameError when invoking task on
	worker
In-Reply-To: <db8ae9450806052000n7fdfa31aw2303ea94ce4ce84e@mail.gmail.com>
References: <db8ae9450806052000n7fdfa31aw2303ea94ce4ce84e@mail.gmail.com>
Message-ID: <fb283d5e0806060201t38d2393eyea5afd6551e541b7@mail.gmail.com>

For running git version of BackgrounDRb, you will need git version of packet.

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/


On Fri, Jun 6, 2008 at 8:30 AM, tanda raho <tandaraho at gmail.com> wrote:
> Hi, I'm brand new to BackgrounDRb and trying to figure out the setup.
>
> I cloned the git repository today as listed on the BackgrounDRb website.
>
> Starting out with a very basic worker, but BackgroundDRb is crashing
> on a NameError. Any help to resolve this issue would be greatly
> appreciated! Thank you in advance!!
>
> I am also attaching a text file with the output below since it might
> be easier to see it in that format.
>
> config/backgroundrb.yml
> :backgroundrb:
>  :ip: 0.0.0.0
>  :port: 11009
>  :environment: development
>  :log: foreground
>
>
> lib/workers/basic_worker.rb
> class BasicWorker < BackgrounDRb::MetaWorker
>  set_worker_name :basic_worker
>  def create(args = nil)
>    # this method is called, when worker is loaded for the first time
>  end
>
>  def keep_count
>      count = 1
>  end
> end
>
>
> Then I am starting BackgrounDRb and using the console to invoke the
> worker, but run into this problem, and not sure what is going on:
>
> [d96440a6 at d96440a6 ~/testapp]$ script/backgroundrb start
> [d96440a6 at d96440a6 ~/testapp]$ ruby script/console
> Loading development environment (Rails 2.0.2)
>>> MiddleMan
> => #<BackgrounDRb::WorkerProxy:0x8675b6c @mutex=#<Mutex:0x8675b44>>
>>> worker = MiddleMan.worker(:basic_worker)
> => #<BackgrounDRb::RailsWorkerProxy:0x995f78c @job_key=nil,
> @middle_man=#<BackgrounDRb::WorkerProxy:0x8675b6c
> @mutex=#<Mutex:0x8675b44>>, @worker_name=:basic_worker>
>>> worker.keep_count
> => nil
>>> /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:266:in `load_missing_constant': uninitialized constant Packet::BinParser (NameError)
>        from /opt/local/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in
> `const_missing'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:170:in
> `post_init'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_connection.rb:21:in
> `invoke_init'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:302:in
> `decorate_handler'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:76:in
> `accept_connection'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:202:in
> `handle_external_messages'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
> `run'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
> `handle_internal_messages'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
> `start_worker'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
> `fork_and_load'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
> `load_workers'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `load_workers'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> `run'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
> `handle_internal_messages'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
> `start_worker'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
> `fork_and_load'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:96:in
> `load_workers'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:91:in
> `load_workers'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
> `run'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
> /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
> `handle_internal_messages'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
> `start_worker'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
> `fork_and_load'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:108:in
> `start_worker'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:191:in
> `initialize'
>        from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:19:in
> `run'
>        from /home/d96440a6/testapp/vendor/plugins/backgroundrb/server/lib/master_worker.rb:188:in
> `initialize'
>        from script/backgroundrb:42:in `new'
>        from script/backgroundrb:42
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From aaron.pfeifer at gmail.com  Fri Jun  6 12:18:58 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Fri, 6 Jun 2008 12:18:58 -0400
Subject: [Backgroundrb-devel] Handling exceptions
In-Reply-To: <fb283d5e0806060200p32a97593o736604da207c781f@mail.gmail.com>
References: <2c7df3ed0806041457n4c189dbu80e5c8710b5efc59@mail.gmail.com>
	<fb283d5e0806060200p32a97593o736604da207c781f@mail.gmail.com>
Message-ID: <2c7df3ed0806060918j2016cee4ya4f5833590254c8d@mail.gmail.com>

Thanks for the feedback... glad to know we've gone down the right path! :)

On Fri, Jun 6, 2008 at 5:00 AM, hemant <gethemant at gmail.com> wrote:

> On Thu, Jun 5, 2008 at 3:27 AM, Aaron Pfeifer <aaron.pfeifer at gmail.com>
> wrote:
> > Is there a convention for handling exceptions within BackgrounDRb
> workers?
> > For example, I have a worker which does some asynchronous data processing
> > (*not* a scheduled worker) based on some things in the database.
>  However,
> > if an exception is raised within that worker, then the process for that
> > worker is killed off.  Ideally I would want that worker to just silently
> > fail and be optimistic that it was an edge case that won't happen every
> time
> > the worker is invoked.
> >
> > As of now, I've just added some basic exception handling to log the error
> > for future inspection like so:
> >
> > class EmailWorker < BackgrounDRb::MetaWorker
> >   set_worker_name :email_worker
> >
> >   def run(data = nil)
> >     do_stuff
> >   rescue Exception => ex
> >     logger.info ex
> >   end
> > end
> >
> > Is this generally the practice for doing exception handling within
> workers
> > or is there another approach?
> >
>
> Currently, there is no other way, but I think, it would be a good idea
> to have a callback ( on_exception :do_this) and continue with things.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080606/abf5da97/attachment.html>

From aaron.pfeifer at gmail.com  Fri Jun  6 14:12:23 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Fri, 6 Jun 2008 14:12:23 -0400
Subject: [Backgroundrb-devel] Edge stability?
Message-ID: <2c7df3ed0806061112k4ddf51daud2344b00c74b5acf@mail.gmail.com>

Hey--

I've recently transitioned to the edge version of BackgrounDRb (2008-05-26)
and I was curious what the opinion was as to the stability of it.  Here's
been my experience with older and newer versions:

Originally I was using revision 286 from subversion (back on 2007-12-17, I
know I know it's old) and it was working fine for development purposes, but
I eventually started seeing very sporadic outages of bdrb (i.e. the process
would just die) when actually testing it with some amount of load.

I then moved to revision 1.0.1 (revision 311) since that's the last tagged
version that was publicly released.  While 1.0.1 removed the sporadic
outages that were happening and could handle a small amount of load, it
still could not handle the amount of work that I needed it to.

I've since moved to the latest revision on GitHub (at the time,
http://github.com/gnufied/backgroundrb/tree/ef202d0251e3ae0543cfaddcc084e0fd159fdd58)
on 2008-05-26.  This version has proved to be the most stable as of yet
based on my own load tests.  This means I'm also using the latest revision
of Packet on GitHub.  The load tests have run over a period of 24 hours and
I haven't seen any problems with BackgrounDRb.

Before I move forward and use this in production, I was wondering if there
were any known concerns or problems about the latest version on GitHub?  Is
there much holding this project back from tagging a new release?

Thanks for any thoughts.

Regards,

Aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080606/92b2df70/attachment.html>

From josephbenik at gmail.com  Sat Jun  7 06:33:40 2008
From: josephbenik at gmail.com (Joseph Benik)
Date: Sat, 7 Jun 2008 03:33:40 -0700
Subject: [Backgroundrb-devel] Process Request and dummy_result
Message-ID: <a0a33a2f0806070333h10f1f6d6jaaf701422ffeebec@mail.gmail.com>

I've only been working with backgroundrb a few days so I don't know
much of the procedures involved with its development....

But I came across what I consider a glitch in the latest GIT version
of the source.


The file server/lib/metaworker.rb at line 247

result = "dummy_result" unless result

should be changed to::

result = "dummy_result" if result.nil?


The problem is if a function returns FALSE it is presently switched to
"dummy_result" by this line ...  effectively changing the return from
FALSE to TRUE ...

From ian at telematter.com  Mon Jun  9 13:12:13 2008
From: ian at telematter.com (ian eyberg)
Date: Mon, 9 Jun 2008 12:12:13 -0500
Subject: [Backgroundrb-devel] cruisecontrol integration
Message-ID: <20080609171213.GA15467@hypatia>

Hi everyone.
 Backgroundrb is doing some really sweet stuff for my project. Thanks!

 I am trying to integrate backgroundrb with cruisecontrol.rb. I started
off by creating a simple Rakefile:::::

desc 'task for starting up backgroundrb'
task :cruise do
  template = <<-EOF

:backgroundrb:
  :ip: 0.0.0.0
  :port: 11006

:development:
  :backgroundrb:
    :log: foreground

:test:
  :backgroundrb:
    :log: foreground

:production:
  :backgroundrb:
    :log: foreground

EOF
  backgroundrb_configuration = ERB.new(template).result(binding)
  #put backgroundrb_configuration, "#{RAILS_ROOT}/config/backgroundrb.yml"

  bfile = File.new("#{RAILS_ROOT}/config/backgroundrb.yml", "w")
  bfile.puts backgroundrb_configuration
  sh "chmod 755 #{RAILS_ROOT}/config/backgroundrb.yml"

  begin
    def shutmedown
      sh "ruby script/backgroundrb stop"
    end

  rescue
    'got hereer.'
  end
      sh "ruby script/backgroundrb start"
end

----------------------------------------------
/home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_config.rb:38:in
`read_config': undefined method `[]' for false:FalseClass (NoMethodError)
        from
/home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:15:in
`init'
        from
/home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:175
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require'
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
`require'
        from
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in
`new_constants_in'
        from
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
`require'
        from
/home/ian/bluffware/backend/vendor/plugins/backgroundrb/init.rb:2:in
`evaluate_init_rb'
         ... 11 levels...
.......................


I looked at the line in question (~/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_config.rb:38) 
and added some output to see what is going on lines 28-32:::

  def self.read_config(config_file)
    #my debugging info..
    puts config_file
    puts File.exist?(config_file)
    puts IO.read(config_file)
    puts YAML.load(ERB.new(IO.read(config_file)).result)

    config = YAML.load(ERB.new(IO.read(config_file)).result)

    environment = RAILS_ENV.to_sym


File.exists says it sees my conf file but IO.read doesn't see anything. This is
strange cause I can repeat this process in irb and everything is there. Also,
the conf file is definitely getting generated cause it's there after the fact.

Does anyone see what I might be overlooking?

From ian at telematter.com  Tue Jun 10 09:49:45 2008
From: ian at telematter.com (ian eyberg)
Date: Tue, 10 Jun 2008 08:49:45 -0500
Subject: [Backgroundrb-devel] cruisecontrol integration
In-Reply-To: <20080609171213.GA15467@hypatia>
References: <20080609171213.GA15467@hypatia>
Message-ID: <20080610134945.GA23294@hypatia>

greets-
  I must've been off my rocker yesterday. I have it fixed and it's running now.
Now all my functional tests using backgroundrb are running again!

-Ian

On 12:12 Mon 09 Jun     , ian eyberg wrote:
> Hi everyone.
>  Backgroundrb is doing some really sweet stuff for my project. Thanks!
> 
>  I am trying to integrate backgroundrb with cruisecontrol.rb. I started
> off by creating a simple Rakefile:::::
> 
> desc 'task for starting up backgroundrb'
> task :cruise do
>   template = <<-EOF
> 
> :backgroundrb:
>   :ip: 0.0.0.0
>   :port: 11006
> 
> :development:
>   :backgroundrb:
>     :log: foreground
> 
> :test:
>   :backgroundrb:
>     :log: foreground
> 
> :production:
>   :backgroundrb:
>     :log: foreground
> 
> EOF
>   backgroundrb_configuration = ERB.new(template).result(binding)
>   #put backgroundrb_configuration, "#{RAILS_ROOT}/config/backgroundrb.yml"
> 
>   bfile = File.new("#{RAILS_ROOT}/config/backgroundrb.yml", "w")
>   bfile.puts backgroundrb_configuration
>   sh "chmod 755 #{RAILS_ROOT}/config/backgroundrb.yml"
> 
>   begin
>     def shutmedown
>       sh "ruby script/backgroundrb stop"
>     end
> 
>   rescue
>     'got hereer.'
>   end
>       sh "ruby script/backgroundrb start"
> end
> 
> ----------------------------------------------
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_config.rb:38:in
> `read_config': undefined method `[]' for false:FalseClass (NoMethodError)
>         from
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:15:in
> `init'
>         from
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:175
>         from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> `gem_original_require'
>         from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> `require'
>         from
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> `require'
>         from
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in
> `new_constants_in'
>         from
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> `require'
>         from
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/init.rb:2:in
> `evaluate_init_rb'
>          ... 11 levels...
> .......................

From aaron.pfeifer at gmail.com  Tue Jun 10 09:57:34 2008
From: aaron.pfeifer at gmail.com (Aaron Pfeifer)
Date: Tue, 10 Jun 2008 09:57:34 -0400
Subject: [Backgroundrb-devel] cruisecontrol integration
In-Reply-To: <20080610134945.GA23294@hypatia>
References: <20080609171213.GA15467@hypatia> <20080610134945.GA23294@hypatia>
Message-ID: <2c7df3ed0806100657i6e2731f9j283687a68a1941ab@mail.gmail.com>

For everyone else's benefit, was this a problem with your rake task?  Also,
is this the only thing you've added to integrate backgroundrb with
cruisecontrol.rb?  I would be really interested in doing this in my own
project.

On Tue, Jun 10, 2008 at 9:49 AM, ian eyberg <ian at telematter.com> wrote:

> greets-
>  I must've been off my rocker yesterday. I have it fixed and it's running
> now.
> Now all my functional tests using backgroundrb are running again!
>
> -Ian
>
> On 12:12 Mon 09 Jun     , ian eyberg wrote:
> > Hi everyone.
> >  Backgroundrb is doing some really sweet stuff for my project. Thanks!
> >
> >  I am trying to integrate backgroundrb with cruisecontrol.rb. I started
> > off by creating a simple Rakefile:::::
> >
> > desc 'task for starting up backgroundrb'
> > task :cruise do
> >   template = <<-EOF
> >
> > :backgroundrb:
> >   :ip: 0.0.0.0
> >   :port: 11006
> >
> > :development:
> >   :backgroundrb:
> >     :log: foreground
> >
> > :test:
> >   :backgroundrb:
> >     :log: foreground
> >
> > :production:
> >   :backgroundrb:
> >     :log: foreground
> >
> > EOF
> >   backgroundrb_configuration = ERB.new(template).result(binding)
> >   #put backgroundrb_configuration,
> "#{RAILS_ROOT}/config/backgroundrb.yml"
> >
> >   bfile = File.new("#{RAILS_ROOT}/config/backgroundrb.yml", "w")
> >   bfile.puts backgroundrb_configuration
> >   sh "chmod 755 #{RAILS_ROOT}/config/backgroundrb.yml"
> >
> >   begin
> >     def shutmedown
> >       sh "ruby script/backgroundrb stop"
> >     end
> >
> >   rescue
> >     'got hereer.'
> >   end
> >       sh "ruby script/backgroundrb start"
> > end
> >
> > ----------------------------------------------
> >
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_config.rb:38:in
> > `read_config': undefined method `[]' for false:FalseClass (NoMethodError)
> >         from
> >
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:15:in
> > `init'
> >         from
> >
> /home/ian/bluffware/backend/vendor/plugins/backgroundrb/lib/backgroundrb.rb:175
> >         from
> /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> > `gem_original_require'
> >         from
> /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> > `require'
> >         from
> >
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> > `require'
> >         from
> >
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:342:in
> > `new_constants_in'
> >         from
> >
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> > `require'
> >         from
> > /home/ian/bluffware/backend/vendor/plugins/backgroundrb/init.rb:2:in
> > `evaluate_init_rb'
> >          ... 11 levels...
> > .......................
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080610/3354c2f2/attachment-0001.html>

From mike at metaswitch.com  Tue Jun 10 10:42:36 2008
From: mike at metaswitch.com (Mike Evans)
Date: Tue, 10 Jun 2008 15:42:36 +0100
Subject: [Backgroundrb-devel] Backgroundrb fixes for transfering large
	amounts of data
Message-ID: <52A0FF47062B0B4C80862F2526E02409064F9DDD@enfimail2.datcon.co.uk>

Hemant

We've continued testing our application with backgroundrb and found a
couple of other problems when transfering large amounts of data.  Both
of these problems are still present in the github version of code.

The first problem is an exception in the Marshal.load call in the
receive_data method in the Packet::MetaPimp class.  The root cause is in
the BinParser module, in the arm of code handling the parser state 1
(reading in the data).  The issue is that, at the marked line of code
the pack_data array will be at most @numeric_length entries because of
the format string passed to the unpack call.  This results in the code
dropping a chunk of data and then hitting the exception in a subsequent
Marshal.load call.

      elsif @parser_state == 1
        pack_data,remaining = new_data.unpack("a#{@numeric_length}a*")
        if pack_data.length < @numeric_length
          @data << pack_data
          @numeric_length = @numeric_length - pack_data.length
        elsif pack_data.length == @numeric_length     <======== this
should be "elsif remaining.length == 0"
          @data << pack_data
          extracter_block.call(@data.join)
          @data = []
          @parser_state = 0
          @length_string = ""
          @numeric_length = 0
        else
          @data << pack_data
          extracter_block.call(@data.join)
          @data = []
          @parser_state = 0
          @length_string = ""
          @numeric_length = 0
          extract(remaining,&extracter_block)
        end
      end

The second problem we hit was ask_status repeatedly returning nil.  The
root cause of this problem is in the read_object method of the
BackgrounDRb::WorkerProxy class when a data record is large enough to
cause connection.read_nonblock to throw the Errno::EAGAIN exception
multiple times.  We changed the code to make sure read_nonblock is
called repeatedly until the tokenizer finds a complete record, and this
fixed the problem.

  def read_object
    begin
      while (true)
        sock_data = ""
        begin
          while(sock_data << @connection.read_nonblock(1023)); end
        rescue Errno::EAGAIN
          @tokenizer.extract(sock_data) { |b_data| return b_data }
        end
      end
    rescue
      raise BackgrounDRb::BdrbConnError.new("Not able to connect")
    end
  end

Regards, Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080610/d02a94bb/attachment.html>

From gethemant at gmail.com  Tue Jun 10 11:01:29 2008
From: gethemant at gmail.com (Hemant Kumar)
Date: Tue, 10 Jun 2008 20:31:29 +0530
Subject: [Backgroundrb-devel] Backgroundrb fixes for transfering large
	amounts of data
In-Reply-To: <52A0FF47062B0B4C80862F2526E02409064F9DDD@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E02409064F9DDD@enfimail2.datcon.co.uk>
Message-ID: <484E9749.1010003@gmail.com>

Mike Evans wrote:
>
> Hemant
>
> We've continued testing our application with backgroundrb and found a 
> couple of other problems when transfering large amounts of data.  Both 
> of these problems are still present in the github version of code.
>
> The first problem is an exception in the Marshal.load call in the 
> receive_data method in the Packet::MetaPimp class.  The root cause is 
> in the BinParser module, in the arm of code handling the parser state 
> 1 (reading in the data).  The issue is that, at the marked line of 
> code the pack_data array will be at most @numeric_length entries 
> because of the format string passed to the unpack call.  This results 
> in the code dropping a chunk of data and then hitting the exception in 
> a subsequent Marshal.load call.
>
>       elsif @parser_state == 1
>         pack_data,remaining = new_data.unpack("a#{@numeric_length}a*")
>         if pack_data.length < @numeric_length
>           @data << pack_data
>           @numeric_length = @numeric_length - pack_data.length
>         elsif pack_data.length == @numeric_length     <======== this 
> should be "elsif remaining.length == 0"
>           @data << pack_data
>           extracter_block.call(@data.join)
>           @data = []
>           @parser_state = 0
>           @length_string = ""
>           @numeric_length = 0
>         else
>           @data << pack_data
>           extracter_block.call(@data.join)
>           @data = []
>           @parser_state = 0
>           @length_string = ""
>           @numeric_length = 0
>           extract(remaining,&extracter_block)
>         end
>       end
>
> The second problem we hit was ask_status repeatedly returning nil.  
> The root cause of this problem is in the read_object method of the 
> BackgrounDRb::WorkerProxy class when a data record is large enough to 
> cause connection.read_nonblock to throw the Errno::EAGAIN exception 
> multiple times.  We changed the code to make sure read_nonblock is 
> called repeatedly until the tokenizer finds a complete record, and 
> this fixed the problem.
>
>   def read_object
>     begin
>       while (true)
>         sock_data = ""
>         begin
>           while(sock_data << @connection.read_nonblock(1023)); end
>         rescue Errno::EAGAIN
>           @tokenizer.extract(sock_data) { |b_data| return b_data }
>         end
>       end
>     rescue
>       raise BackgrounDRb::BdrbConnError.new("Not able to connect")
>     end
>   end
>
> Regards, Mike
>
If you update to THE latest github version of BackgrounDRb, you will 
find that above thing is already fixed and read is no more nonblocking 
for clients ( blocking read makes much more sense for clients ).

Also, I have made BinParser class to be iterative ( just for better 
stability, assuming your data is large enough to throw StackLevel Too 
deep errors), but thats not yet pushed to master version of packet. I 
will implement your fix tonight, when I push the changes to master 
repository of packet.




From mike at metaswitch.com  Tue Jun 10 11:06:58 2008
From: mike at metaswitch.com (Mike Evans)
Date: Tue, 10 Jun 2008 16:06:58 +0100
Subject: [Backgroundrb-devel] Backgroundrb fixes for transfering large
	amounts of data
In-Reply-To: <484E9749.1010003@gmail.com>
References: <52A0FF47062B0B4C80862F2526E02409064F9DDD@enfimail2.datcon.co.uk>
	<484E9749.1010003@gmail.com>
Message-ID: <52A0FF47062B0B4C80862F2526E02409064F9E20@enfimail2.datcon.co.uk>

Hemant

Thanks for the quick response - I'll admit I hadn't resynced for a week
or so!  Agreed that non-blocking reads make more sense for the client.

We're still testing with a patched version of packet-0.1.5 and the svn
release of backgroundrb because of the problems we hit getting RoR model
objects to load properly on the worker tasks with the latest code - have
you found a fix for this yet?

Mike 

-----Original Message-----
From: Hemant Kumar [mailto:gethemant at gmail.com] 
Sent: 10 June 2008 16:01
To: Mike Evans
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: Backgroundrb fixes for transfering large amounts of data

Mike Evans wrote:
>
> Hemant
>
> We've continued testing our application with backgroundrb and found a 
> couple of other problems when transfering large amounts of data.  Both

> of these problems are still present in the github version of code.
>
> The first problem is an exception in the Marshal.load call in the 
> receive_data method in the Packet::MetaPimp class.  The root cause is 
> in the BinParser module, in the arm of code handling the parser state
> 1 (reading in the data).  The issue is that, at the marked line of 
> code the pack_data array will be at most @numeric_length entries 
> because of the format string passed to the unpack call.  This results 
> in the code dropping a chunk of data and then hitting the exception in

> a subsequent Marshal.load call.
>
>       elsif @parser_state == 1
>         pack_data,remaining = new_data.unpack("a#{@numeric_length}a*")
>         if pack_data.length < @numeric_length
>           @data << pack_data
>           @numeric_length = @numeric_length - pack_data.length
>         elsif pack_data.length == @numeric_length     <======== this 
> should be "elsif remaining.length == 0"
>           @data << pack_data
>           extracter_block.call(@data.join)
>           @data = []
>           @parser_state = 0
>           @length_string = ""
>           @numeric_length = 0
>         else
>           @data << pack_data
>           extracter_block.call(@data.join)
>           @data = []
>           @parser_state = 0
>           @length_string = ""
>           @numeric_length = 0
>           extract(remaining,&extracter_block)
>         end
>       end
>
> The second problem we hit was ask_status repeatedly returning nil.  
> The root cause of this problem is in the read_object method of the 
> BackgrounDRb::WorkerProxy class when a data record is large enough to 
> cause connection.read_nonblock to throw the Errno::EAGAIN exception 
> multiple times.  We changed the code to make sure read_nonblock is 
> called repeatedly until the tokenizer finds a complete record, and 
> this fixed the problem.
>
>   def read_object
>     begin
>       while (true)
>         sock_data = ""
>         begin
>           while(sock_data << @connection.read_nonblock(1023)); end
>         rescue Errno::EAGAIN
>           @tokenizer.extract(sock_data) { |b_data| return b_data }
>         end
>       end
>     rescue
>       raise BackgrounDRb::BdrbConnError.new("Not able to connect")
>     end
>   end
>
> Regards, Mike
>
If you update to THE latest github version of BackgrounDRb, you will
find that above thing is already fixed and read is no more nonblocking
for clients ( blocking read makes much more sense for clients ).

Also, I have made BinParser class to be iterative ( just for better
stability, assuming your data is large enough to throw StackLevel Too
deep errors), but thats not yet pushed to master version of packet. I
will implement your fix tonight, when I push the changes to master
repository of packet.




From gethemant at gmail.com  Tue Jun 10 13:40:56 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 10 Jun 2008 23:10:56 +0530
Subject: [Backgroundrb-devel] Backgroundrb fixes for transfering large
	amounts of data
In-Reply-To: <52A0FF47062B0B4C80862F2526E02409064F9E20@enfimail2.datcon.co.uk>
References: <52A0FF47062B0B4C80862F2526E02409064F9DDD@enfimail2.datcon.co.uk>
	<484E9749.1010003@gmail.com>
	<52A0FF47062B0B4C80862F2526E02409064F9E20@enfimail2.datcon.co.uk>
Message-ID: <fb283d5e0806101040x47e036f0i4533b147c48687ae@mail.gmail.com>

On Tue, Jun 10, 2008 at 8:36 PM, Mike Evans <mike at metaswitch.com> wrote:
> Hemant
>
> Thanks for the quick response - I'll admit I hadn't resynced for a week
> or so!  Agreed that non-blocking reads make more sense for the client.

You meant blocking!

>
> We're still testing with a patched version of packet-0.1.5 and the svn
> release of backgroundrb because of the problems we hit getting RoR model
> objects to load properly on the worker tasks with the latest code - have
> you found a fix for this yet?
>

Yeah,  you will find model loading behavior much more reliable in
current git version of backgroundrb. Also, I did get in your fix, so
no need to worry. Basically, there are only two ways:

1. Get that damn regexp correct.
2. Load entire models, plugins everything explicitly, because just
loading environment.rb, doesn't load models by default in rails.

I went for option#1, to keep your bdrb worker process lean.
All, in all, with git version of bdrb and packet, everything should be
much more smoother.

From neilmock at gmail.com  Tue Jun 10 19:56:27 2008
From: neilmock at gmail.com (Neil Mock)
Date: Tue, 10 Jun 2008 18:56:27 -0500
Subject: [Backgroundrb-devel] adding results from threads to a collection
	and returning it
Message-ID: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>

Forgive me if this has been addressed somewhere, but I have searched and
can't come up with anything.

I am basically trying to distribute several web page scraping tasks among
different threads, and have the results from each added to an Array which is
ultimately returned by the backgroundrb worker.  Here is an example of what
I'm trying to do in a worker method:

     pages = Array.new

     pages_to_scrape.each do |url|
          thread_pool.defer(url) do |url|
            begin
              # model object performs the scraping
              page = ScrapedPage.new(page.url)
              pages << page
            rescue
              logger.info "page scrape failed"
            end
          end
        end
      end

    return pages

>From monitoring the backgroundrb logs, it appears that all of the pages are
completed successfully in the threads.  However, the array that is returned
is empty.  This is to be expected I suppose because the threads don't
complete before the array is returned, but my question is: how can I make
the worker wait to return the array only when all of the threads are
complete?

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080610/3eb915be/attachment-0001.html>

From gethemant at gmail.com  Tue Jun 10 23:35:33 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 11 Jun 2008 09:05:33 +0530
Subject: [Backgroundrb-devel] adding results from threads to a
	collection and returning it
In-Reply-To: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
References: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
Message-ID: <fb283d5e0806102035i9af6413y319007645a9e52d@mail.gmail.com>

On Wed, Jun 11, 2008 at 5:26 AM, Neil Mock <neilmock at gmail.com> wrote:
> Forgive me if this has been addressed somewhere, but I have searched and
> can't come up with anything.
>
> I am basically trying to distribute several web page scraping tasks among
> different threads, and have the results from each added to an Array which is
> ultimately returned by the backgroundrb worker.  Here is an example of what
> I'm trying to do in a worker method:
>
>      pages = Array.new
>
>      pages_to_scrape.each do |url|
>           thread_pool.defer(url) do |url|
>             begin
>               # model object performs the scraping
>               page = ScrapedPage.new(page.url)
>               pages << page
>             rescue
>               logger.info "page scrape failed"
>             end
>           end
>         end
>       end
>
>     return pages
>
> From monitoring the backgroundrb logs, it appears that all of the pages are
> completed successfully in the threads.  However, the array that is returned
> is empty.  This is to be expected I suppose because the threads don't
> complete before the array is returned, but my question is: how can I make
> the worker wait to return the array only when all of the threads are
> complete?
>

Actually, you are doing couple of things wrong. First, you are
accessing a variable that you created outside thread_pool from inside
of pool and hence have a big thread unsafe code, which can cause
anything from deadlocks to random crashes.

Thread pools are for running concurrent tasks in background without
any reporting, its for fire and forget kinda of thing. However, I am
contemplating some change in behaviour of thread pools, which will
enable what you want perhaps, so unless your need is dire, please
don't use thread pools like above snippet.

From gethemant at gmail.com  Thu Jun 12 15:54:01 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 13 Jun 2008 01:24:01 +0530
Subject: [Backgroundrb-devel] adding results from threads to a
	collection and returning it
In-Reply-To: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
References: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
Message-ID: <fb283d5e0806121254x1ed9bb7bu7a66256a2712f04@mail.gmail.com>

On Wed, Jun 11, 2008 at 5:26 AM, Neil Mock <neilmock at gmail.com> wrote:
> Forgive me if this has been addressed somewhere, but I have searched and
> can't come up with anything.
>
> I am basically trying to distribute several web page scraping tasks among
> different threads, and have the results from each added to an Array which is
> ultimately returned by the backgroundrb worker.  Here is an example of what
> I'm trying to do in a worker method:
>
>      pages = Array.new
>
>      pages_to_scrape.each do |url|
>           thread_pool.defer(url) do |url|
>             begin
>               # model object performs the scraping
>               page = ScrapedPage.new(page.url)
>               pages << page
>             rescue
>               logger.info "page scrape failed"
>             end
>           end
>         end
>       end
>
>     return pages
>
> From monitoring the backgroundrb logs, it appears that all of the pages are
> completed successfully in the threads.  However, the array that is returned
> is empty.  This is to be expected I suppose because the threads don't
> complete before the array is returned, but my question is: how can I make
> the worker wait to return the array only when all of the threads are
> complete?
>

Neil,

I have a solution for you in git version:

http://gnufied.org/2008/06/12/unthreaded-threads-of-hobbiton/

From stevie at slowbicycle.com  Fri Jun 13 04:45:29 2008
From: stevie at slowbicycle.com (Stevie Clifton)
Date: Fri, 13 Jun 2008 04:45:29 -0400
Subject: [Backgroundrb-devel] adding results from threads to a
	collection and returning it
In-Reply-To: <fb283d5e0806121254x1ed9bb7bu7a66256a2712f04@mail.gmail.com>
References: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
	<fb283d5e0806121254x1ed9bb7bu7a66256a2712f04@mail.gmail.com>
Message-ID: <a426f29f0806130145j47c223b1sb1c5be3df678ffe1@mail.gmail.com>

Hey Hemant,

A couple of questions about fetch_parallely:

1)  Does it operate in the same way as thread_pool.defer, where the
number of concurrent threads are limited by :pool_size?

2)  Why did you choose to introduce another method instead of
providing a thread-safe register_status? (more out of curiosity than
anything else -- in my code I've overridden register_status to use a
Mutex, and am wondering what the benefit of fetch_parallely would be
to this)

Thanks!

stevie

On Thu, Jun 12, 2008 at 3:54 PM, hemant <gethemant at gmail.com> wrote:
> On Wed, Jun 11, 2008 at 5:26 AM, Neil Mock <neilmock at gmail.com> wrote:
>> Forgive me if this has been addressed somewhere, but I have searched and
>> can't come up with anything.
>>
>> I am basically trying to distribute several web page scraping tasks among
>> different threads, and have the results from each added to an Array which is
>> ultimately returned by the backgroundrb worker.  Here is an example of what
>> I'm trying to do in a worker method:
>>
>>      pages = Array.new
>>
>>      pages_to_scrape.each do |url|
>>           thread_pool.defer(url) do |url|
>>             begin
>>               # model object performs the scraping
>>               page = ScrapedPage.new(page.url)
>>               pages << page
>>             rescue
>>               logger.info "page scrape failed"
>>             end
>>           end
>>         end
>>       end
>>
>>     return pages
>>
>> From monitoring the backgroundrb logs, it appears that all of the pages are
>> completed successfully in the threads.  However, the array that is returned
>> is empty.  This is to be expected I suppose because the threads don't
>> complete before the array is returned, but my question is: how can I make
>> the worker wait to return the array only when all of the threads are
>> complete?
>>
>
> Neil,
>
> I have a solution for you in git version:
>
> http://gnufied.org/2008/06/12/unthreaded-threads-of-hobbiton/
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Fri Jun 13 06:33:04 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 13 Jun 2008 16:03:04 +0530
Subject: [Backgroundrb-devel] adding results from threads to a
	collection and returning it
In-Reply-To: <a426f29f0806130145j47c223b1sb1c5be3df678ffe1@mail.gmail.com>
References: <3bd796950806101656g3c9aa5c5oa248a95f39907a1b@mail.gmail.com>
	<fb283d5e0806121254x1ed9bb7bu7a66256a2712f04@mail.gmail.com>
	<a426f29f0806130145j47c223b1sb1c5be3df678ffe1@mail.gmail.com>
Message-ID: <fb283d5e0806130333m18d4588co51826f46ba924149@mail.gmail.com>

On Fri, Jun 13, 2008 at 2:15 PM, Stevie Clifton <stevie at slowbicycle.com> wrote:
> Hey Hemant,
>
> A couple of questions about fetch_parallely:
>
> 1)  Does it operate in the same way as thread_pool.defer, where the
> number of concurrent threads are limited by :pool_size?
>
> 2)  Why did you choose to introduce another method instead of
> providing a thread-safe register_status? (more out of curiosity than
> anything else -- in my code I've overridden register_status to use a
> Mutex, and am wondering what the benefit of fetch_parallely would be
> to this)

register_status is going to invoke send_data at one point or another.
yeah sure, prolly i can make outbound_data instance variable protected
by a mutex, but thats going to slow down the whole operation by a
large margin. Thats not simply point of a event driven network
programming library.  It will mean that, I will have to check for
mutex, each time i write. What a waste of time it will be!

Now, on the other hand, if we can make sure that, we retrieve results
from thread pool in a thread safe manner and then invoke send_data,
everything is nice and dandy. fetch_parallely does exactly that. Name
is a bit dubious, i didn't want to break existing functionality and at
the same time, wanted to add this. Let me know, if you have better
name.

From walther at diechmann.net  Sat Jun 14 19:33:09 2008
From: walther at diechmann.net (Walther H Diechmann)
Date: Sun, 15 Jun 2008 01:33:09 +0200
Subject: [Backgroundrb-devel] does BackgrounDRB not "know" the rails
 environment it is running within/in parallel with?]
Message-ID: <48545535.10607@diechmann.net>

Hi,

it probably is a noob question but I'm confused by this error message:

active_support/dependencies.rb:478:in `const_missing': uninitialized 
constant Invoice::ApplicationController

Situation:
I have an InvoicesController with a create method that in turn calls 
upon a MiddleMan.worker(:billing_worker), which in turn finds an Invoice 
and calls upon  @invoice.print_invoice 
"public/forms/invoices/html/#{invoice.invoice_number}.html" to generate 
a html version of an invoice.

The class Invoice.rb has this print_invoice method:


 def print_invoice path_and_file
   htmlstring = self.render_to_string "invoices/show", :layout => false
   File.open(path_and_file, "w+") do |f|
     f.puts( htmlstring)
     f.close
   end
   true
 end

The self.render_to_string does look like this

   # Renders a record instance to a string using the provided template 
and additional variables.
   def render_to_string(template, variables={})
     assigns = 
variables.reverse_merge(self.class.class_name.underscore.to_sym => self)
     self.class.render_string(template, assigns)
   end  
and the self.class.render_string looks like this


   # Renders a template to a string with the included variable assignments
   def self.render_string(template, assigns)
     viewer = Class.new(ApplicationController)
     view = Class.new(ActionView::Base)
     view.send(:include, viewer.master_helper_module)
     path = ActionController::Base.view_paths rescue 
ActionController::Base.view_root
     view.new(path, assigns, viewer).render(template)      end

I have a show.html.erb under app/views/invoices which holds the HTML to 
go with the show action on the InvoicesController.

Everything is working perfectly if I open the script/console and calls 
invoice.print_invoice from there!

However, if I hand it over to (the) MiddleMan - I'm left with the above 
error message!

Does anybody know why the invoice.print_invoice does not work, when 
called in the context of billing_worker?


cheers,
walt



class BillingWorker < BackgrounDRb::MetaWorker
 set_worker_name :billing_worker
 def create(args = nil)
   # this method is called, when worker is loaded for the first time
   logger.info "Faktura baggrundsjob startet!"
 end

 def build_invoice(invoice_id = nil)
   invoice = Invoice.find(invoice_id)  unless invoice_id.nil?
   logger.info "Faktura ID #{invoice_id} skal dannes!"

   unless invoice.nil?
     result = false
     logger.info "Faktura ID #{invoice_id} dannes!"
     result = invoice.print_invoice 
"public/forms/invoices/html/#{invoice.invoice_number}.html"
     #result = %x[script/get_invoice #{invoice.id} 
#{invoice.invoice_number}]
     if result
       logger.info "Faktura nr #{invoice.invoice_number} blev dannet - 
som HTML!"
       result = %x[script/get_invoice_pdf #{invoice.invoice_number} 
#{invoice.invoice_number}]
       #result=false
       if result
         logger.info "Faktura nr #{invoice.invoice_number} blev dannet - 
som PDF!"
         invoice.pdf_resource = 
"forms/invoices/pdf/#{invoice.invoice_number}.pdf"
         invoice.project.invoiced= invoice.all_invoiced?
         invoice.project.new_contact_id=-1 # we will not be needing the 
find/create services of project.rb
         invoice.project.delivery_at_date = nil
         invoice.project.save
         logger.info "Projekt til faktura nr #{invoice.invoice_number} 
blev opdateret!"
         invoice.save
         invoice.invoice_items.each { |ii| 
Task.find(ii.task_id).update_attribute( :invoiced_at, DateTime.now) 
unless ii.task_id.nil? }
         logger.info "Faktura nr #{invoice.invoice_number} blev dannet 
med faktura linier!"
       else
         logger.info "Faktura med ID #{invoice_id} blev ikke dannet - 
som PDF!"
       end
     else
       logger.info "Faktura med ID #{invoice_id} blev ikke dannet - som 
HTML!"
     end
   else
     logger.info "Faktura med ID #{invoice_id} blev ikke fundet!"
   end
 end

end


-------------- next part --------------
An embedded message was scrubbed...
From: Walther H Diechmann <walt at alco.dk>
Subject: does BackgrounDRB not "know" the rails environment it is running
	within/in parallel with?
Date: Sun, 15 Jun 2008 01:21:45 +0200
Size: 4468
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080615/141cea80/attachment.eml>

From raghu.srinivasan at gmail.com  Sat Jun 14 20:29:07 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Sat, 14 Jun 2008 17:29:07 -0700
Subject: [Backgroundrb-devel] What does this error mean?
Message-ID: <6482c06a0806141729qa40a3ebp811af2179552a7db@mail.gmail.com>

Below is the content of my backgroundrb_server_11007.log
This seems to happen in a non-repeatable way - so every 2 days or sometimes
after 4-5 days, my b/g process dies with this error. Is this something
anyone else has seen - any fixes?
Thanks,
Raghu
===============true

/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
`schedule_write': You have a nil object when you didn't expect it!
(NoMethodError)
The error occurred while evaluating nil.instance        from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
`write_and_schedule'
       from
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in
`each_with_index'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each_with_index'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`write_and_schedule'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
`dump_object'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:31:in
`send_data'
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:40:in
`send_request'
        ... 22 levels...
       from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in
`run'
       from /home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
       from ./script/backgroundrb:42:in `new'
       from ./script/backgroundrb:42

================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080614/80595415/attachment-0001.html>

From ian.lesperance at gmail.com  Mon Jun 16 14:28:08 2008
From: ian.lesperance at gmail.com (Ian Lesperance)
Date: Mon, 16 Jun 2008 11:28:08 -0700
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
Message-ID: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>

I've noticed that I can never just restart BackgrounDRb by itself.  As soon
as I do that, Mongrel can no longer pass requests off to it.  The only way
to get it working again is by restarting Mongrel.  As annoying as that may
be, I suppose I could justify it to myself, since BackgrounDRb is
essentially resetting itself and its connections.

But what seems odd is that Mongrel (or, rather, MiddleMan) doesn't seem to
notice that it's requests aren't getting through.  That is, no errors are
ever thrown.  The requests just die silently.

Is this the expected behavior?  Shouldn't MiddleMan notice that it's
connection to BackgrounDRb just got severed?  Or am I missing something?

Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080616/e131d2ab/attachment.html>

From gethemant at gmail.com  Mon Jun 16 21:34:05 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 17 Jun 2008 07:04:05 +0530
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
Message-ID: <fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>

On Mon, Jun 16, 2008 at 11:58 PM, Ian Lesperance
<ian.lesperance at gmail.com> wrote:
> I've noticed that I can never just restart BackgrounDRb by itself.  As soon
> as I do that, Mongrel can no longer pass requests off to it.  The only way
> to get it working again is by restarting Mongrel.  As annoying as that may
> be, I suppose I could justify it to myself, since BackgrounDRb is
> essentially resetting itself and its connections.
>
> But what seems odd is that Mongrel (or, rather, MiddleMan) doesn't seem to
> notice that it's requests aren't getting through.  That is, no errors are
> ever thrown.  The requests just die silently.
>
> Is this the expected behavior?  Shouldn't MiddleMan notice that it's
> connection to BackgrounDRb just got severed?  Or am I missing something?
>

Are you running svn version of the plugin?

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

In git version essentially a new clean connection is made for each
request and hence you shouldn't see above mentioned problem. Please
try upgrading and let me know, how it went?

From webmaster at wolniartysci.pl  Wed Jun 18 09:04:05 2008
From: webmaster at wolniartysci.pl (Wojciech Knapik)
Date: Wed, 18 Jun 2008 15:04:05 +0200
Subject: [Backgroundrb-devel] A fix for bdrb crash when running in background
Message-ID: <485907C5.2010009@wolniartysci.pl>


Hi everyone

  I just spent way too much time trying to figure out why the fsck bdrb 
wouldn't run in background (while it worked perfecly in foreground), so 
I figured I'll share my unimpressive solution to save others the trouble.
  So the problem was that bdrb, when started with `script/backgroundrb 
start`, would exit, with this error msg:

/usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in 
`log': PGError: server closed the connection unexpectedly 
(ActiveRecord::StatementInvalid)
         This probably means the server terminated abnormally
         before or while processing the request.
:           SELECT tablename
             FROM pg_tables
            WHERE schemaname IN (E'"$user"',E'public')
         from 
/usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:395:in 
`query'
         from 
/usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:441:in 
`tables'

  Starting it with `script/backgroundrb` didn't cause that problem, 
everything worked fine.
  This happened exactly the same way on a G5/OSX/Postgres 8.2.4/Rails 
2.0.2 workstation and a Xeon/Gentoo/Postgres 8.3.1/Rails 2.0.2 server. 
Considering the different setups and the complete lack of complaints 
from Postgres, even with highest logging verbosity AND the smoothly 
working rails app, I figured it must be a problem with backgroundrb, but 
whatever changes I tried, nothing worked. So fast forward (to spare you 
the story of the cursing and the hopeless googling) - here's the 
solution - line 32 of script/backgroubdrb:

  29 case ARGV[0]
  30 when 'start'
  31   if fork
  32     sleep(5)
  33     exit
  34   else

  Apparently leaving the child on it's own too early is not a good thing :-o
  So there. Here's my good deed for the day.

cheers,
Wojtek

PS. Thanks guys for bdrb!


From gethemant at gmail.com  Wed Jun 18 13:32:20 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 18 Jun 2008 23:02:20 +0530
Subject: [Backgroundrb-devel] A fix for bdrb crash when running in
	background
In-Reply-To: <485907C5.2010009@wolniartysci.pl>
References: <485907C5.2010009@wolniartysci.pl>
Message-ID: <fb283d5e0806181032y78e6069etd4c1df19cee411dc@mail.gmail.com>

On Wed, Jun 18, 2008 at 6:34 PM, Wojciech Knapik
<webmaster at wolniartysci.pl> wrote:
>
> Hi everyone
>
>  I just spent way too much time trying to figure out why the fsck bdrb
> wouldn't run in background (while it worked perfecly in foreground), so I
> figured I'll share my unimpressive solution to save others the trouble.
>  So the problem was that bdrb, when started with `script/backgroundrb
> start`, would exit, with this error msg:
>
> /usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
> `log': PGError: server closed the connection unexpectedly
> (ActiveRecord::StatementInvalid)
>        This probably means the server terminated abnormally
>        before or while processing the request.
> :           SELECT tablename
>            FROM pg_tables
>           WHERE schemaname IN (E'"$user"',E'public')
>        from
> /usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:395:in
> `query'
>        from
> /usr/lib64/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:441:in
> `tables'
>
>  Starting it with `script/backgroundrb` didn't cause that problem,
> everything worked fine.
>  This happened exactly the same way on a G5/OSX/Postgres 8.2.4/Rails 2.0.2
> workstation and a Xeon/Gentoo/Postgres 8.3.1/Rails 2.0.2 server. Considering
> the different setups and the complete lack of complaints from Postgres, even
> with highest logging verbosity AND the smoothly working rails app, I figured
> it must be a problem with backgroundrb, but whatever changes I tried,
> nothing worked. So fast forward (to spare you the story of the cursing and
> the hopeless googling) - here's the solution - line 32 of
> script/backgroubdrb:
>
>  29 case ARGV[0]
>  30 when 'start'
>  31   if fork
>  32     sleep(5)
>  33     exit
>  34   else
>
>  Apparently leaving the child on it's own too early is not a good thing :-o
>  So there. Here's my good deed for the day.
>


Damn right! Thanks man.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From ian.lesperance at gmail.com  Wed Jun 18 16:09:12 2008
From: ian.lesperance at gmail.com (Ian Lesperance)
Date: Wed, 18 Jun 2008 13:09:12 -0700
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
	<fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
Message-ID: <8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>

I am hesitant to jump onto the edge, since this is being used in a
production application.  Also, since I believe I've managed to eliminate our
reason for needing to restart BackgrounDRb, it's more of an inconvenience at
this point.

Do you have plans to tag a release anytime soon, be it through Subversion or
Git?

Ian

On Mon, Jun 16, 2008 at 6:34 PM, hemant <gethemant at gmail.com> wrote:

> On Mon, Jun 16, 2008 at 11:58 PM, Ian Lesperance
> <ian.lesperance at gmail.com> wrote:
> > I've noticed that I can never just restart BackgrounDRb by itself.  As
> soon
> > as I do that, Mongrel can no longer pass requests off to it.  The only
> way
> > to get it working again is by restarting Mongrel.  As annoying as that
> may
> > be, I suppose I could justify it to myself, since BackgrounDRb is
> > essentially resetting itself and its connections.
> >
> > But what seems odd is that Mongrel (or, rather, MiddleMan) doesn't seem
> to
> > notice that it's requests aren't getting through.  That is, no errors are
> > ever thrown.  The requests just die silently.
> >
> > Is this the expected behavior?  Shouldn't MiddleMan notice that it's
> > connection to BackgrounDRb just got severed?  Or am I missing something?
> >
>
> Are you running svn version of the plugin?
>
>
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>
> In git version essentially a new clean connection is made for each
> request and hence you shouldn't see above mentioned problem. Please
> try upgrading and let me know, how it went?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080618/75c83b4f/attachment.html>

From gethemant at gmail.com  Wed Jun 18 16:31:53 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 19 Jun 2008 02:01:53 +0530
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
	<fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
	<8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>
Message-ID: <fb283d5e0806181331x749b9b50v1fc5cc920f4c0520@mail.gmail.com>

On Thu, Jun 19, 2008 at 1:39 AM, Ian Lesperance
<ian.lesperance at gmail.com> wrote:
> I am hesitant to jump onto the edge, since this is being used in a
> production application.  Also, since I believe I've managed to eliminate our
> reason for needing to restart BackgrounDRb, it's more of an inconvenience at
> this point.
>
> Do you have plans to tag a release anytime soon, be it through Subversion or
> Git?

I am not committing a time line, but I intend to release new version
this weekend if everything I planned goes well. Also, I hear from
folks( and in my own experience), git version is a _lot_ more stable.
If you don't want to change in production at least try in production.

From gethemant at gmail.com  Wed Jun 18 16:44:46 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 19 Jun 2008 02:14:46 +0530
Subject: [Backgroundrb-devel] Please comment on upcoming changes of
	backgroundrb
Message-ID: <fb283d5e0806181344oe23604alfd08fa74cb703ef1@mail.gmail.com>

Folks,

I am getting ready for a new release of BackgrounDRb. It will be
mostly tagging of git release which is being used in production. Not
to mention, Packet and BackgrounDRb has seen a _lot_ of improvement
and fixes since last release.

Also, there are few changes that I want to introduce, please comment on them:

1. As posted earlier - A way of running threads inside backgroundrb
and yet able to use result saving and stuff like that easily.
Currently method is named fetch_parallely which i am planning to
rename to run_concurrently. #defer stays as it is.

http://gnufied.org/2008/06/12/unthreaded-threads-of-hobbiton/

2. Ability to cluster and connect to multiple backgroundrb servers. It
involves some additions to backgroundrb.yml like:

# following section is totally optional, and only useful if you are
trying to cluster of backgroundrb server
# if you do not specify this section backgroundrb will assume that,
from rails you are connecting to the
# backgroundrb server which has been specified in previous section
:client:
  :drb_servers: "10.0.0.1:11006,10.0.0.2:11007"

So, when you say:

MiddleMan.worker(:hello_worker).fooo(@user)

BackgrounDRb will delegate the task to your backgroundrb servers in
round robin manner. By using:

MiddleMan.worker(:hello_worker)

you can one specific instance of worker which is tied to one
particular server. Also, #new_worker will be working a round robin
manner, but you must call #delete on returned object.

3. With clustering, there comes the question of storing worker
results. So far, backgroundrb result storage has been a bit of
problem. In new version, I am planning to rename ask_status to
ask_result. Also, register_status will be deprecated and for storing
results, you will have to use:

cache[some_key]  = result

where cache is local cache in your worker.  note that, here you are
defining a key for your result. But in memory process storage of
results won't work if backgroundrb servers are clustered and hence,
you will have to use memcache based storage if you are going to
cluster your workers. mechanism will be the same, you will specify the
key which will be combined with worker_name and worker_key.

Also, job_key wherever used will be replaced with worker_key since I
find that name confusing.

Thats all folks. Please try git version. Stress test it and let me
know about any problems.

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From jjolma at gmail.com  Thu Jun 19 09:24:00 2008
From: jjolma at gmail.com (Jeff Jolma)
Date: Thu, 19 Jun 2008 09:24:00 -0400
Subject: [Backgroundrb-devel] how to run multiple instances of the same
	worker
Message-ID: <9432da220806190624o789a224qfeb1d968953ceb73@mail.gmail.com>

Hi all.

How do I run multiple instances of the same worker in one BackgrounDRB
instance.  My worker sits in a loop polling a job queue and hitting a web
service.  Since this is so network bound, I'd like to run multiple instances
of the worker.

Thanks,
Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080619/29ebab39/attachment.html>

From gethemant at gmail.com  Thu Jun 19 09:33:37 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 19 Jun 2008 19:03:37 +0530
Subject: [Backgroundrb-devel] how to run multiple instances of the same
	worker
In-Reply-To: <9432da220806190624o789a224qfeb1d968953ceb73@mail.gmail.com>
References: <9432da220806190624o789a224qfeb1d968953ceb73@mail.gmail.com>
Message-ID: <fb283d5e0806190633j4538446cse69c117abf2ac05f@mail.gmail.com>

On Thu, Jun 19, 2008 at 6:54 PM, Jeff Jolma <jjolma at gmail.com> wrote:
> Hi all.
>
> How do I run multiple instances of the same worker in one BackgrounDRB
> instance.  My worker sits in a loop polling a job queue and hitting a web
> service.  Since this is so network bound, I'd like to run multiple instances
> of the worker.
>

MiddleMan.new_worker(:worker => <worker_name>,:job_key => "og boy")

From rue at thinlayer.co.uk  Thu Jun 19 09:16:10 2008
From: rue at thinlayer.co.uk (Rue Turner)
Date: Thu, 19 Jun 2008 14:16:10 +0100
Subject: [Backgroundrb-devel] job queuing?
Message-ID: <200806191416.10847.rue@thinlayer.co.uk>

Hi,

Did I hear correctly that bgrb has a job queuing engine built in so you can 
inject jobs into a queue and the bgrb pick them up and get on with the work?

If so can anyone point me at any docs or if not, what solutions are you using?

thanks

From gethemant at gmail.com  Thu Jun 19 10:16:35 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 19 Jun 2008 19:46:35 +0530
Subject: [Backgroundrb-devel] job queuing?
In-Reply-To: <200806191416.10847.rue@thinlayer.co.uk>
References: <200806191416.10847.rue@thinlayer.co.uk>
Message-ID: <fb283d5e0806190716w8ad2cefv5eb6d4b392ea9c4b@mail.gmail.com>

On Thu, Jun 19, 2008 at 6:46 PM, Rue Turner <rue at thinlayer.co.uk> wrote:
> Hi,
>
> Did I hear correctly that bgrb has a job queuing engine built in so you can
> inject jobs into a queue and the bgrb pick them up and get on with the work?
>
> If so can anyone point me at any docs or if not, what solutions are you using?
>
> thanks

Not exactly how others do, but mechanism is something like this:

class HelloWorker < BackgrounDRb::MetaWorker
  set_worker_name :hello_worker
  def enque_task args
     thread_pool.defer(args) do |block_args|
        # do some work here
    end
  end
end

>From Controller:

MiddleMan.worker(:hello_worker).enque_task(some_data)

So depending upon your thread pool size, that many of tasks will run
concurrently and rest of them will be in queue.

From stevie at slowbicycle.com  Fri Jun 20 11:22:29 2008
From: stevie at slowbicycle.com (Stevie Clifton)
Date: Fri, 20 Jun 2008 11:22:29 -0400
Subject: [Backgroundrb-devel] Please comment on upcoming changes of
	backgroundrb
In-Reply-To: <fb283d5e0806181344oe23604alfd08fa74cb703ef1@mail.gmail.com>
References: <fb283d5e0806181344oe23604alfd08fa74cb703ef1@mail.gmail.com>
Message-ID: <a426f29f0806200822u2b191f74m23face277c0a965c@mail.gmail.com>

Hey Hemant,

This is great.  Thanks for putting the effort to getting the git
version tagged so people feel comfortable using it in production!  I'm
assuming when you say that you'll tag the git release you'll also push
out new bdrd and packet gems, correct?

I had a quick question about fetch_parallely/run_concurrently.  Is it
basically doing the same thing as defer(), but just with a thread-safe
callback?  If so, it might be more intuitive to make the name of the
method reflect that, such as "defer_concurrently" so as not to make
people think that they do completely different things.

Also, if the above is true (run_concurrently is a thread-safe version
of defer), I think it would be less cumbersome if you just extended
defer() to allow the same functionality by passing the name of a
callback method in an options hash instead of creating an entirely new
method.  This way people could just use the block passed to defer() as
the request_proc, and another method at the worker level for the
response_proc instead of creating a bunch of procs to pass to the
method.  For example, instead of:

run_concurrently(args, request_proc, response_proc)

You could do:

def example_worker(args)
  defer(args, :callback => :my_callback_method) |args|
    # same as the body of request_proc
  end
end

def my_callback_method(result)
  # same as body of response_proc (call register_status or whatever)
end

I looked through the code in git for fetch_parallely, and I'm not sure
if this would give you the same flexibility, but it would be a more
intuitive solution to a user IMHO.

stevie

On Wed, Jun 18, 2008 at 4:44 PM, hemant <gethemant at gmail.com> wrote:
> Folks,
>
> I am getting ready for a new release of BackgrounDRb. It will be
> mostly tagging of git release which is being used in production. Not
> to mention, Packet and BackgrounDRb has seen a _lot_ of improvement
> and fixes since last release.
>
> Also, there are few changes that I want to introduce, please comment on them:
>
> 1. As posted earlier - A way of running threads inside backgroundrb
> and yet able to use result saving and stuff like that easily.
> Currently method is named fetch_parallely which i am planning to
> rename to run_concurrently. #defer stays as it is.
>
> http://gnufied.org/2008/06/12/unthreaded-threads-of-hobbiton/
>
> 2. Ability to cluster and connect to multiple backgroundrb servers. It
> involves some additions to backgroundrb.yml like:
>
> # following section is totally optional, and only useful if you are
> trying to cluster of backgroundrb server
> # if you do not specify this section backgroundrb will assume that,
> from rails you are connecting to the
> # backgroundrb server which has been specified in previous section
> :client:
>  :drb_servers: "10.0.0.1:11006,10.0.0.2:11007"
>
> So, when you say:
>
> MiddleMan.worker(:hello_worker).fooo(@user)
>
> BackgrounDRb will delegate the task to your backgroundrb servers in
> round robin manner. By using:
>
> MiddleMan.worker(:hello_worker)
>
> you can one specific instance of worker which is tied to one
> particular server. Also, #new_worker will be working a round robin
> manner, but you must call #delete on returned object.
>
> 3. With clustering, there comes the question of storing worker
> results. So far, backgroundrb result storage has been a bit of
> problem. In new version, I am planning to rename ask_status to
> ask_result. Also, register_status will be deprecated and for storing
> results, you will have to use:
>
> cache[some_key]  = result
>
> where cache is local cache in your worker.  note that, here you are
> defining a key for your result. But in memory process storage of
> results won't work if backgroundrb servers are clustered and hence,
> you will have to use memcache based storage if you are going to
> cluster your workers. mechanism will be the same, you will specify the
> key which will be combined with worker_name and worker_key.
>
> Also, job_key wherever used will be replaced with worker_key since I
> find that name confusing.
>
> Thats all folks. Please try git version. Stress test it and let me
> know about any problems.
>
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Fri Jun 20 13:26:14 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 20 Jun 2008 22:56:14 +0530
Subject: [Backgroundrb-devel] Please comment on upcoming changes of
	backgroundrb
In-Reply-To: <a426f29f0806200822u2b191f74m23face277c0a965c@mail.gmail.com>
References: <fb283d5e0806181344oe23604alfd08fa74cb703ef1@mail.gmail.com>
	<a426f29f0806200822u2b191f74m23face277c0a965c@mail.gmail.com>
Message-ID: <fb283d5e0806201026k6b8cede4r4e51c5193d2aa3ec@mail.gmail.com>

On Fri, Jun 20, 2008 at 8:52 PM, Stevie Clifton <stevie at slowbicycle.com> wrote:
> Hey Hemant,
>
> This is great.  Thanks for putting the effort to getting the git
> version tagged so people feel comfortable using it in production!  I'm
> assuming when you say that you'll tag the git release you'll also push
> out new bdrd and packet gems, correct?
>
> I had a quick question about fetch_parallely/run_concurrently.  Is it
> basically doing the same thing as defer(), but just with a thread-safe
> callback?  If so, it might be more intuitive to make the name of the
> method reflect that, such as "defer_concurrently" so as not to make
> people think that they do completely different things.
>
> Also, if the above is true (run_concurrently is a thread-safe version
> of defer), I think it would be less cumbersome if you just extended
> defer() to allow the same functionality by passing the name of a
> callback method in an options hash instead of creating an entirely new
> method.  This way people could just use the block passed to defer() as
> the request_proc, and another method at the worker level for the
> response_proc instead of creating a bunch of procs to pass to the
> method.  For example, instead of:
>
> run_concurrently(args, request_proc, response_proc)
>
> You could do:
>
> def example_worker(args)
>  defer(args, :callback => :my_callback_method) |args|
>    # same as the body of request_proc
>  end
> end
>
> def my_callback_method(result)
>  # same as body of response_proc (call register_status or whatever)
> end
>
> I looked through the code in git for fetch_parallely, and I'm not sure
> if this would give you the same flexibility, but it would be a more
> intuitive solution to a user IMHO.
>

Well, thanks for looking up. Actually I made quite bit of changes and
pushed an update to git on testcase branch.

http://github.com/gnufied/backgroundrb/commits/testcase

Well, I take your advice for run_concurrent method. But I have made
result storage completely thread safe and hence user can call it from
threads without any worries.

Actually, I made quite a bit of API changes today:

Here is a sample API, if you try new branch:

http://pastie.org/218967

Or

# Run a task like:

# this is our dear remote process
class HelloWorker
  set_worker_name :hello_worker
  def barbar t_user
    # runs method some_task inside thread pool
    thread_pool.defer(t_user,method(:some_task))
  end

  def some_task user
    user_feeds = user.feeds
    loop do
      # user can retrieve/add/edit objects from result/cache without
worrying about thread safety
      # when you call job_key in your threads it automatically
resolves to job_key for the task being executed
      # there may be another task being executed in another thread,
but since job_key is Thread local variable
      # it will always resolve to the correct one.
      old_counter = cache[job_key]
      cache[job_key] += 10
    end
  end
end

# invoke tasks/methods in worker async
MiddleMan.worker(:hello_worker).async_barbar(<some_job_key>, at user)

# invoke method in worker sync
MiddleMan.worker(:hello_worker).barbar(<some_job_key>, at user)

# ask the result object stored with job_key "wow"
MiddleMan.worker(:hello_worker).ask_result(<job_key_or_key_with_which_you_stored_result>)
# If you are doing this in production, i will strongly advise to use
mecache for result storage


# there is an issue if user invokes multiple tasks in thread pool
directly from one of the worker
# under current settings they are going to end up with same job key

# Also, new_worker can't have same method invocation conventions
because it accepts more parameters.


Again, MiddleMan wraps basically a cluster of bdrb servers now and when you say:

MiddleMan.worker(:hello_worker).async_barbar(<some_job_key>, at user)

It will be invoked in a round robin manner in all the bdrb servers
specified in configuration file. Complete configuration file looks
like:

# A Sample YAML configuration file
---
:backgroundrb:
  :ip: 0.0.0.0 #ip on which backgroundrb server is running
  :port: 11006 #port on which backgroundrb server is running
  :environment: production # rails environment loaded, defaults to development
  :debug_log: true # whether to print debug logs to a seperate worker,
defaults to true
  :log: foreground # will print log messages to STDOUT, defaults to
seperate log worker
  :result_storage: memcache # store results in a mecache cluster, you
also need to specify location of your memcache clusters in next
section

:memcache: "10.0.0.1:11211,10.0.0.2:11211" #=> location of mecache
clusters seperated by comma

# following section is totally optional, and only useful if you are
trying to cluster of backgroundrb server
# if you do not specify this section backgroundrb will assume that,
from rails you are connecting to the
# backgroundrb server which has been specified in previous section
:client:
  :drb_servers: "10.0.0.1:11006,10.0.0.2:11007"

# You specify your worker schedules here
:schedules:
  :foo_worker: # worker name
    :barbar: #worker method
      :trigger_args: */5 * * * * * * #worker schedule



Please comment on this.

From joost at spacebabies.nl  Fri Jun 20 18:21:46 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Sat, 21 Jun 2008 00:21:46 +0200
Subject: [Backgroundrb-devel] Any luck using Rails 2.1 config.gem?
Message-ID: <76986433-F229-4A7C-9213-12C3ACCAF6F1@spacebabies.nl>

I've seen Ticket #92 where a load error in the chronic gem pops up.

In that ticket is a mention of using gemsonrails to freeze gems in  
vendor/gems. I am using the built-in functionality in Rails 2.1 to do  
that; using the config.gem statement in environment.rb.

However when I do this, those gems are not available to backgroundrb:

ruby script/backgroundrb start
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/ 
1.8/rubygems/custom_require.rb:27:in `gem_original_require': no such  
file to load -- packet (LoadError)
	from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ 
ruby/1.8/rubygems/custom_require.rb:27:in `require'
	from script/backgroundrb:16


What road should I travel in finding a solution:

1 - manually traverse the vendor/gems/* dirs and include all gems there
2 - leaving this to Rails, so load and initialize Rails first (no idea  
how)
3 - never bother freezing these gems and keep them in the system  
instead.


I am tempted to travel road #3 since chronic, packet and hoe are only  
required for backgroundrb and have nothing to do with my Rails app.  
But some hosters might find it problematic to install gems.


Any ideas?

From joost at spacebabies.nl  Fri Jun 20 18:52:40 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Sat, 21 Jun 2008 00:52:40 +0200
Subject: [Backgroundrb-devel] Rails Configuration in Workers
Message-ID: <9ADB5A7E-5BAA-4C41-80B6-E9E95F44B25F@spacebabies.nl>

@Scott Wilson: this relates to my previous email.

I applied the patch you attached to http://rubyforge.org/pipermail/backgroundrb-devel/2008-June/001800.html

This has no effect w/regards to the gems I unpacked using config.gem.

I think the load path is unaffected until Rails actually boots...

Joost.

From gethemant at gmail.com  Fri Jun 20 22:56:24 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 21 Jun 2008 08:26:24 +0530
Subject: [Backgroundrb-devel] Rails Configuration in Workers
In-Reply-To: <9ADB5A7E-5BAA-4C41-80B6-E9E95F44B25F@spacebabies.nl>
References: <9ADB5A7E-5BAA-4C41-80B6-E9E95F44B25F@spacebabies.nl>
Message-ID: <fb283d5e0806201956uf0dacc9oc2b872e2d5c9b6f6@mail.gmail.com>

On Sat, Jun 21, 2008 at 4:22 AM, Joost Baaij <joost at spacebabies.nl> wrote:
> @Scott Wilson: this relates to my previous email.
>
> I applied the patch you attached to
> http://rubyforge.org/pipermail/backgroundrb-devel/2008-June/001800.html
>
> This has no effect w/regards to the gems I unpacked using config.gem.
>
> I think the load path is unaffected until Rails actually boots...
>

Any idea, how to reproduce this behavior? Problem, is, i have never
been able to reproduce this in my setup. Did you had gems installed
through gemsonrails?

From f.schwach at uea.ac.uk  Tue Jun 24 13:26:55 2008
From: f.schwach at uea.ac.uk (Frank Schwach)
Date: Tue, 24 Jun 2008 18:26:55 +0100
Subject: [Backgroundrb-devel] best approach to managing workers
	and	getting status
In-Reply-To: <f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
Message-ID: <1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>

Jack,
I just found your interesting post in the archive and I would like to
come back to this. I need to implement something like this:

I have some very long running tasks (several hours) that should run on a
remote machine and talk to the database on the Rails server. I need to
keep track of jobs including those that have been run in the past, so a
table for background jobs with their status as you describe would be the
best solution for me. 

I am just wondering whether backgroundrb wouldn't be a bit of an
overkill in the scenario you describe? In the new "Advanced Rails
Recipes" from the Pragmatic Programmers Bookshelf there is a recipe
using a simple daemonized ruby process that polls the database for
pending jobs and uses acts_as_state_machine to set the state of the jobs
(there is also a nice BackgrounDRb recipe in the book by the way). 
I am just wondering if the daemonized process isn't easier to handle in
this case since you don't integrate your app with backgroundrb very
tightly anyway?

I would be grateful for any suggestions because there seem to be lots of
possible solutions for this problem and some more or less well
documented plugins and I haven't used any of them before. I need a
simple and robust method that doesn't have too many dependencies and
doesn't require too much maintenance because I want to make the finished
app available for others to install on their local systems. 

Thanks in advance,

Frank

On Mon, 2008-05-05 at 10:35 +0200, Jack Nutting wrote:
> The way that I (and perhaps many others here?) handle this is to use
>  the database to hold a "work queue", and have a long-running worker
>  that polls the database periodically and handles any pending requests.
>   In your case for example, you could have an AudioFileWork model,
>  containing fields for "release_id" and "pending";  Your app would
>  create a new instance of that with pending=true, and your background
>  worker would poll for rows where pending=true, then mark them as false
>  when they are complete.  For your post-creation interactions (to show
>  the status with ajax) you'd use the id of the created AudioFileWork
>  instead of a job key.
> 
>  There are many advantages to this approach:
> 
>  - you always have a known number of workers (the number you configure
>  and start), so you won't have uncontrollable memory usage explosions
>  if your site gets busy, just slower response times
>  - you can check the status of a request by querying the database
>  - there is very little in-memory data that is lost in unhappy events
>  (a crash or unhandled exception)
>  - you have a historical record in the databse of all work that is
>  completed by a background worker
>  - you get a very loose coupling to backgroundrb.  In my case, I simply
>  specify in a model class that it needs processing, and it just
>  happens.  My app doesn't "talk" directly with backgroundrb at all,
>  except for an admin page where I can make sure it's up and running.
> 
> 
> 


From jnutting at gmail.com  Wed Jun 25 03:32:56 2008
From: jnutting at gmail.com (Jack Nutting)
Date: Wed, 25 Jun 2008 09:32:56 +0200
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
Message-ID: <f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>

On Tue, Jun 24, 2008 at 7:26 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> Jack,
> I just found your interesting post in the archive and I would like to
> come back to this. I need to implement something like this:
>
> I have some very long running tasks (several hours) that should run on a
> remote machine and talk to the database on the Rails server. I need to
> keep track of jobs including those that have been run in the past, so a
> table for background jobs with their status as you describe would be the
> best solution for me.
>
> I am just wondering whether backgroundrb wouldn't be a bit of an
> overkill in the scenario you describe? In the new "Advanced Rails
> Recipes" from the Pragmatic Programmers Bookshelf there is a recipe
> using a simple daemonized ruby process that polls the database for
> pending jobs and uses acts_as_state_machine to set the state of the jobs
> (there is also a nice BackgrounDRb recipe in the book by the way).
> I am just wondering if the daemonized process isn't easier to handle in
> this case since you don't integrate your app with backgroundrb very
> tightly anyway?
>
> I would be grateful for any suggestions because there seem to be lots of
> possible solutions for this problem and some more or less well
> documented plugins and I haven't used any of them before. I need a
> simple and robust method that doesn't have too many dependencies and
> doesn't require too much maintenance because I want to make the finished
> app available for others to install on their local systems.

This is an interesting question, Frank.  My usage of backgroundrb is
somewhat of an edge case, and most of what I'm doing with it could
definitely be done with a simpler system.  I initially chose
backgroundrb for my project because it seemed to make the most sense
at the time (for what I *thought* I needed; actual needs changed with
further exploration of the problem space), and I was enough of a ruby
newbie that it felt comfortable for me to have a packaged solution
that (mostly) "just worked".  If I were starting from scratch today, I
might make a different decision.

However, it's not only inertia that keeps me using backgroundrb.  For
one thing, backgroundrb does provide some handy things--centralized
logging, IPC for storing runtime status info about my processes,
etc--that would take some time for me to implement if I were rolling
my own solutions with a daemonized script, and from my perspective
that would be wasted time, since I have those things working today
thanks to backgroundrb.  Another reason for me to keep it is that I
have a few spots in my system where I'm considering using some of
backgroundrb's other key features, like launching a short-lived
process to handle something in response to some action happening in
the main application

-- 
// jack
// http://www.nuthole.com

From gethemant at gmail.com  Wed Jun 25 06:26:22 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 25 Jun 2008 15:56:22 +0530
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
Message-ID: <fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>

On Wed, Jun 25, 2008 at 1:02 PM, Jack Nutting <jnutting at gmail.com> wrote:
> On Tue, Jun 24, 2008 at 7:26 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
>> Jack,
>> I just found your interesting post in the archive and I would like to
>> come back to this. I need to implement something like this:
>>
>> I have some very long running tasks (several hours) that should run on a
>> remote machine and talk to the database on the Rails server. I need to
>> keep track of jobs including those that have been run in the past, so a
>> table for background jobs with their status as you describe would be the
>> best solution for me.
>>
>> I am just wondering whether backgroundrb wouldn't be a bit of an
>> overkill in the scenario you describe? In the new "Advanced Rails
>> Recipes" from the Pragmatic Programmers Bookshelf there is a recipe
>> using a simple daemonized ruby process that polls the database for
>> pending jobs and uses acts_as_state_machine to set the state of the jobs
>> (there is also a nice BackgrounDRb recipe in the book by the way).
>> I am just wondering if the daemonized process isn't easier to handle in
>> this case since you don't integrate your app with backgroundrb very
>> tightly anyway?
>>
>> I would be grateful for any suggestions because there seem to be lots of
>> possible solutions for this problem and some more or less well
>> documented plugins and I haven't used any of them before. I need a
>> simple and robust method that doesn't have too many dependencies and
>> doesn't require too much maintenance because I want to make the finished
>> app available for others to install on their local systems.
>
> This is an interesting question, Frank.  My usage of backgroundrb is
> somewhat of an edge case, and most of what I'm doing with it could
> definitely be done with a simpler system.  I initially chose
> backgroundrb for my project because it seemed to make the most sense
> at the time (for what I *thought* I needed; actual needs changed with
> further exploration of the problem space), and I was enough of a ruby
> newbie that it felt comfortable for me to have a packaged solution
> that (mostly) "just worked".  If I were starting from scratch today, I
> might make a different decision.
>
> However, it's not only inertia that keeps me using backgroundrb.  For
> one thing, backgroundrb does provide some handy things--centralized
> logging, IPC for storing runtime status info about my processes,
> etc--that would take some time for me to implement if I were rolling
> my own solutions with a daemonized script, and from my perspective
> that would be wasted time, since I have those things working today
> thanks to backgroundrb.  Another reason for me to keep it is that I
> have a few spots in my system where I'm considering using some of
> backgroundrb's other key features, like launching a short-lived
> process to handle something in response to some action happening in
> the main application
>

Well, I am working on couple of  new things with BackgrounDRb. Result
storage and retrieval is one of them,as I mentioned in earlier mails
and solicited opinions from fellows who are using bdrb. You can
checkout

http://github.com/gnufied/backgroundrb/commits/testcase

So whats there on this branch of BackgrounDRb which will become master
very soon.

1> True clustering system for clustering backgroundrb servers running
on N nodes. Tasks are dispatched in a round robin manner, but you can
specify the host on which you want execute task:

MiddleMan.worker(:foo_worker).async_some_work(:args => "lol")

^^ will choose any server in a round robin manner and run "some_work"
method in the specified worker. You can also specify:
:host => <local or all or "10.0.0.6:11001">

which overrides the default behaviour and run specified method on
local bdrb server, all bdrb server or specified server.

2> Clustering is failsafe and if one bdrb node goes down, all the
requests are immediately started to being routed to remaining servers.
Once that node comes up, it automatically starts participating in
clustering process.

3> Results can be stored in memcache and register_status method has
been replace by a "cache" object available in all workers. Hence you
can cache results with:

cache[@user.id] = some_data

in your workers and later you can retrieve results using:

MiddleMan.worker(:foo_worker).ask_result(@user.id)

I will seriously recommend using memcache if you are clustering bdrb
servers. Also, cache object's caching mechanism is completely thread
safe and hence can be used from within the thread pool or anywhere you
want.

4> Apart from memory based job queue that you can use with thread
pools, testcase branch implements database based job queues. So, to
enquue a particular task:

MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)

some_task method will be automatically called in first availbable
worker and task will be dequed from database.Also, jobs with duplicate
keys automatically get rejected.

Note that, above things are already working on test case branch. I
think, these features make bdrb a very compelling choice.

Some things that I will finish in a day or two:

5> Similar to worker method invocation, with each scheduled method,
you can specify host on which this task should run. For example, if
you have 5 bdrb servers and you have scheduled billing task to run
every sunday. Now, you don't want billing task to run on sunday on all
the servers. So, by default scheduled task will run on the server on
which its been created but you can specify host on which it should
run.

From f.schwach at uea.ac.uk  Wed Jun 25 07:00:16 2008
From: f.schwach at uea.ac.uk (Frank Schwach)
Date: Wed, 25 Jun 2008 12:00:16 +0100
Subject: [Backgroundrb-devel] best approach to managing workers
	and	getting status
In-Reply-To: <fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
Message-ID: <1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>

Hemant,
These latest additions to backgroundrb look pretty cool. Unfortunately,
I don't think I will be able to use it this way because in my setup I
can't run anything on the cluster nodes directly. I have to submit jobs
to a queuing system on the cluster's master node, which is why I think a
simple daemon running on the master node that polls the (remote) db for
pending jobs and then submits these to the queue would probably be
better for my case - but I'm far from being an expert on distributed
systems so any suggestions are very welcome!


On Wed, 2008-06-25 at 15:56 +0530, hemant wrote:
> On Wed, Jun 25, 2008 at 1:02 PM, Jack Nutting <jnutting at gmail.com> wrote:
> > On Tue, Jun 24, 2008 at 7:26 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> >> Jack,
> >> I just found your interesting post in the archive and I would like to
> >> come back to this. I need to implement something like this:
> >>
> >> I have some very long running tasks (several hours) that should run on a
> >> remote machine and talk to the database on the Rails server. I need to
> >> keep track of jobs including those that have been run in the past, so a
> >> table for background jobs with their status as you describe would be the
> >> best solution for me.
> >>
> >> I am just wondering whether backgroundrb wouldn't be a bit of an
> >> overkill in the scenario you describe? In the new "Advanced Rails
> >> Recipes" from the Pragmatic Programmers Bookshelf there is a recipe
> >> using a simple daemonized ruby process that polls the database for
> >> pending jobs and uses acts_as_state_machine to set the state of the jobs
> >> (there is also a nice BackgrounDRb recipe in the book by the way).
> >> I am just wondering if the daemonized process isn't easier to handle in
> >> this case since you don't integrate your app with backgroundrb very
> >> tightly anyway?
> >>
> >> I would be grateful for any suggestions because there seem to be lots of
> >> possible solutions for this problem and some more or less well
> >> documented plugins and I haven't used any of them before. I need a
> >> simple and robust method that doesn't have too many dependencies and
> >> doesn't require too much maintenance because I want to make the finished
> >> app available for others to install on their local systems.
> >
> > This is an interesting question, Frank.  My usage of backgroundrb is
> > somewhat of an edge case, and most of what I'm doing with it could
> > definitely be done with a simpler system.  I initially chose
> > backgroundrb for my project because it seemed to make the most sense
> > at the time (for what I *thought* I needed; actual needs changed with
> > further exploration of the problem space), and I was enough of a ruby
> > newbie that it felt comfortable for me to have a packaged solution
> > that (mostly) "just worked".  If I were starting from scratch today, I
> > might make a different decision.
> >
> > However, it's not only inertia that keeps me using backgroundrb.  For
> > one thing, backgroundrb does provide some handy things--centralized
> > logging, IPC for storing runtime status info about my processes,
> > etc--that would take some time for me to implement if I were rolling
> > my own solutions with a daemonized script, and from my perspective
> > that would be wasted time, since I have those things working today
> > thanks to backgroundrb.  Another reason for me to keep it is that I
> > have a few spots in my system where I'm considering using some of
> > backgroundrb's other key features, like launching a short-lived
> > process to handle something in response to some action happening in
> > the main application
> >
> 
> Well, I am working on couple of  new things with BackgrounDRb. Result
> storage and retrieval is one of them,as I mentioned in earlier mails
> and solicited opinions from fellows who are using bdrb. You can
> checkout
> 
> http://github.com/gnufied/backgroundrb/commits/testcase
> 
> So whats there on this branch of BackgrounDRb which will become master
> very soon.
> 
> 1> True clustering system for clustering backgroundrb servers running
> on N nodes. Tasks are dispatched in a round robin manner, but you can
> specify the host on which you want execute task:
> 
> MiddleMan.worker(:foo_worker).async_some_work(:args => "lol")
> 
> ^^ will choose any server in a round robin manner and run "some_work"
> method in the specified worker. You can also specify:
> :host => <local or all or "10.0.0.6:11001">
> 
> which overrides the default behaviour and run specified method on
> local bdrb server, all bdrb server or specified server.
> 
> 2> Clustering is failsafe and if one bdrb node goes down, all the
> requests are immediately started to being routed to remaining servers.
> Once that node comes up, it automatically starts participating in
> clustering process.
> 
> 3> Results can be stored in memcache and register_status method has
> been replace by a "cache" object available in all workers. Hence you
> can cache results with:
> 
> cache[@user.id] = some_data
> 
> in your workers and later you can retrieve results using:
> 
> MiddleMan.worker(:foo_worker).ask_result(@user.id)
> 
> I will seriously recommend using memcache if you are clustering bdrb
> servers. Also, cache object's caching mechanism is completely thread
> safe and hence can be used from within the thread pool or anywhere you
> want.
> 
> 4> Apart from memory based job queue that you can use with thread
> pools, testcase branch implements database based job queues. So, to
> enquue a particular task:
> 
> MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)
> 
> some_task method will be automatically called in first availbable
> worker and task will be dequed from database.Also, jobs with duplicate
> keys automatically get rejected.
> 
> Note that, above things are already working on test case branch. I
> think, these features make bdrb a very compelling choice.
> 
> Some things that I will finish in a day or two:
> 
> 5> Similar to worker method invocation, with each scheduled method,
> you can specify host on which this task should run. For example, if
> you have 5 bdrb servers and you have scheduled billing task to run
> every sunday. Now, you don't want billing task to run on sunday on all
> the servers. So, by default scheduled task will run on the server on
> which its been created but you can specify host on which it should
> run.
-- 
+++++++++++++++++++++++++++++++
Dr Frank Schwach
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ
Tel: 0044/(0)1603 - 592 405
www.cmp.uea.ac.uk
++++++++++++++++++++++++++++++++

From gethemant at gmail.com  Wed Jun 25 07:20:48 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 25 Jun 2008 16:50:48 +0530
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
Message-ID: <fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>

On Wed, Jun 25, 2008 at 4:30 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> Hemant,
> These latest additions to backgroundrb look pretty cool. Unfortunately,
> I don't think I will be able to use it this way because in my setup I
> can't run anything on the cluster nodes directly. I have to submit jobs
> to a queuing system on the cluster's master node, which is why I think a
> simple daemon running on the master node that polls the (remote) db for
> pending jobs and then submits these to the queue would probably be
> better for my case - but I'm far from being an expert on distributed
> systems so any suggestions are very welcome!

Hmm, so use db queuing mechanism inbuilt in bdrb. bdrb still stays
lightweight because most of these changes aren't affecting core and
have really went into client side of code.

MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)

Above snippet does exactly that. But anyways, i think you feel its too
complicated for your setup? I can't help that feeling. Its
complicated, if its complicated to setup and use, but its not. Those
features totally stay out of your way, if you don't need them.

From dan at revelationglobal.com  Wed Jun 25 17:33:51 2008
From: dan at revelationglobal.com (Dan Herrera)
Date: Wed, 25 Jun 2008 14:33:51 -0700
Subject: [Backgroundrb-devel] Too many clients already
Message-ID: <b89b94f00806251433m7693393cue22cf4aefbea47f9@mail.gmail.com>

Greetings,

We are currently working on offloading some long running tasks to
backgroundrb, and we are using the thread_pool.defer method.  Our
intent originally was to have one worker running, and have it process
requests as they came in in threads, so that we don't have a lot of
these workers running concurrently.

It seems in our testing that backgroundrb is not closing these worker
threads as they finish, and will eventually stop queueing new requests
with the message:  FATAL:  sorry, too many clients already.

Since we intend to open just one worker, and have it queue and process
requests as they come in inside the thread pool, how do we close these
threads once we are finished?  Or am I misunderstanding how this model
should work, and we should be instantiating a new worker per request?

Here is the code we are using:

http://pastie.org/private/jnyues8lwlhvoqbmqd8w

Thanks,

dan

From gethemant at gmail.com  Wed Jun 25 22:35:05 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 26 Jun 2008 08:05:05 +0530
Subject: [Backgroundrb-devel] Too many clients already
In-Reply-To: <b89b94f00806251433m7693393cue22cf4aefbea47f9@mail.gmail.com>
References: <b89b94f00806251433m7693393cue22cf4aefbea47f9@mail.gmail.com>
Message-ID: <fb283d5e0806251935l70d042a5i6264a03f51cbcc2c@mail.gmail.com>

On Thu, Jun 26, 2008 at 3:03 AM, Dan Herrera <dan at revelationglobal.com> wrote:
> Greetings,
>
> We are currently working on offloading some long running tasks to
> backgroundrb, and we are using the thread_pool.defer method.  Our
> intent originally was to have one worker running, and have it process
> requests as they came in in threads, so that we don't have a lot of
> these workers running concurrently.
>
> It seems in our testing that backgroundrb is not closing these worker
> threads as they finish, and will eventually stop queueing new requests
> with the message:  FATAL:  sorry, too many clients already.
>
> Since we intend to open just one worker, and have it queue and process
> requests as they come in inside the thread pool, how do we close these
> threads once we are finished?  Or am I misunderstanding how this model
> should work, and we should be instantiating a new worker per request?
>
> Here is the code we are using:
>
> http://pastie.org/private/jnyues8lwlhvoqbmqd8w
>

Those worker threads are always active in thread pool ( or rather
waiting for new tasks in queue object, enqueued via #defer method) and
hence there is no point in stopping them, once they are done, since if
no work is there in queue, they will anyways sleep.

On which line you are getting the error? Can you paste full exception?
Could be some issue with Queue class.

From dan at revelationglobal.com  Thu Jun 26 12:32:41 2008
From: dan at revelationglobal.com (Dan Herrera)
Date: Thu, 26 Jun 2008 09:32:41 -0700
Subject: [Backgroundrb-devel] Too many clients already
In-Reply-To: <fb283d5e0806251935l70d042a5i6264a03f51cbcc2c@mail.gmail.com>
References: <b89b94f00806251433m7693393cue22cf4aefbea47f9@mail.gmail.com>
	<fb283d5e0806251935l70d042a5i6264a03f51cbcc2c@mail.gmail.com>
Message-ID: <b89b94f00806260932i396195c7ta6ede278e9ac54e5@mail.gmail.com>

On Wed, Jun 25, 2008 at 7:35 PM, hemant <gethemant at gmail.com> wrote:
> On Thu, Jun 26, 2008 at 3:03 AM, Dan Herrera <dan at revelationglobal.com> wrote:
>> Greetings,
>>
>> We are currently working on offloading some long running tasks to
>> backgroundrb, and we are using the thread_pool.defer method.  Our
>> intent originally was to have one worker running, and have it process
>> requests as they came in in threads, so that we don't have a lot of
>> these workers running concurrently.
>>
>> It seems in our testing that backgroundrb is not closing these worker
>> threads as they finish, and will eventually stop queueing new requests
>> with the message:  FATAL:  sorry, too many clients already.
>>
>> Since we intend to open just one worker, and have it queue and process
>> requests as they come in inside the thread pool, how do we close these
>> threads once we are finished?  Or am I misunderstanding how this model
>> should work, and we should be instantiating a new worker per request?
>>
>> Here is the code we are using:
>>
>> http://pastie.org/private/jnyues8lwlhvoqbmqd8w
>>
>
> Those worker threads are always active in thread pool ( or rather
> waiting for new tasks in queue object, enqueued via #defer method) and
> hence there is no point in stopping them, once they are done, since if
> no work is there in queue, they will anyways sleep.
>
> On which line you are getting the error? Can you paste full exception?
> Could be some issue with Queue class.

Here is the full trace:

http://pastie.org/222713

We get this error when running the test several times in a row.  It
could be that our test is not properly closing the connection to
backgroundrb.  The test reports that the error occurs in the
before(:all) block in an RSpec.

Thanks for your help!

dan

From reynard.list at gmail.com  Thu Jun 26 20:31:19 2008
From: reynard.list at gmail.com (Reynard)
Date: Thu, 26 Jun 2008 20:31:19 -0400
Subject: [Backgroundrb-devel] zombie packet_worker
Message-ID: <26e148920806261731j9a1d920y8353da97a78f8d29@mail.gmail.com>

Hi,

I have just upgraded to the latest version, and now I have problem with lots
of packet_worker process hanging around

26820 pts/3    Z      0:03 [packet_worker_r] <defunct>

basically, I always create a new worker and call exit after it's done, so
multiple process can be running at the same time. with the old version
(before using packet), it deletes the process after it's done, now it
doesn't seem to delete it. has the API change? I cannot find anything in the
website documentation.
This is my worker code:

def create(args = nil)
  # do work
  exit
end


- reynard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080626/7c493f10/attachment.html>

From reynard.list at gmail.com  Thu Jun 26 21:17:32 2008
From: reynard.list at gmail.com (Reynard)
Date: Thu, 26 Jun 2008 21:17:32 -0400
Subject: [Backgroundrb-devel] zombie packet_worker
In-Reply-To: <26e148920806261731j9a1d920y8353da97a78f8d29@mail.gmail.com>
References: <26e148920806261731j9a1d920y8353da97a78f8d29@mail.gmail.com>
Message-ID: <26e148920806261817m5f615a14n975d438216da66e3@mail.gmail.com>

I think I just solved it, it deletes itself if I don't call exit.

- reynard


On Thu, Jun 26, 2008 at 8:31 PM, Reynard <reynard.list at gmail.com> wrote:

> Hi,
>
> I have just upgraded to the latest version, and now I have problem with
> lots of packet_worker process hanging around
>
> 26820 pts/3    Z      0:03 [packet_worker_r] <defunct>
>
> basically, I always create a new worker and call exit after it's done, so
> multiple process can be running at the same time. with the old version
> (before using packet), it deletes the process after it's done, now it
> doesn't seem to delete it. has the API change? I cannot find anything in the
> website documentation.
> This is my worker code:
>
> def create(args = nil)
>   # do work
>   exit
> end
>
>
> - reynard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080626/385717f9/attachment.html>

From reynard.list at gmail.com  Thu Jun 26 23:06:37 2008
From: reynard.list at gmail.com (Reynard)
Date: Thu, 26 Jun 2008 23:06:37 -0400
Subject: [Backgroundrb-devel] zombie packet_worker
In-Reply-To: <26e148920806261817m5f615a14n975d438216da66e3@mail.gmail.com>
References: <26e148920806261731j9a1d920y8353da97a78f8d29@mail.gmail.com>
	<26e148920806261817m5f615a14n975d438216da66e3@mail.gmail.com>
Message-ID: <26e148920806262006t11b232f5rdff62394a0bd11da@mail.gmail.com>

hmm actually it doesn't. how can I prevent packet_worker defunct process
hanging around?

- reynard

On Thu, Jun 26, 2008 at 9:17 PM, Reynard <reynard.list at gmail.com> wrote:

> I think I just solved it, it deletes itself if I don't call exit.
>
> - reynard
>
>
> On Thu, Jun 26, 2008 at 8:31 PM, Reynard <reynard.list at gmail.com> wrote:
>
>> Hi,
>>
>> I have just upgraded to the latest version, and now I have problem with
>> lots of packet_worker process hanging around
>>
>> 26820 pts/3    Z      0:03 [packet_worker_r] <defunct>
>>
>> basically, I always create a new worker and call exit after it's done, so
>> multiple process can be running at the same time. with the old version
>> (before using packet), it deletes the process after it's done, now it
>> doesn't seem to delete it. has the API change? I cannot find anything in the
>> website documentation.
>> This is my worker code:
>>
>> def create(args = nil)
>>   # do work
>>   exit
>> end
>>
>>
>> - reynard
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080626/8faaae46/attachment.html>

From gethemant at gmail.com  Fri Jun 27 03:12:43 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 27 Jun 2008 12:42:43 +0530
Subject: [Backgroundrb-devel] zombie packet_worker
In-Reply-To: <26e148920806262006t11b232f5rdff62394a0bd11da@mail.gmail.com>
References: <26e148920806261731j9a1d920y8353da97a78f8d29@mail.gmail.com>
	<26e148920806261817m5f615a14n975d438216da66e3@mail.gmail.com>
	<26e148920806262006t11b232f5rdff62394a0bd11da@mail.gmail.com>
Message-ID: <fb283d5e0806270012p608f9d13x8270622ca8918e7e@mail.gmail.com>

If you are using master branch of packet, then just pull again from
the repository and your problem should be gone.

If you are using svn trunk, here is the changelog thats required to
fix zombie processes:

http://github.com/gnufied/packet/commit/81b95c0cf83201c088d258b612e85a36019e6812



On Fri, Jun 27, 2008 at 8:36 AM, Reynard <reynard.list at gmail.com> wrote:
> hmm actually it doesn't. how can I prevent packet_worker defunct process
> hanging around?
>
> - reynard
>
> On Thu, Jun 26, 2008 at 9:17 PM, Reynard <reynard.list at gmail.com> wrote:
>>
>> I think I just solved it, it deletes itself if I don't call exit.
>>
>> - reynard
>>
>>
>> On Thu, Jun 26, 2008 at 8:31 PM, Reynard <reynard.list at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I have just upgraded to the latest version, and now I have problem with
>>> lots of packet_worker process hanging around
>>>
>>> 26820 pts/3    Z      0:03 [packet_worker_r] <defunct>
>>>
>>> basically, I always create a new worker and call exit after it's done, so
>>> multiple process can be running at the same time. with the old version
>>> (before using packet), it deletes the process after it's done, now it
>>> doesn't seem to delete it. has the API change? I cannot find anything in the
>>> website documentation.
>>> This is my worker code:
>>>
>>> def create(args = nil)
>>>   # do work
>>>   exit
>>> end
>>>
>>>
>>> - reynard
>>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From dan at revelationglobal.com  Fri Jun 27 17:28:58 2008
From: dan at revelationglobal.com (Dan Herrera)
Date: Fri, 27 Jun 2008 14:28:58 -0700
Subject: [Backgroundrb-devel] Tests failing when using 'reload_on_schedule
	true'
Message-ID: <b89b94f00806271428w744b60dbpaba0bc63fb2a91e6@mail.gmail.com>

Hi All,

When I add the line 'reload_on_schedule true' to my worker, my tests
will no longer run with the following exception:

Starting ferret DRb
server.../Users/revelation/projects/revelation-project/config/../lib/workers/alert_worker.rb:3:
undefined method `reload_on_schedule' for AlertWorker:Class
(NoMethodError)

class AlertWorker < BackgrounDRb::MetaWorker
  set_worker_name :alert_worker
  reload_on_schedule true

 ...
end

Anyone else seen this?

Thanks!

dan

From gethemant at gmail.com  Sat Jun 28 00:32:43 2008
From: gethemant at gmail.com (gethemant at gmail.com)
Date: Sat, 28 Jun 2008 10:02:43 +0530
Subject: [Backgroundrb-devel] Tests failing when using
	'reload_on_schedule true'
In-Reply-To: <b89b94f00806271428w744b60dbpaba0bc63fb2a91e6@mail.gmail.com>
References: <b89b94f00806271428w744b60dbpaba0bc63fb2a91e6@mail.gmail.com>
Message-ID: <fb283d5e0806272132v7e580543ude9507708aaed40@mail.gmail.com>

Just open bdrb_test_helper.rb file from test directory and mock out the method.

On 6/28/08, Dan Herrera <dan at revelationglobal.com> wrote:
> Hi All,
>
> When I add the line 'reload_on_schedule true' to my worker, my tests
> will no longer run with the following exception:
>
> Starting ferret DRb
> server.../Users/revelation/projects/revelation-project/config/../lib/workers/alert_worker.rb:3:
> undefined method `reload_on_schedule' for AlertWorker:Class
> (NoMethodError)
>
> class AlertWorker < BackgrounDRb::MetaWorker
>   set_worker_name :alert_worker
>   reload_on_schedule true
>
>  ...
> end
>
> Anyone else seen this?
>
> Thanks!
>
> dan
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From dan at revelationglobal.com  Sat Jun 28 00:44:28 2008
From: dan at revelationglobal.com (Dan Herrera)
Date: Fri, 27 Jun 2008 21:44:28 -0700
Subject: [Backgroundrb-devel] Tests failing when using
	'reload_on_schedule true'
In-Reply-To: <fb283d5e0806272132v7e580543ude9507708aaed40@mail.gmail.com>
References: <b89b94f00806271428w744b60dbpaba0bc63fb2a91e6@mail.gmail.com>
	<fb283d5e0806272132v7e580543ude9507708aaed40@mail.gmail.com>
Message-ID: <b89b94f00806272144v1447e4d7v3dc6860782ae3597@mail.gmail.com>

On Fri, Jun 27, 2008 at 9:32 PM,  <gethemant at gmail.com> wrote:
> Just open bdrb_test_helper.rb file from test directory and mock out the method.

Excellent, thank you!

dan

>
> On 6/28/08, Dan Herrera <dan at revelationglobal.com> wrote:
>> Hi All,
>>
>> When I add the line 'reload_on_schedule true' to my worker, my tests
>> will no longer run with the following exception:
>>
>> Starting ferret DRb
>> server.../Users/revelation/projects/revelation-project/config/../lib/workers/alert_worker.rb:3:
>> undefined method `reload_on_schedule' for AlertWorker:Class
>> (NoMethodError)
>>
>> class AlertWorker < BackgrounDRb::MetaWorker
>>   set_worker_name :alert_worker
>>   reload_on_schedule true
>>
>>  ...
>> end
>>
>> Anyone else seen this?
>>
>> Thanks!
>>
>> dan
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>

From bslayerw at gmail.com  Sun Jun 29 05:08:26 2008
From: bslayerw at gmail.com (Byron Wright)
Date: Sun, 29 Jun 2008 02:08:26 -0700
Subject: [Backgroundrb-devel] Cron scheduling for week days
Message-ID: <CFC72D65-FA79-4140-B33B-3039687A67ED@gmail.com>

Hi,

I'm attempting to setup a cron style scheduling for a method and  
cannot seem to get this to work as expected. I'm trying to call this  
method every morning at 2:00 am week days only (not on sat and sun).  
This is the config that never triggers.

:schedules:
:scheduled_worker:
  :do_at_two_weekdays_only:
    :trigger_args: 0 0 2 * * 1-5 *

thanks,
Byron