From gethemant at gmail.com  Fri Nov  2 06:05:45 2007
From: gethemant at gmail.com (hemant)
Date: Fri, 2 Nov 2007 15:35:45 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb available
	now from svn
Message-ID: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>

Hi,

A pre-release version of backgroundrb is available now from svn.
Download it from here:

http://svn.devjavu.com/backgroundrb/branches/version099/

Since this release marks significant migration from existing
practices, i intend to keep trunk untouched for a while.

There are no install scripts, but you should copy "backgroundrb" file
from script directory of plugin to script directory of rails. You
should also config "backgroundrb.yml" file from config directory of
plugin to config directory of rails root.

There is one sample worker available in
BACKGROUNDRB_ROOT/sample_worker directory.
There is also a sample client available.

Currently, passing of ActiveRecord objects is not supported. Although,
We can do that, but in most situations, passing 'id' of object is
enough i thought. Also, you may encounter some bugs when you are
passing large objects around.

I will try to explain meat of a sample worker:

class FooWorker < MetaWorker
  set_worker_name :foo_worker
  attr_accessor :count
  def worker_init
    puts "Starting Foo Worker"
    add_periodic_timer(4) { increment_status}
  end
  def process_request p_data
    p p_data
  end

  def increment_status
    @count ||= 0
    @count += 1
    register_status(@count)
  end
end

First, I intend to wrap MetaWorker within a namespace(read module), so
pardon me there.

So, when backgroundrb starts it reads all the workers in
WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
worker_init is called in child process of each worker. You can use
worker_init for initializing a worker.

"process_request" is the method which will be normally invoked when
you receive requests from rails. But there is a exception, in each
worker there is a method called "register_status" available, which can
be invoked from workers to register their status with master.

Now, from rails, you can query status of a worker using

MiddleMan.ask_status(:worker => :foo_worker)

All the status requests are server by master itself and hence
"process_request" inside worker will not be invoked for them.

You will find support for schedules very rough around the edges.
Reason is, You can use

add_periodic_timer(5) { foo_job }

to do any kind of scheduling. add_periodic_timer takes number of
seconds as argument.
for one shot timers, you can use:

add_timer(5) { foo_job }


Core of new version of bdrb is implemented on a networking library
called "packet".  It needs its own introduction. But now, inside each
worker you can start evented network connections. You have some
methods like:

connect("localhost",5678,FooHandler)
start_server("localhost"m,5678,FooServer)

inside each worker, which can be used to let workers do fancy stuff.
This is just a by product of using "packet" library.

I know, thing is rough around the edge and needs more polish. But
still, i hope it will be more stable than previous versions of
backgroundrb.

Please report all the bugs here.

Thanks

-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri Nov  2 06:26:44 2007
From: gethemant at gmail.com (hemant)
Date: Fri, 2 Nov 2007 15:56:44 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
Message-ID: <fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>

On 11/2/07, hemant <gethemant at gmail.com> wrote:
> Hi,
>
> A pre-release version of backgroundrb is available now from svn.
> Download it from here:
>
> http://svn.devjavu.com/backgroundrb/branches/version099/
>
> Since this release marks significant migration from existing
> practices, i intend to keep trunk untouched for a while.
>
> There are no install scripts, but you should copy "backgroundrb" file
> from script directory of plugin to script directory of rails. You
> should also config "backgroundrb.yml" file from config directory of
> plugin to config directory of rails root.
>
> There is one sample worker available in
> BACKGROUNDRB_ROOT/sample_worker directory.
> There is also a sample client available.
>
> Currently, passing of ActiveRecord objects is not supported. Although,
> We can do that, but in most situations, passing 'id' of object is
> enough i thought. Also, you may encounter some bugs when you are
> passing large objects around.
>
> I will try to explain meat of a sample worker:
>
> class FooWorker < MetaWorker
>   set_worker_name :foo_worker
>   attr_accessor :count
>   def worker_init
>     puts "Starting Foo Worker"
>     add_periodic_timer(4) { increment_status}
>   end
>   def process_request p_data
>     p p_data
>   end
>
>   def increment_status
>     @count ||= 0
>     @count += 1
>     register_status(@count)
>   end
> end
>
> First, I intend to wrap MetaWorker within a namespace(read module), so
> pardon me there.
>
> So, when backgroundrb starts it reads all the workers in
> WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
> worker_init is called in child process of each worker. You can use
> worker_init for initializing a worker.
>
> "process_request" is the method which will be normally invoked when
> you receive requests from rails. But there is a exception, in each
> worker there is a method called "register_status" available, which can
> be invoked from workers to register their status with master.
>
> Now, from rails, you can query status of a worker using
>
> MiddleMan.ask_status(:worker => :foo_worker)
>
> All the status requests are server by master itself and hence
> "process_request" inside worker will not be invoked for them.
>
> You will find support for schedules very rough around the edges.
> Reason is, You can use
>
> add_periodic_timer(5) { foo_job }
>
> to do any kind of scheduling. add_periodic_timer takes number of
> seconds as argument.
> for one shot timers, you can use:
>
> add_timer(5) { foo_job }
>
>
> Core of new version of bdrb is implemented on a networking library
> called "packet".  It needs its own introduction. But now, inside each
> worker you can start evented network connections. You have some
> methods like:
>
> connect("localhost",5678,FooHandler)
> start_server("localhost"m,5678,FooServer)
>
> inside each worker, which can be used to let workers do fancy stuff.
> This is just a by product of using "packet" library.
>
> I know, thing is rough around the edge and needs more polish. But
> still, i hope it will be more stable than previous versions of
> backgroundrb.
>
> Please report all the bugs here.
>

Couple of things, I wanted to add.

If you want to access your workers from other machines, then ip
section in config file should be something thats accessible from all
machines.

Also, you can't restart your workers from rails. Although I would like
to have support for starting and stopping of workers dynamically, but
it was possible in this release.

I am looking for suggestions on how to smoothen scheduling of worker
methods. Currently its just add_periodic_timer and whatever we
implement is just going to be "sugar" around it, yet I need some
opinions.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From ezmobius at gmail.com  Fri Nov  2 13:03:04 2007
From: ezmobius at gmail.com (Ezra Zygmuntowicz)
Date: Fri, 2 Nov 2007 10:03:04 -0700
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>
Message-ID: <9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>

Kick ass hemant. Looks good. I'm playing with it now and will give  
feedback,

Cheers-
-Ezra

On Nov 2, 2007, at 3:26 AM, hemant wrote:

> On 11/2/07, hemant <gethemant at gmail.com> wrote:
>> Hi,
>>
>> A pre-release version of backgroundrb is available now from svn.
>> Download it from here:
>>
>> http://svn.devjavu.com/backgroundrb/branches/version099/
>>
>> Since this release marks significant migration from existing
>> practices, i intend to keep trunk untouched for a while.
>>
>> There are no install scripts, but you should copy "backgroundrb" file
>> from script directory of plugin to script directory of rails. You
>> should also config "backgroundrb.yml" file from config directory of
>> plugin to config directory of rails root.
>>
>> There is one sample worker available in
>> BACKGROUNDRB_ROOT/sample_worker directory.
>> There is also a sample client available.
>>
>> Currently, passing of ActiveRecord objects is not supported.  
>> Although,
>> We can do that, but in most situations, passing 'id' of object is
>> enough i thought. Also, you may encounter some bugs when you are
>> passing large objects around.
>>
>> I will try to explain meat of a sample worker:
>>
>> class FooWorker < MetaWorker
>>  set_worker_name :foo_worker
>>  attr_accessor :count
>>  def worker_init
>>    puts "Starting Foo Worker"
>>    add_periodic_timer(4) { increment_status}
>>  end
>>  def process_request p_data
>>    p p_data
>>  end
>>
>>  def increment_status
>>    @count ||= 0
>>    @count += 1
>>    register_status(@count)
>>  end
>> end
>>
>> First, I intend to wrap MetaWorker within a namespace(read module),  
>> so
>> pardon me there.
>>
>> So, when backgroundrb starts it reads all the workers in
>> WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
>> worker_init is called in child process of each worker. You can use
>> worker_init for initializing a worker.
>>
>> "process_request" is the method which will be normally invoked when
>> you receive requests from rails. But there is a exception, in each
>> worker there is a method called "register_status" available, which  
>> can
>> be invoked from workers to register their status with master.
>>
>> Now, from rails, you can query status of a worker using
>>
>> MiddleMan.ask_status(:worker => :foo_worker)
>>
>> All the status requests are server by master itself and hence
>> "process_request" inside worker will not be invoked for them.
>>
>> You will find support for schedules very rough around the edges.
>> Reason is, You can use
>>
>> add_periodic_timer(5) { foo_job }
>>
>> to do any kind of scheduling. add_periodic_timer takes number of
>> seconds as argument.
>> for one shot timers, you can use:
>>
>> add_timer(5) { foo_job }
>>
>>
>> Core of new version of bdrb is implemented on a networking library
>> called "packet".  It needs its own introduction. But now, inside each
>> worker you can start evented network connections. You have some
>> methods like:
>>
>> connect("localhost",5678,FooHandler)
>> start_server("localhost"m,5678,FooServer)
>>
>> inside each worker, which can be used to let workers do fancy stuff.
>> This is just a by product of using "packet" library.
>>
>> I know, thing is rough around the edge and needs more polish. But
>> still, i hope it will be more stable than previous versions of
>> backgroundrb.
>>
>> Please report all the bugs here.
>>
>
> Couple of things, I wanted to add.
>
> If you want to access your workers from other machines, then ip
> section in config file should be something thats accessible from all
> machines.
>
> Also, you can't restart your workers from rails. Although I would like
> to have support for starting and stopping of workers dynamically, but
> it was possible in this release.
>
> I am looking for suggestions on how to smoothen scheduling of worker
> methods. Currently its just add_periodic_timer and whatever we
> implement is just going to be "sugar" around it, yet I need some
> opinions.
>
>
> -- 
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


From gethemant at gmail.com  Fri Nov  2 13:55:13 2007
From: gethemant at gmail.com (hemant)
Date: Fri, 2 Nov 2007 23:25:13 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>
	<9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>
Message-ID: <fb283d5e0711021055o52c58ecfh18714b4ce4713f1d@mail.gmail.com>

On 11/2/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> Kick ass hemant. Looks good. I'm playing with it now and will give
> feedback,
>

Thanks Ezra, for kind words. Do, let me know about any bugs you find.

I expect the API to change in coming days. I can imagine a rails page,
which will tell you about all your workers and what they are doing. You
can stop,start them from web interface. Lets see how far we can get.


-
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri Nov  2 14:15:25 2007
From: gethemant at gmail.com (hemant)
Date: Fri, 2 Nov 2007 23:45:25 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <fb283d5e0711021055o52c58ecfh18714b4ce4713f1d@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>
	<9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>
	<fb283d5e0711021055o52c58ecfh18714b4ce4713f1d@mail.gmail.com>
Message-ID: <fb283d5e0711021115g1e4d4f86nd4a900e5a2ab9a65@mail.gmail.com>

On 11/2/07, hemant <gethemant at gmail.com> wrote:
> On 11/2/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> > Kick ass hemant. Looks good. I'm playing with it now and will give
> > feedback,
> >
>
> Thanks Ezra, for kind words. Do, let me know about any bugs you find.
>
> I expect the API to change in coming days. I can imagine a rails page,
> which will tell you about all your workers and what they are doing. You
> can stop,start them from web interface. Lets see how far we can get.
>

Damn, I the only external dependency is:

http://rubyforge.org/projects/uuid/

You need to grab this to run it.

-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri Nov  2 13:50:04 2007
From: gethemant at gmail.com (Hemant Kumar)
Date: Fri, 02 Nov 2007 23:20:04 +0530
Subject: [Backgroundrb-devel] pre-release version of
	backgroundrb	available now from svn
In-Reply-To: <9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<fb283d5e0711020326v22324177q6a6f8cf3a49ac266@mail.gmail.com>
	<9B678121-8BDC-4768-8D68-DA4FE8742034@brainspl.at>
Message-ID: <1194025804.22847.3.camel@localhost.localdomain>


On Fri, 2007-11-02 at 10:03 -0700, Ezra Zygmuntowicz wrote:
> Kick ass hemant. Looks good. I'm playing with it now and will give  
> feedback,
> 
Thanks Ezra, for kind words. Do, let me know about any bugs you find.

I expect the API to change in coming days. I can imagine a rails page,
which will tell you about all your workers and what they are doing. You
can stop,start them from web interface. Lets see how far we can get.



> Cheers-
> -Ezra
> 
> On Nov 2, 2007, at 3:26 AM, hemant wrote:
> 
> > On 11/2/07, hemant <gethemant at gmail.com> wrote:
> >> Hi,
> >>
> >> A pre-release version of backgroundrb is available now from svn.
> >> Download it from here:
> >>
> >> http://svn.devjavu.com/backgroundrb/branches/version099/
> >>
> >> Since this release marks significant migration from existing
> >> practices, i intend to keep trunk untouched for a while.
> >>
> >> There are no install scripts, but you should copy "backgroundrb" file
> >> from script directory of plugin to script directory of rails. You
> >> should also config "backgroundrb.yml" file from config directory of
> >> plugin to config directory of rails root.
> >>
> >> There is one sample worker available in
> >> BACKGROUNDRB_ROOT/sample_worker directory.
> >> There is also a sample client available.
> >>
> >> Currently, passing of ActiveRecord objects is not supported.  
> >> Although,
> >> We can do that, but in most situations, passing 'id' of object is
> >> enough i thought. Also, you may encounter some bugs when you are
> >> passing large objects around.
> >>
> >> I will try to explain meat of a sample worker:
> >>
> >> class FooWorker < MetaWorker
> >>  set_worker_name :foo_worker
> >>  attr_accessor :count
> >>  def worker_init
> >>    puts "Starting Foo Worker"
> >>    add_periodic_timer(4) { increment_status}
> >>  end
> >>  def process_request p_data
> >>    p p_data
> >>  end
> >>
> >>  def increment_status
> >>    @count ||= 0
> >>    @count += 1
> >>    register_status(@count)
> >>  end
> >> end
> >>
> >> First, I intend to wrap MetaWorker within a namespace(read module),  
> >> so
> >> pardon me there.
> >>
> >> So, when backgroundrb starts it reads all the workers in
> >> WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
> >> worker_init is called in child process of each worker. You can use
> >> worker_init for initializing a worker.
> >>
> >> "process_request" is the method which will be normally invoked when
> >> you receive requests from rails. But there is a exception, in each
> >> worker there is a method called "register_status" available, which  
> >> can
> >> be invoked from workers to register their status with master.
> >>
> >> Now, from rails, you can query status of a worker using
> >>
> >> MiddleMan.ask_status(:worker => :foo_worker)
> >>
> >> All the status requests are server by master itself and hence
> >> "process_request" inside worker will not be invoked for them.
> >>
> >> You will find support for schedules very rough around the edges.
> >> Reason is, You can use
> >>
> >> add_periodic_timer(5) { foo_job }
> >>
> >> to do any kind of scheduling. add_periodic_timer takes number of
> >> seconds as argument.
> >> for one shot timers, you can use:
> >>
> >> add_timer(5) { foo_job }
> >>
> >>
> >> Core of new version of bdrb is implemented on a networking library
> >> called "packet".  It needs its own introduction. But now, inside each
> >> worker you can start evented network connections. You have some
> >> methods like:
> >>
> >> connect("localhost",5678,FooHandler)
> >> start_server("localhost"m,5678,FooServer)
> >>
> >> inside each worker, which can be used to let workers do fancy stuff.
> >> This is just a by product of using "packet" library.
> >>
> >> I know, thing is rough around the edge and needs more polish. But
> >> still, i hope it will be more stable than previous versions of
> >> backgroundrb.
> >>
> >> Please report all the bugs here.
> >>
> >
> > Couple of things, I wanted to add.
> >
> > If you want to access your workers from other machines, then ip
> > section in config file should be something thats accessible from all
> > machines.
> >
> > Also, you can't restart your workers from rails. Although I would like
> > to have support for starting and stopping of workers dynamically, but
> > it was possible in this release.
> >
> > I am looking for suggestions on how to smoothen scheduling of worker
> > methods. Currently its just add_periodic_timer and whatever we
> > implement is just going to be "sugar" around it, yet I need some
> > opinions.
> >
> >
> > -- 
> > Let them talk of their oriental summer climes of everlasting
> > conservatories; give me the privilege of making my own summer with my
> > own coals.
> >
> > http://gnufied.org
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
> 


From brendan.grainger at gmail.com  Sun Nov  4 15:22:35 2007
From: brendan.grainger at gmail.com (Brendan Grainger)
Date: Sun, 4 Nov 2007 15:22:35 -0500
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
Message-ID: <1342D2FE-183C-497B-B0B8-9E0675547674@gmail.com>

Hi Hemant,

The new API looks great, but I'm having a couple of problems I wonder  
if you know about.

* The first is that there seems to be a missing guid.rb file in the  
repository? In packet.rb there is:

require 'guid'

I created my own guid.rb and added a hexdigest method which seemed to  
fix the problem.

* The second is when I start and stop backgroundrb it prints out:


Loaded suite script/backgroundrb
Started

Finished in 0.00025 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

It looks like it's loading the test suite somehow, but I'm confused  
about how.

* Finally I always get a nice little stack trace when I call  
backgroundrb stop

$ /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/ 
framework/nbio.rb:12:in `read_data': Packet::DisconnectError  
(Packet::DisconnectError)
         from /Users/brendan/Workspace/kuripai/hac/vendor/plugins/ 
backgroundrb/framework/worker.rb:43:in `handle_internal_messages'
         from /Users/brendan/Workspace/kuripai/hac/vendor/plugins/ 
backgroundrb/framework/core.rb:131:in `start_reactor'
         from /Users/brendan/Workspace/kuripai/hac/vendor/plugins/ 
backgroundrb/framework/core.rb:129:in `each'
         from /Users/brendan/Workspace/kuripai/hac/vendor/plugins/ 
backgroundrb/framework/core.rb:129:in `start_reactor'

Thanks!
Brendan

On Nov 2, 2007, at 6:05 AM, hemant wrote:

> Hi,
>
> A pre-release version of backgroundrb is available now from svn.
> Download it from here:
>
> http://svn.devjavu.com/backgroundrb/branches/version099/
>
> Since this release marks significant migration from existing
> practices, i intend to keep trunk untouched for a while.
>
> There are no install scripts, but you should copy "backgroundrb" file
> from script directory of plugin to script directory of rails. You
> should also config "backgroundrb.yml" file from config directory of
> plugin to config directory of rails root.
>
> There is one sample worker available in
> BACKGROUNDRB_ROOT/sample_worker directory.
> There is also a sample client available.
>
> Currently, passing of ActiveRecord objects is not supported. Although,
> We can do that, but in most situations, passing 'id' of object is
> enough i thought. Also, you may encounter some bugs when you are
> passing large objects around.
>
> I will try to explain meat of a sample worker:
>
> class FooWorker < MetaWorker
>   set_worker_name :foo_worker
>   attr_accessor :count
>   def worker_init
>     puts "Starting Foo Worker"
>     add_periodic_timer(4) { increment_status}
>   end
>   def process_request p_data
>     p p_data
>   end
>
>   def increment_status
>     @count ||= 0
>     @count += 1
>     register_status(@count)
>   end
> end
>
> First, I intend to wrap MetaWorker within a namespace(read module), so
> pardon me there.
>
> So, when backgroundrb starts it reads all the workers in
> WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
> worker_init is called in child process of each worker. You can use
> worker_init for initializing a worker.
>
> "process_request" is the method which will be normally invoked when
> you receive requests from rails. But there is a exception, in each
> worker there is a method called "register_status" available, which can
> be invoked from workers to register their status with master.
>
> Now, from rails, you can query status of a worker using
>
> MiddleMan.ask_status(:worker => :foo_worker)
>
> All the status requests are server by master itself and hence
> "process_request" inside worker will not be invoked for them.
>
> You will find support for schedules very rough around the edges.
> Reason is, You can use
>
> add_periodic_timer(5) { foo_job }
>
> to do any kind of scheduling. add_periodic_timer takes number of
> seconds as argument.
> for one shot timers, you can use:
>
> add_timer(5) { foo_job }
>
>
> Core of new version of bdrb is implemented on a networking library
> called "packet".  It needs its own introduction. But now, inside each
> worker you can start evented network connections. You have some
> methods like:
>
> connect("localhost",5678,FooHandler)
> start_server("localhost"m,5678,FooServer)
>
> inside each worker, which can be used to let workers do fancy stuff.
> This is just a by product of using "packet" library.
>
> I know, thing is rough around the edge and needs more polish. But
> still, i hope it will be more stable than previous versions of
> backgroundrb.
>
> Please report all the bugs here.
>
> Thanks
>
> -- 
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071104/ee42868c/attachment-0001.html 

From gethemant at gmail.com  Sun Nov  4 15:49:56 2007
From: gethemant at gmail.com (hemant)
Date: Mon, 5 Nov 2007 02:19:56 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <1342D2FE-183C-497B-B0B8-9E0675547674@gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<1342D2FE-183C-497B-B0B8-9E0675547674@gmail.com>
Message-ID: <fb283d5e0711041249h4bb87b82t195fbd2e357651df@mail.gmail.com>

On 11/5/07, Brendan Grainger <brendan.grainger at gmail.com> wrote:
>  Hi Hemant,
>
> The new API looks great, but I'm having a couple of problems I wonder if you
> know about.
>
> * The first is that there seems to be a missing guid.rb file in the
> repository? In packet.rb there is:
>
>
> require 'guid'

Yes it depends on uuid gem, which can be downloaded from:

http://rubyforge.org/projects/uuid/

>
> I created my own guid.rb and added a hexdigest method which seemed to fix
> the problem.

Yes that would solve the problem.
>
> * The second is when I start and stop backgroundrb it prints out:
>
>
> Loaded suite script/backgroundrb
> Started
>
> Finished in 0.00025 seconds.
>
> 0 tests, 0 assertions, 0 failures, 0 errors
>

Weird, I will have a look into this. If your app is not too critical,
you can zip entire code and send me. I will surely look into this.


> It looks like it's loading the test suite somehow, but I'm confused about
> how.
>
> * Finally I always get a nice little stack trace when I call backgroundrb
> stop
>
> $
> /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/framework/nbio.rb:12:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>         from
> /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/framework/worker.rb:43:in
> `handle_internal_messages'
>         from
> /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/framework/core.rb:131:in
> `start_reactor'
>         from
> /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/framework/core.rb:129:in
> `each'
>         from
> /Users/brendan/Workspace/kuripai/hac/vendor/plugins/backgroundrb/framework/core.rb:129:in
> `start_reactor'
>

He he, Don't bother about these stacktraces. I will have to install
some signal handlers, and they will be gone. Since, those were not top
priority, I left it.

Also, I am pushing some changes, which will fix some issues with
timers. Please sync to latest code.

> Thanks!
> Brendan
>
>
> On Nov 2, 2007, at 6:05 AM, hemant wrote:
>
> Hi,
>
> A pre-release version of backgroundrb is available now from svn.
> Download it from here:
>
> http://svn.devjavu.com/backgroundrb/branches/version099/
>
> Since this release marks significant migration from existing
> practices, i intend to keep trunk untouched for a while.
>
> There are no install scripts, but you should copy "backgroundrb" file
> from script directory of plugin to script directory of rails. You
> should also config "backgroundrb.yml" file from config directory of
> plugin to config directory of rails root.
>
> There is one sample worker available in
> BACKGROUNDRB_ROOT/sample_worker directory.
> There is also a sample client available.
>
> Currently, passing of ActiveRecord objects is not supported. Although,
> We can do that, but in most situations, passing 'id' of object is
> enough i thought. Also, you may encounter some bugs when you are
> passing large objects around.
>
> I will try to explain meat of a sample worker:
>
> class FooWorker < MetaWorker
>   set_worker_name :foo_worker
>   attr_accessor :count
>   def worker_init
>     puts "Starting Foo Worker"
>     add_periodic_timer(4) { increment_status}
>   end
>   def process_request p_data
>     p p_data
>   end
>
>   def increment_status
>     @count ||= 0
>     @count += 1
>     register_status(@count)
>   end
> end
>
> First, I intend to wrap MetaWorker within a namespace(read module), so
> pardon me there.
>
> So, when backgroundrb starts it reads all the workers in
> WORKER_ROOT(which should be RAILS_ROOT/lib/workers) directory and
> worker_init is called in child process of each worker. You can use
> worker_init for initializing a worker.
>
> "process_request" is the method which will be normally invoked when
> you receive requests from rails. But there is a exception, in each
> worker there is a method called "register_status" available, which can
> be invoked from workers to register their status with master.
>
> Now, from rails, you can query status of a worker using
>
> MiddleMan.ask_status(:worker => :foo_worker)
>
> All the status requests are server by master itself and hence
> "process_request" inside worker will not be invoked for them.
>
> You will find support for schedules very rough around the edges.
> Reason is, You can use
>
> add_periodic_timer(5) { foo_job }
>
> to do any kind of scheduling. add_periodic_timer takes number of
> seconds as argument.
> for one shot timers, you can use:
>
> add_timer(5) { foo_job }
>
>
> Core of new version of bdrb is implemented on a networking library
> called "packet".  It needs its own introduction. But now, inside each
> worker you can start evented network connections. You have some
> methods like:
>
> connect("localhost",5678,FooHandler)
> start_server("localhost"m,5678,FooServer)
>
> inside each worker, which can be used to let workers do fancy stuff.
> This is just a by product of using "packet" library.
>
> I know, thing is rough around the edge and needs more polish. But
> still, i hope it will be more stable than previous versions of
> backgroundrb.
>
> Please report all the bugs here.
>
> Thanks
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Mon Nov  5 11:51:55 2007
From: gethemant at gmail.com (hemant)
Date: Mon, 5 Nov 2007 22:21:55 +0530
Subject: [Backgroundrb-devel] pre-release version of backgroundrb
	available now from svn
In-Reply-To: <fb283d5e0711041249h4bb87b82t195fbd2e357651df@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com>
	<1342D2FE-183C-497B-B0B8-9E0675547674@gmail.com>
	<fb283d5e0711041249h4bb87b82t195fbd2e357651df@mail.gmail.com>
Message-ID: <fb283d5e0711050851r6ff8d5ahd1e00a9b381d8352@mail.gmail.com>

> > You will find support for schedules very rough around the edges.
> > Reason is, You can use
> >
> > add_periodic_timer(5) { foo_job }

Ok, I guess new release has been quite cold, with lots of changes from
existing practices. May be, time to implement some more interesting
ideas. I am still working on couple of things and I am a bit confused
on how to export a plugin from a project. I am using svn export, and
svn import to export the current plugin, but it kinda sucks.

If you guys have some opinions, about developing a plugin with a rails
project, please mention.

Now, here is the interesting part. I am working to implement
"runt"(http://runt.rubyforge.org/) temporal expressions in
backgroundrb, so as you can schedule tasks such as billing and stuff
more easily.

What you guys, think? Will it be useful to takeout existing cron style
scheduling from trunk and implement a runt interface in experimental
version?



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From jim.scott at yahoo.com  Mon Nov  5 14:31:40 2007
From: jim.scott at yahoo.com (Jim Scott)
Date: Mon, 5 Nov 2007 12:31:40 -0700
Subject: [Backgroundrb-devel] pre-release version of
	backgroundrbavailable now from svn
In-Reply-To: <fb283d5e0711050851r6ff8d5ahd1e00a9b381d8352@mail.gmail.com>
References: <fb283d5e0711020305k366ce2d4ua32567d6b6582fa4@mail.gmail.com><1342D2FE-183C-497B-B0B8-9E0675547674@gmail.com><fb283d5e0711041249h4bb87b82t195fbd2e357651df@mail.gmail.com>
	<fb283d5e0711050851r6ff8d5ahd1e00a9b381d8352@mail.gmail.com>
Message-ID: <001e01c81fe2$7e555290$9601a8c0@LIGHTFOOT>

I JUST completed (this weekend) a scheduling application that pushes jobs
into BackgroundRb.  It was painful, and will be painful, to maintain, and is
not portable at all from this application to any other application without
some significant rework.

I believe there is a very real difference between processing, and scheduling
of the processing.

I'm all for it.  If I can help, let me know.

Jim Scott

-----Original Message-----
From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant
Sent: Monday, November 05, 2007 9:52 AM
To: Brendan Grainger
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] pre-release version of
backgroundrbavailable now from svn

> > You will find support for schedules very rough around the edges.
> > Reason is, You can use
> >
> > add_periodic_timer(5) { foo_job }

Ok, I guess new release has been quite cold, with lots of changes from
existing practices. May be, time to implement some more interesting
ideas. I am still working on couple of things and I am a bit confused
on how to export a plugin from a project. I am using svn export, and
svn import to export the current plugin, but it kinda sucks.

If you guys have some opinions, about developing a plugin with a rails
project, please mention.

Now, here is the interesting part. I am working to implement
"runt"(http://runt.rubyforge.org/) temporal expressions in
backgroundrb, so as you can schedule tasks such as billing and stuff
more easily.

What you guys, think? Will it be useful to takeout existing cron style
scheduling from trunk and implement a runt interface in experimental
version?



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org
_______________________________________________
Backgroundrb-devel mailing list
Backgroundrb-devel at rubyforge.org
http://rubyforge.org/mailman/listinfo/backgroundrb-devel

No virus found in this incoming message.
Checked by AVG Free Edition. 
Version: 7.5.503 / Virus Database: 269.15.21/1110 - Release Date: 11/4/2007
9:37 PM
 

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.503 / Virus Database: 269.15.21/1110 - Release Date: 11/4/2007
9:37 PM
 


From andre at myavalaunch.com  Mon Nov  5 14:40:42 2007
From: andre at myavalaunch.com (Andre Borrelly)
Date: Mon, 5 Nov 2007 12:40:42 -0700
Subject: [Backgroundrb-devel] pre-release version of
 backgroundrbavailable now from svn
In-Reply-To: <fb283d5e0711050851r6ff8d5ahd1e00a9b381d8352@mail.gmail.com>
References: <fb283d5e0711050851r6ff8d5ahd1e00a9b381d8352@mail.gmail.com>
Message-ID: <30acfe7f5ee16f3386df830eb6c4af2a@localhost>




On Mon, 5 Nov 2007 22:21:55 +0530, hemant <gethemant at gmail.com> wrote:
>> > You will find support for schedules very rough around the edges.
>> > Reason is, You can use
>> >
>> > add_periodic_timer(5) { foo_job }
> 
> Ok, I guess new release has been quite cold, with lots of changes from
> existing practices. May be, time to implement some more interesting
> ideas. I am still working on couple of things and I am a bit confused
> on how to export a plugin from a project. I am using svn export, and
> svn import to export the current plugin, but it kinda sucks.
> 
> If you guys have some opinions, about developing a plugin with a rails
> project, please mention.
> 
> Now, here is the interesting part. I am working to implement
> "runt"(http://runt.rubyforge.org/) temporal expressions in
> backgroundrb, so as you can schedule tasks such as billing and stuff
> more easily.
> 
> What you guys, think? Will it be useful to takeout existing cron style
> scheduling from trunk and implement a runt interface in experimental
> version?

My vote would be to see multiple scheduler classes: simple scheduler, cron scheduler, runt scheduler, etc.
   
> 
> 
> 
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
> 
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From reynard.list at gmail.com  Wed Nov  7 12:51:45 2007
From: reynard.list at gmail.com (Reynard)
Date: Wed, 7 Nov 2007 12:51:45 -0500
Subject: [Backgroundrb-devel] workers cleanup
Message-ID: <26e148920711070951y230c0603i1fb8b6b87ae5d177@mail.gmail.com>

Hi,

I'm using backgroundrb 0.2.1 and having problem with worker processes
staying around when the rails application does not explicitly delete
them.

Is there a way to automatically delete the worker processes when it
has completed the job?
Also does killing the process manually (using kill command) harm the
backgroundrb server? I have some occasions where the backgroundrb
server just hangs, or throws strange errors (which suggests that the
worker.results data was corrupted)

thanks,
- reynard

From gethemant at gmail.com  Wed Nov  7 13:36:42 2007
From: gethemant at gmail.com (hemant)
Date: Thu, 8 Nov 2007 00:06:42 +0530
Subject: [Backgroundrb-devel] workers cleanup
In-Reply-To: <26e148920711070951y230c0603i1fb8b6b87ae5d177@mail.gmail.com>
References: <26e148920711070951y230c0603i1fb8b6b87ae5d177@mail.gmail.com>
Message-ID: <fb283d5e0711071036j2386635h285eb3a53a69c00c@mail.gmail.com>

On 11/7/07, Reynard <reynard.list at gmail.com> wrote:
> Hi,
>
> I'm using backgroundrb 0.2.1 and having problem with worker processes
> staying around when the rails application does not explicitly delete
> them.
>
> Is there a way to automatically delete the worker processes when it
> has completed the job?
You can use worker.delete method or from inside of a worker
self.delete to terminate the worker.

> Also does killing the process manually (using kill command) harm the
> backgroundrb server? I have some occasions where the backgroundrb
> server just hangs, or throws strange errors (which suggests that the
> worker.results data was corrupted)
>

There are some unsolved issues with result data corruption and stuff.
A new release of backgroundrb is around the corner ( this weekend, i
hope to get it out of pre-release, and implement crontab schedulers,
unix schedulers and stuff).

-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From reynard.list at gmail.com  Wed Nov  7 14:02:46 2007
From: reynard.list at gmail.com (Reynard)
Date: Wed, 7 Nov 2007 14:02:46 -0500
Subject: [Backgroundrb-devel] workers cleanup
In-Reply-To: <fb283d5e0711071036j2386635h285eb3a53a69c00c@mail.gmail.com>
References: <26e148920711070951y230c0603i1fb8b6b87ae5d177@mail.gmail.com>
	<fb283d5e0711071036j2386635h285eb3a53a69c00c@mail.gmail.com>
Message-ID: <26e148920711071102r5d240771i187b5b566f6865ab@mail.gmail.com>

Thanks for the reply.
I see, since results outlive the worker, so I can call self.delete
right at the end of do_work method without worrying the results would
be gone, right?  Wouldn't this be always preferrable (the worker
delete itself when done)? All examples I see suggest that the Rails
application has to delete the worker.

Just curious how long does the persistent Result worker keeps
worker.results data around? or is there a config to set this? since
this might increase the memory usage when you get a lot of worker
request, right?

hemant, sorry for the duplicate email, I didn't hit reply all at first :)

- reynard

On Nov 7, 2007 1:36 PM, hemant <gethemant at gmail.com> wrote:
> On 11/7/07, Reynard <reynard.list at gmail.com> wrote:
> > Hi,
> >
> > I'm using backgroundrb 0.2.1 and having problem with worker processes
> > staying around when the rails application does not explicitly delete
> > them.
> >
> > Is there a way to automatically delete the worker processes when it
> > has completed the job?
> You can use worker.delete method or from inside of a worker
> self.delete to terminate the worker.
>
> > Also does killing the process manually (using kill command) harm the
> > backgroundrb server? I have some occasions where the backgroundrb
> > server just hangs, or throws strange errors (which suggests that the
> > worker.results data was corrupted)
> >
>
> There are some unsolved issues with result data corruption and stuff.
> A new release of backgroundrb is around the corner ( this weekend, i
> hope to get it out of pre-release, and implement crontab schedulers,
> unix schedulers and stuff).
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>

From reynard.list at gmail.com  Fri Nov  9 10:08:33 2007
From: reynard.list at gmail.com (Reynard)
Date: Fri, 9 Nov 2007 10:08:33 -0500
Subject: [Backgroundrb-devel] Backgroundrb alternative idea
Message-ID: <26e148920711090708g2bf1de6dm50444d7e5a0c8fae@mail.gmail.com>

Hi,

I was just thinking if this simple idea would serve to replace backgroundrb:

What if the worker is replaced with just a ruby script (can be
anything actually: php, perl, etc)
Rails application can just spawn new process by calling this worker
script (passing data, and possibly uniq id)

The worker script then store the result in database keyed by the uniq
id (which is accessible by the Rails app), So the Rails app can check
on the result by querying the database.

Isn't that exactly what backgroundrb does?

I would love to get comments / suggestions.

Regards,
- reynard

From jonathan.wallace at gmail.com  Fri Nov  9 10:41:04 2007
From: jonathan.wallace at gmail.com (Jonathan Wallace)
Date: Fri, 9 Nov 2007 10:41:04 -0500
Subject: [Backgroundrb-devel] Backgroundrb alternative idea
In-Reply-To: <26e148920711090708g2bf1de6dm50444d7e5a0c8fae@mail.gmail.com>
References: <26e148920711090708g2bf1de6dm50444d7e5a0c8fae@mail.gmail.com>
Message-ID: <fd0631d50711090741r10d1ee4bhb72ef692591c2ecd@mail.gmail.com>

That's exactly the use case we determined we had when we decided to
stop using backgroundrb at my day job.  A database queue served our
purposes quite well.  A minor adjustment is that we cron the ruby task
running script to run periodically.  The task is only submitted to the
queue by the user and not necessarily spawned immediately as you
suggest below.  This should insure that the hardware resources aren't
overwhelmed by a click happy user as only one job would be processed
at a time from the queue.  This also allows us to scale up queue
processing servers simply and easily.

However, in another project I'm involved in, backgroundrb is a perfect
fit because it is necessary for rails to provide the user with status
information of the long running task continually as it is running.
Though this could be accomplished using the database as an interim
status store similar to what you mention below, it is more responsive
to use backgroundrb and the results worker in our case.  Since the
workers don't exist any longer than necessary, we have yet to run into
issues of stability like those reported to this list.

Jonathan


On Nov 9, 2007 10:08 AM, Reynard <reynard.list at gmail.com> wrote:
> Hi,
>
> I was just thinking if this simple idea would serve to replace backgroundrb:
>
> What if the worker is replaced with just a ruby script (can be
> anything actually: php, perl, etc)
> Rails application can just spawn new process by calling this worker
> script (passing data, and possibly uniq id)
>
> The worker script then store the result in database keyed by the uniq
> id (which is accessible by the Rails app), So the Rails app can check
> on the result by querying the database.
>
> Isn't that exactly what backgroundrb does?
>
> I would love to get comments / suggestions.
>
> Regards,
> - reynard
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Jonathan Wallace

From reynard.list at gmail.com  Fri Nov  9 11:07:11 2007
From: reynard.list at gmail.com (Reynard)
Date: Fri, 9 Nov 2007 11:07:11 -0500
Subject: [Backgroundrb-devel] Backgroundrb alternative idea
In-Reply-To: <fd0631d50711090741r10d1ee4bhb72ef692591c2ecd@mail.gmail.com>
References: <26e148920711090708g2bf1de6dm50444d7e5a0c8fae@mail.gmail.com>
	<fd0631d50711090741r10d1ee4bhb72ef692591c2ecd@mail.gmail.com>
Message-ID: <26e148920711090807i6d59e7e6k69d7b6937a1e42ee@mail.gmail.com>

Thanks for the advice on performance consideration, I guess I'll just
have to try it and test to make sure the perfomance will be ok. It's
good to know someone else has done it though, and this probably just a
temporary solution until the new backgroundrb is released and stable
:)

Regards,
- reynard


On Nov 9, 2007 10:41 AM, Jonathan Wallace <jonathan.wallace at gmail.com> wrote:
> That's exactly the use case we determined we had when we decided to
> stop using backgroundrb at my day job.  A database queue served our
> purposes quite well.  A minor adjustment is that we cron the ruby task
> running script to run periodically.  The task is only submitted to the
> queue by the user and not necessarily spawned immediately as you
> suggest below.  This should insure that the hardware resources aren't
> overwhelmed by a click happy user as only one job would be processed
> at a time from the queue.  This also allows us to scale up queue
> processing servers simply and easily.
>
> However, in another project I'm involved in, backgroundrb is a perfect
> fit because it is necessary for rails to provide the user with status
> information of the long running task continually as it is running.
> Though this could be accomplished using the database as an interim
> status store similar to what you mention below, it is more responsive
> to use backgroundrb and the results worker in our case.  Since the
> workers don't exist any longer than necessary, we have yet to run into
> issues of stability like those reported to this list.
>
> Jonathan
>

From orion at mh2o.com  Fri Nov  9 11:05:02 2007
From: orion at mh2o.com (Orion Delwaterman)
Date: Fri, 9 Nov 2007 11:05:02 -0500
Subject: [Backgroundrb-devel] Why the switch to SMTP
Message-ID: <205576B004DDBD438F6560241E0751A0157F9074@sbsexc02.mh2onyc.com>

Can anyone explain why BackgrounDRb is switching form DRb to SMTP
protocol?  That seems like a lot of overhead to add to the messaging
back and forth?

Orion

From gethemant at gmail.com  Fri Nov  9 15:06:55 2007
From: gethemant at gmail.com (hemant)
Date: Sat, 10 Nov 2007 01:36:55 +0530
Subject: [Backgroundrb-devel] Why the switch to SMTP
In-Reply-To: <205576B004DDBD438F6560241E0751A0157F9074@sbsexc02.mh2onyc.com>
References: <205576B004DDBD438F6560241E0751A0157F9074@sbsexc02.mh2onyc.com>
Message-ID: <fb283d5e0711091206w24e64eeqef10f3c92f2497d9@mail.gmail.com>

On Nov 9, 2007 9:35 PM, Orion Delwaterman <orion at mh2o.com> wrote:
> Can anyone explain why BackgrounDRb is switching form DRb to SMTP
> protocol?  That seems like a lot of overhead to add to the messaging
> back and forth?

Its not being switched to SMTP protocol.  I don't know from where that
information comes from.
But yes we are definitely getting rid of drb.



Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From orion at mh2o.com  Fri Nov  9 15:07:55 2007
From: orion at mh2o.com (Orion Delwaterman)
Date: Fri, 9 Nov 2007 15:07:55 -0500
Subject: [Backgroundrb-devel] Why the switch to SMTP
Message-ID: <205576B004DDBD438F6560241E0751A0157F907B@sbsexc02.mh2onyc.com>

So what protocol is it being switched to?

-----Original Message-----
From: hemant [mailto:gethemant at gmail.com] 
Sent: Friday, November 09, 2007 3:07 PM
To: Orion Delwaterman
Cc: backgroundrb-devel at rubyforge.org
Subject: Re: [Backgroundrb-devel] Why the switch to SMTP

On Nov 9, 2007 9:35 PM, Orion Delwaterman <orion at mh2o.com> wrote:
> Can anyone explain why BackgrounDRb is switching form DRb to SMTP
> protocol?  That seems like a lot of overhead to add to the messaging
> back and forth?

Its not being switched to SMTP protocol.  I don't know from where that
information comes from.
But yes we are definitely getting rid of drb.



Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri Nov  9 15:18:59 2007
From: gethemant at gmail.com (hemant)
Date: Sat, 10 Nov 2007 01:48:59 +0530
Subject: [Backgroundrb-devel] Why the switch to SMTP
In-Reply-To: <205576B004DDBD438F6560241E0751A0157F907B@sbsexc02.mh2onyc.com>
References: <205576B004DDBD438F6560241E0751A0157F907B@sbsexc02.mh2onyc.com>
Message-ID: <fb283d5e0711091218t573201c5p9a5bbcae91794370@mail.gmail.com>

On Nov 10, 2007 1:37 AM, Orion Delwaterman <orion at mh2o.com> wrote:
> So what protocol is it being switched to?
For internal communication between workers, its just Marshal.load and
Marshal.dump with a uniq message header ( or footer ), that indicates
end of message. When worker or master reads a message thats complete,
it calls Marshal.load on it, and passes it around. I guess, that would
be even lighter than DRb, but may not be as much versatile. But with
this release of bdrb, I am more concerned about stability of stuff
than features.


>
>
> -----Original Message-----
> From: hemant [mailto:gethemant at gmail.com]
> Sent: Friday, November 09, 2007 3:07 PM
> To: Orion Delwaterman
> Cc: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Why the switch to SMTP
>
> On Nov 9, 2007 9:35 PM, Orion Delwaterman <orion at mh2o.com> wrote:
> > Can anyone explain why BackgrounDRb is switching form DRb to SMTP
> > protocol?  That seems like a lot of overhead to add to the messaging
> > back and forth?
>
> Its not being switched to SMTP protocol.  I don't know from where that
> information comes from.
> But yes we are definitely getting rid of drb.
>
>
>
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From orion at mh2o.com  Mon Nov 12 11:05:36 2007
From: orion at mh2o.com (Orion Delwaterman)
Date: Mon, 12 Nov 2007 11:05:36 -0500
Subject: [Backgroundrb-devel] Workers calling Workers
Message-ID: <205576B004DDBD438F6560241E0751A0157F9083@sbsexc02.mh2onyc.com>

Hello,

 

We are working on a centralized report generating project, which calls
for data processing on different servers and then centralizes the
reports into a single project.  What we would like to do is have a
Monitor BackgrounDRb worker call another BackgrounDRb worker to process
data on a different server, and then have our Monitor Worker collect the
data and then store it according to our specs (all rails workers).

 

We are doing some initial testing and trying to get your basic Hello
World program working, so we set up two BackgrounDRb servers on the same
server, and changed the MiddleMan object to the hash MiddleMen which has
references to both servers.  From the main Rails application we are able
to access either BackgrounDRb server from the main rails application and
launch workers without issue.  However, when we try and reference a
worker from another we get a RangeError telling us that Ruby is trying
to access a recycled object (My guess is that its asking for junk Memory
Address).

 

Here's the relevant code and error.  The exception is being thrown on
the last line of the code:

 

    producer_args = {:report_id => report.id, :args => 

YAML.load(report.args)}

    producer_args.merge!(:debug_mode => true) if @debug_mode

    task.middleman.new_worker(:class => 

"#{task.worker_name}_#{TaskManager::Task::WORKER_POSTFIX}".to_sym,

                              :job_key => producer_key,

                              :args => producer_args

                              )

 

    producer = task.middleman.worker(producer_key)

 

 

# Logfile created on Fri Nov 09 17:30:03 -0500 2007 by logger.rb/1.5.2.9

20071109-17:30:03 (27237) Starting WorkerLogger

20071109-17:30:04 (27238) In ResultsWorker

20071109-17:30:20 (27238) Work started

20071109-17:30:20 (27238) 0xdba48138 is recycled object - (RangeError)

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:375:in 

`_id2ref'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:375:in
`to_obj'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1402:in 

`to_obj'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1704:in 

`to_obj'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1005:in
`_load'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:585:in
`load'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:585:in
`load'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/thread.rb:33:in 

`exclusive'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:581:in
`load'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:633:in 

`recv_reply'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:921:in 

`recv_reply'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1195:in 

`send_message'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1086:in 

`method_missing'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1170:in
`open'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1085:in 

`method_missing'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1107:in 

`with_friend'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1084:in 

`method_missing'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/lib/workers/task_monitoring_worker.
rb:29:in 

`do_work'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:56:in 

`send'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:56:in 

`work_thread'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:70:in 

`call'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:70:in 

`work_thread'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:68:in 

`initialize'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:68:in 

`new'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/worker.rb:68:in 

`work_thread'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:138:in `send'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:138:in 

`method_missing'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:135:in `ex'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/sync.rb:229:in 

`synchronize'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:135:in `ex'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:138:in 

`method_missing'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in 

`__send__'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in 

`perform_without_block'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1515:in 

`perform'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1589:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in
`loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in
`start'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1430:in
`run'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in
`start'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in
`run'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1347:in 

`initialize'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in
`new'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in 

`start_service'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:396:in 

`initialize'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:391:in
`times'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/slave-1.2.1/lib/slave.rb:391:in 

`initialize'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/middleman.rb:210:in 

`new'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/middleman.rb:210:in 

`new_worker'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/thread_pool.rb:36:in 

`dispatch'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/thread_pool.rb:22:in 

`initialize'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/thread_pool.rb:22:in 

`new'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/thread_pool.rb:22:in 

`dispatch'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb/middleman.rb:199:in 

`new_worker'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in 

`__send__'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1555:in 

`perform_without_block'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1515:in 

`perform'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1589:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in
`loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1585:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in
`start'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1581:in 

`main_loop'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1430:in
`run'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in
`start'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1427:in
`run'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1347:in 

`initialize'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in
`new'

20071109-17:30:20 (27238) /usr/local/lib/ruby/1.8/drb/drb.rb:1627:in 

`start_service'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb_server.rb:316:in 

`run'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/application.
rb:194:in 

`call'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/application.
rb:194:in 

`start_proc'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/daemonize.rb
:192:in 

`call'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/daemonize.rb
:192:in 

`call_as_daemon'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/application.
rb:198:in 

`start_proc'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/application.
rb:234:in 

`start'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/controller.r
b:69:in 

`run'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons.rb:185:in 

`run_proc'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/cmdline.rb:1
05:in 

`call'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons/cmdline.rb:1
05:in 

`catch_exceptions'

20071109-17:30:20 (27238) 

/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.7/lib/daemons.rb:184:in 

`run_proc'

20071109-17:30:20 (27238) 

/usr/local/hydra.mh2onyc.com/current/vendor/plugins/backgroundrb/server/
lib/backgroundrb_server.rb:302:in 

`run'

20071109-17:30:20 (27238) script/backgroundrb:29

 

 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071112/be402b46/attachment-0001.html 

From du at bestwaytech.com  Mon Nov 12 15:31:43 2007
From: du at bestwaytech.com (Danila Ulyanov)
Date: Mon, 12 Nov 2007 14:31:43 -0600
Subject: [Backgroundrb-devel] Worker start lag
Message-ID: <4738B82F.50900@bestwaytech.com>

I am trying to use backgroundrb to execute searches that may take a long
time in the background. So a user connects, executes search worker, then
periodically pulls to see if result is available using asynchronous
javascript request. This way a few long searches don't make the site
unusable to rest by freezing up mongrel connections.

Here is the problem - I seem to get a 2-3 second lag from the moment
worker is created and before the worker actually starts running. It's ok
for long searches, but it is very noticeable for short searches, since
it turns 0.3 second searches into 3.3 seconds searches.

Is there anything I can optimize to have workers start faster?

Thanks!

Danila


From gethemant at gmail.com  Wed Nov 14 00:01:43 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 10:31:43 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
Message-ID: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>

Hi Folks,


BackgrounDRb is a Ruby job server and scheduler. Its main intent is to be
used with Ruby on Rails applications for offloading long-running tasks.
Since a Rails application blocks while serving a request it is best to
move long-running tasks off into a background process that is divorced
from http request/response cycle.

This new release of BackgrounDRb is also modular and can be used without
Rails. So any Ruby program or framework can use it.

Copyright (c) 2006 Ezra Zygmuntowicz,skaar[at]waste[dot]org,
Copyright (c) 2007 Hemant Kumar (mail[at]gnufied[dot]org)

== Usage

=== Installation
Getting the code:
  svn co http://svn.devjavu.com/backgroundrb/branches/version10/

Installation with svn externals:
  svn propedit svn:externals vendor/plugins
  [add the following line:]
  backgroundrb http://svn.devjavu.com/backgroundrb/branches/version10/
  [exit editor]

  svn ci -m 'updating svn:external svn property for backgroundrb' vendor/plugins
  svn up vendor/plugins
  rake backgroundrb:setup

Installation with piston:
  piston import
http://svn.devjavu.com/backgroundrb/branches/version10/ backgroundrb

=== Configuration
  Use rake task for initial configuration:

* Cron style scheduling and config

  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006
  |
  | :schedules:
  |   :foo_worker:
  |     :worker_method: foobar
  |     :trigger_args: */5 * * * * * *

* Normal Unix scheduler
  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006
  | :schedules:
  |   :foo_worker:
  |     :start: <%= Time.now + 5.seconds %>
  |     :end: <%= Time.now + 10.minutes %>
  |     :repeat_interval: <%= 1.minute %>

* Plain config
  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006

=== Scheduling
  There are three schemes for periodic execution and scheduling.
  - Cron Scheduling
    You can use configuration file for cron scheduling of workers.
Method specified in configuration
    file would be called periodically. You should take care of the
fact that, time gap between periodic
    invocation of a method should be more than the time thats actually
required to execute the method.
    If a method takes longer time than the time window specified, your
method invocations would lag
    perpetually.
  - Normal Scheduler
    You can use second form of scheduling as shown in config file.
  - add_periodic_timer method
    A third and very basic form of scheduling that you can use is,
"add_periodic_timer" method. You can call
    method from anywhere in your worker.

           def create
             add_periodic_timer(5) { say_hello }
           end

    Above snippet would register the proc for periodic execution at
every 5 seconds.

=== Code
  Install the plugin, and run setup task. Create a worker, using
worker generator.

          ./script/generatr worker bar

  You will have a bar_worker.rb in your RAILS_ROOT/lib/workers/(
called WORKER_ROOT henceforth ).
  Generated code will look like this:

  class BarWorker < BackgrounDRb::MetaWorker
    set_worker_name :bar_worker
    def create
      # this method is called, when worker is loaded for the first time
      puts "starting a bar worker"
    end

    def process_request(p_data)
      user_input = p_data[:data]
      result = self.send(user_input[:method],user_input[:data])
      send_response(p_data,result)
    end
  end

  'create' method gets called, when worker is loaded and created. Each
worker runs in its
  own process and you can use 'create' for initializing worker specific stuff.
  All the requests, sent from rails, to bdrb would be received in
process_request method.
  Received data, contains client address as well, which will be used
to send the response
  back to client. You just need to extract the data from received data
and call appropriate
  method. Once, you are done with executing your method, you can use
"send_response" to send
  the result back to client ( rails in our case ).

  It should be noted that, you must pass original data as an parameter
with send_response
  method because original data contains client address.

  Following code snippet, would ask bdrb to execute method
'add_values' in 'foo_worker' with
  arguments '10+10' and return the result.

       MiddleMan.send_request(:worker => :foo_worker, :method =>
:add_values,:data => "10+10")

  You can also use register_status as described in following snippet
to register status of
  your worker with master, which can be directly queried from rails.

       register_status(some_status_data)

  From rails, you can query status object using following code:

       MiddleMan.ask_status(:worker => :foo_worker)

  Above code would return status object of 'foo_worker'. When you call
register_status
  from a worker, it replaces older state of the worker with master.
Since, master process
  stores status of the worker, all the status queries are served by
master itself. It can be
  used to store result hashes and stuff. Unlike previous versions of
bdrb, there shouldn't be
  any data corruption.

=== Legacy and deprecated stuff

   Although, You need to wrap your head a bit for understanding
"evented" model of network programming,
   but it gets easier once you get hang of it. Much of the older stuff
is deprecated. Here is a brief list:

   - ACL : gone, trust to thy firewalls.
   - Runtime scheduling through MiddleMan proxy object.
     I don't know if many people used this feature, and is probably
easy to implement back.
   - Threads: find .|grep 'Thread' , gone
   - Passing of arguments from configuration file.
     Again, I am not sure many people used it either, if you are
passing arguments to periodic
     methods from configuration files, you may as well hardcode that
argument in worker itself.

   Some people, asked, if we can do "progress bar" or "file_upload"
with new code base. The answer is yes, but
   again, changes would be required. "progress bar" example would look
like this:

     class ProgressWorker < BackgrounDRb::MetaWorker
       set_worker_name :progress_worker
       def create
         @counter = 0
         add_peridic_timer(2) { increment_counter }
       end
       def increment_counter
         @counter += 1
         register_status(@counter)
       end
     end

   And using MiddleMan proxy, you can keep queering status of your progress bar:

       MiddleMan.ask_status(:worker => :progress_worker)

   I would welcome, anyone who contributes more examples back. You can
even use callbacks
   or Deferable pattern for invoking callbacks.

=== Exciting new stuff
  * Rock solid stable ( or will be , after few bug reports )
  * Master Process is using a hash for storing status of each worker.
I am thinking of adding something like this:

        #backgroundrb.yml
        status_storage:
          storage: db
          database: worker_status
    Above code would fullfill thy wishes of persitent worker status
storage. Any more ideas, welcome.
  * Each worker comes with Event loop of its own and can potentially
do lots of fancy stuff. Two noteworthy methods are:

         connect(ip,port,Handler)
         start_worker(ip,port,Handler)

    If you are familiar with EventMachine or Twisted style of network
programming, above methods allow you to
    start tcp servers inside your workers or lets you connect to
external tcp servers. For Each accepted client or
    connected socket a instance of Handler class would be created and
integrated with main event loop.
    This can be used for worker to worker communication between
backgroundrb servers running on two machines.

    You are encouraged to look into framework directory, and see the
code that implements all this stuff.Guts of
    new bdrb is based on this library, which would be released soon as
separately.

== Online Resources
- http://svn.devjavu.com/backgroundrb/branches/version10/
- http://backgroundrb.devjavu.com (trac)
- http://backgroundrb.rubyforge.org (rdoc)




-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed Nov 14 04:34:16 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 15:04:16 +0530
Subject: [Backgroundrb-devel] Worker start lag
In-Reply-To: <4738B82F.50900@bestwaytech.com>
References: <4738B82F.50900@bestwaytech.com>
Message-ID: <fb283d5e0711140134x6bb62d16g60c0decb44ffba77@mail.gmail.com>

On Nov 13, 2007 2:01 AM, Danila Ulyanov <du at bestwaytech.com> wrote:
> I am trying to use backgroundrb to execute searches that may take a long
> time in the background. So a user connects, executes search worker, then
> periodically pulls to see if result is available using asynchronous
> javascript request. This way a few long searches don't make the site
> unusable to rest by freezing up mongrel connections.
>
> Here is the problem - I seem to get a 2-3 second lag from the moment
> worker is created and before the worker actually starts running. It's ok
> for long searches, but it is very noticeable for short searches, since
> it turns 0.3 second searches into 3.3 seconds searches.
>
> Is there anything I can optimize to have workers start faster?
>

Checkout new version of plugin and you will find your lags disappear.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From p.wood at lancaster.ac.uk  Wed Nov 14 08:20:00 2007
From: p.wood at lancaster.ac.uk (Wood, Peter)
Date: Wed, 14 Nov 2007 13:20:00 -0000
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
Message-ID: <89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>

Heya all,

A coworker and I have checked out the new version of BackgrounDRb in
order to build one of our latest projects, so far we've just tried to
replicate some of the functionality we had when testing the old trunk
version.

I may be missing something obvious, but from the code it doesn't appear
that you can set off multiple copies of the same worker like you could
in the old trunk version. We intend to use BackgrounDRb to set off snmp
scans of various devices on our network, some of which take some time to
complete. It is likely that this will happen more than once at any
particular moment in time.

At the moment, using the sample code we've moved the reply to the
MiddleMan to before the work, therefore actually letting the calling
application carry on (which is what we need). However a second call to
the same worker obviously hangs until the first call has completed.

I think what I'm essentially saying is that we want multiple instances
of the same worker which can be identified by a key (as it was in the
trunk). Is this possible with the new version of BackgrounDRb, or is it
on the map?

Peter.

Peter A. Wood                     
Network Security Specialist       
Technical Services Group          
Lancaster University

PGP Fingerprint:
C3B5 376B 16C5 F8D1 E3AE  617F 5718 C338 1D06 0689

-----Original Message-----
From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant
Sent: 14 November 2007 05:02
To: backgroundrb-devel at rubyforge.org
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now

Hi Folks,


BackgrounDRb is a Ruby job server and scheduler. Its main intent is to
be used with Ruby on Rails applications for offloading long-running
tasks.
Since a Rails application blocks while serving a request it is best to
move long-running tasks off into a background process that is divorced
from http request/response cycle.

This new release of BackgrounDRb is also modular and can be used without
Rails. So any Ruby program or framework can use it.

Copyright (c) 2006 Ezra Zygmuntowicz,skaar[at]waste[dot]org,
Copyright (c) 2007 Hemant Kumar (mail[at]gnufied[dot]org)

== Usage

=== Installation
Getting the code:
  svn co http://svn.devjavu.com/backgroundrb/branches/version10/

Installation with svn externals:
  svn propedit svn:externals vendor/plugins
  [add the following line:]
  backgroundrb http://svn.devjavu.com/backgroundrb/branches/version10/
  [exit editor]

  svn ci -m 'updating svn:external svn property for backgroundrb'
vendor/plugins
  svn up vendor/plugins
  rake backgroundrb:setup

Installation with piston:
  piston import
http://svn.devjavu.com/backgroundrb/branches/version10/ backgroundrb

=== Configuration
  Use rake task for initial configuration:

* Cron style scheduling and config

  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006
  |
  | :schedules:
  |   :foo_worker:
  |     :worker_method: foobar
  |     :trigger_args: */5 * * * * * *

* Normal Unix scheduler
  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006
  | :schedules:
  |   :foo_worker:
  |     :start: <%= Time.now + 5.seconds %>
  |     :end: <%= Time.now + 10.minutes %>
  |     :repeat_interval: <%= 1.minute %>

* Plain config
  | :backgroundrb:
  |   :ip: localhost
  |   :port: 11006

=== Scheduling
  There are three schemes for periodic execution and scheduling.
  - Cron Scheduling
    You can use configuration file for cron scheduling of workers.
Method specified in configuration
    file would be called periodically. You should take care of the fact
that, time gap between periodic
    invocation of a method should be more than the time thats actually
required to execute the method.
    If a method takes longer time than the time window specified, your
method invocations would lag
    perpetually.
  - Normal Scheduler
    You can use second form of scheduling as shown in config file.
  - add_periodic_timer method
    A third and very basic form of scheduling that you can use is,
"add_periodic_timer" method. You can call
    method from anywhere in your worker.

           def create
             add_periodic_timer(5) { say_hello }
           end

    Above snippet would register the proc for periodic execution at
every 5 seconds.

=== Code
  Install the plugin, and run setup task. Create a worker, using worker
generator.

          ./script/generatr worker bar

  You will have a bar_worker.rb in your RAILS_ROOT/lib/workers/( called
WORKER_ROOT henceforth ).
  Generated code will look like this:

  class BarWorker < BackgrounDRb::MetaWorker
    set_worker_name :bar_worker
    def create
      # this method is called, when worker is loaded for the first time
      puts "starting a bar worker"
    end

    def process_request(p_data)
      user_input = p_data[:data]
      result = self.send(user_input[:method],user_input[:data])
      send_response(p_data,result)
    end
  end

  'create' method gets called, when worker is loaded and created. Each
worker runs in its
  own process and you can use 'create' for initializing worker specific
stuff.
  All the requests, sent from rails, to bdrb would be received in
process_request method.
  Received data, contains client address as well, which will be used to
send the response
  back to client. You just need to extract the data from received data
and call appropriate
  method. Once, you are done with executing your method, you can use
"send_response" to send
  the result back to client ( rails in our case ).

  It should be noted that, you must pass original data as an parameter
with send_response
  method because original data contains client address.

  Following code snippet, would ask bdrb to execute method 'add_values'
in 'foo_worker' with
  arguments '10+10' and return the result.

       MiddleMan.send_request(:worker => :foo_worker, :method =>
:add_values,:data => "10+10")

  You can also use register_status as described in following snippet to
register status of
  your worker with master, which can be directly queried from rails.

       register_status(some_status_data)

  From rails, you can query status object using following code:

       MiddleMan.ask_status(:worker => :foo_worker)

  Above code would return status object of 'foo_worker'. When you call
register_status
  from a worker, it replaces older state of the worker with master.
Since, master process
  stores status of the worker, all the status queries are served by
master itself. It can be
  used to store result hashes and stuff. Unlike previous versions of
bdrb, there shouldn't be
  any data corruption.

=== Legacy and deprecated stuff

   Although, You need to wrap your head a bit for understanding
"evented" model of network programming,
   but it gets easier once you get hang of it. Much of the older stuff
is deprecated. Here is a brief list:

   - ACL : gone, trust to thy firewalls.
   - Runtime scheduling through MiddleMan proxy object.
     I don't know if many people used this feature, and is probably easy
to implement back.
   - Threads: find .|grep 'Thread' , gone
   - Passing of arguments from configuration file.
     Again, I am not sure many people used it either, if you are passing
arguments to periodic
     methods from configuration files, you may as well hardcode that
argument in worker itself.

   Some people, asked, if we can do "progress bar" or "file_upload"
with new code base. The answer is yes, but
   again, changes would be required. "progress bar" example would look
like this:

     class ProgressWorker < BackgrounDRb::MetaWorker
       set_worker_name :progress_worker
       def create
         @counter = 0
         add_peridic_timer(2) { increment_counter }
       end
       def increment_counter
         @counter += 1
         register_status(@counter)
       end
     end

   And using MiddleMan proxy, you can keep queering status of your
progress bar:

       MiddleMan.ask_status(:worker => :progress_worker)

   I would welcome, anyone who contributes more examples back. You can
even use callbacks
   or Deferable pattern for invoking callbacks.

=== Exciting new stuff
  * Rock solid stable ( or will be , after few bug reports )
  * Master Process is using a hash for storing status of each worker.
I am thinking of adding something like this:

        #backgroundrb.yml
        status_storage:
          storage: db
          database: worker_status
    Above code would fullfill thy wishes of persitent worker status
storage. Any more ideas, welcome.
  * Each worker comes with Event loop of its own and can potentially do
lots of fancy stuff. Two noteworthy methods are:

         connect(ip,port,Handler)
         start_worker(ip,port,Handler)

    If you are familiar with EventMachine or Twisted style of network
programming, above methods allow you to
    start tcp servers inside your workers or lets you connect to
external tcp servers. For Each accepted client or
    connected socket a instance of Handler class would be created and
integrated with main event loop.
    This can be used for worker to worker communication between
backgroundrb servers running on two machines.

    You are encouraged to look into framework directory, and see the
code that implements all this stuff.Guts of
    new bdrb is based on this library, which would be released soon as
separately.

== Online Resources
- http://svn.devjavu.com/backgroundrb/branches/version10/
- http://backgroundrb.devjavu.com (trac)
- http://backgroundrb.rubyforge.org (rdoc)




--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org
_______________________________________________
Backgroundrb-devel mailing list
Backgroundrb-devel at rubyforge.org
http://rubyforge.org/mailman/listinfo/backgroundrb-devel

From gethemant at gmail.com  Wed Nov 14 08:55:51 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 19:25:51 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
Message-ID: <fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>

On Nov 14, 2007 6:50 PM, Wood, Peter <p.wood at lancaster.ac.uk> wrote:
> Heya all,
>
> A coworker and I have checked out the new version of BackgrounDRb in
> order to build one of our latest projects, so far we've just tried to
> replicate some of the functionality we had when testing the old trunk
> version.
>
> I may be missing something obvious, but from the code it doesn't appear
> that you can set off multiple copies of the same worker like you could
> in the old trunk version. We intend to use BackgrounDRb to set off snmp
> scans of various devices on our network, some of which take some time to
> complete. It is likely that this will happen more than once at any
> particular moment in time.
>
> At the moment, using the sample code we've moved the reply to the
> MiddleMan to before the work, therefore actually letting the calling
> application carry on (which is what we need). However a second call to
> the same worker obviously hangs until the first call has completed.
>
> I think what I'm essentially saying is that we want multiple instances
> of the same worker which can be identified by a key (as it was in the
> trunk). Is this possible with the new version of BackgrounDRb, or is it
> on the map?
>

With new version you can't have two instances of a worker active at
the same time. Although, I do think of having that facility soon, but
for the time being you will have to put up with writing copies of
workers. BTW, how many copies do you need and how much time, does
it take to complete scan of devices on network?



>
> -----Original Message-----
> From: backgroundrb-devel-bounces at rubyforge.org
> [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant
> Sent: 14 November 2007 05:02
> To: backgroundrb-devel at rubyforge.org
> Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
>
> Hi Folks,
>
>
> BackgrounDRb is a Ruby job server and scheduler. Its main intent is to
> be used with Ruby on Rails applications for offloading long-running
> tasks.
> Since a Rails application blocks while serving a request it is best to
> move long-running tasks off into a background process that is divorced
> from http request/response cycle.
>
> This new release of BackgrounDRb is also modular and can be used without
> Rails. So any Ruby program or framework can use it.
>
> Copyright (c) 2006 Ezra Zygmuntowicz,skaar[at]waste[dot]org,
> Copyright (c) 2007 Hemant Kumar (mail[at]gnufied[dot]org)
>
> == Usage
>
> === Installation
> Getting the code:
>   svn co http://svn.devjavu.com/backgroundrb/branches/version10/
>
> Installation with svn externals:
>   svn propedit svn:externals vendor/plugins
>   [add the following line:]
>   backgroundrb http://svn.devjavu.com/backgroundrb/branches/version10/
>   [exit editor]
>
>   svn ci -m 'updating svn:external svn property for backgroundrb'
> vendor/plugins
>   svn up vendor/plugins
>   rake backgroundrb:setup
>
> Installation with piston:
>   piston import
> http://svn.devjavu.com/backgroundrb/branches/version10/ backgroundrb
>
> === Configuration
>   Use rake task for initial configuration:
>
> * Cron style scheduling and config
>
>   | :backgroundrb:
>   |   :ip: localhost
>   |   :port: 11006
>   |
>   | :schedules:
>   |   :foo_worker:
>   |     :worker_method: foobar
>   |     :trigger_args: */5 * * * * * *
>
> * Normal Unix scheduler
>   | :backgroundrb:
>   |   :ip: localhost
>   |   :port: 11006
>   | :schedules:
>   |   :foo_worker:
>   |     :start: <%= Time.now + 5.seconds %>
>   |     :end: <%= Time.now + 10.minutes %>
>   |     :repeat_interval: <%= 1.minute %>
>
> * Plain config
>   | :backgroundrb:
>   |   :ip: localhost
>   |   :port: 11006
>
> === Scheduling
>   There are three schemes for periodic execution and scheduling.
>   - Cron Scheduling
>     You can use configuration file for cron scheduling of workers.
> Method specified in configuration
>     file would be called periodically. You should take care of the fact
> that, time gap between periodic
>     invocation of a method should be more than the time thats actually
> required to execute the method.
>     If a method takes longer time than the time window specified, your
> method invocations would lag
>     perpetually.
>   - Normal Scheduler
>     You can use second form of scheduling as shown in config file.
>   - add_periodic_timer method
>     A third and very basic form of scheduling that you can use is,
> "add_periodic_timer" method. You can call
>     method from anywhere in your worker.
>
>            def create
>              add_periodic_timer(5) { say_hello }
>            end
>
>     Above snippet would register the proc for periodic execution at
> every 5 seconds.
>
> === Code
>   Install the plugin, and run setup task. Create a worker, using worker
> generator.
>
>           ./script/generatr worker bar
>
>   You will have a bar_worker.rb in your RAILS_ROOT/lib/workers/( called
> WORKER_ROOT henceforth ).
>   Generated code will look like this:
>
>   class BarWorker < BackgrounDRb::MetaWorker
>     set_worker_name :bar_worker
>     def create
>       # this method is called, when worker is loaded for the first time
>       puts "starting a bar worker"
>     end
>
>     def process_request(p_data)
>       user_input = p_data[:data]
>       result = self.send(user_input[:method],user_input[:data])
>       send_response(p_data,result)
>     end
>   end
>
>   'create' method gets called, when worker is loaded and created. Each
> worker runs in its
>   own process and you can use 'create' for initializing worker specific
> stuff.
>   All the requests, sent from rails, to bdrb would be received in
> process_request method.
>   Received data, contains client address as well, which will be used to
> send the response
>   back to client. You just need to extract the data from received data
> and call appropriate
>   method. Once, you are done with executing your method, you can use
> "send_response" to send
>   the result back to client ( rails in our case ).
>
>   It should be noted that, you must pass original data as an parameter
> with send_response
>   method because original data contains client address.
>
>   Following code snippet, would ask bdrb to execute method 'add_values'
> in 'foo_worker' with
>   arguments '10+10' and return the result.
>
>        MiddleMan.send_request(:worker => :foo_worker, :method =>
> :add_values,:data => "10+10")
>
>   You can also use register_status as described in following snippet to
> register status of
>   your worker with master, which can be directly queried from rails.
>
>        register_status(some_status_data)
>
>   From rails, you can query status object using following code:
>
>        MiddleMan.ask_status(:worker => :foo_worker)
>
>   Above code would return status object of 'foo_worker'. When you call
> register_status
>   from a worker, it replaces older state of the worker with master.
> Since, master process
>   stores status of the worker, all the status queries are served by
> master itself. It can be
>   used to store result hashes and stuff. Unlike previous versions of
> bdrb, there shouldn't be
>   any data corruption.
>
> === Legacy and deprecated stuff
>
>    Although, You need to wrap your head a bit for understanding
> "evented" model of network programming,
>    but it gets easier once you get hang of it. Much of the older stuff
> is deprecated. Here is a brief list:
>
>    - ACL : gone, trust to thy firewalls.
>    - Runtime scheduling through MiddleMan proxy object.
>      I don't know if many people used this feature, and is probably easy
> to implement back.
>    - Threads: find .|grep 'Thread' , gone
>    - Passing of arguments from configuration file.
>      Again, I am not sure many people used it either, if you are passing
> arguments to periodic
>      methods from configuration files, you may as well hardcode that
> argument in worker itself.
>
>    Some people, asked, if we can do "progress bar" or "file_upload"
> with new code base. The answer is yes, but
>    again, changes would be required. "progress bar" example would look
> like this:
>
>      class ProgressWorker < BackgrounDRb::MetaWorker
>        set_worker_name :progress_worker
>        def create
>          @counter = 0
>          add_peridic_timer(2) { increment_counter }
>        end
>        def increment_counter
>          @counter += 1
>          register_status(@counter)
>        end
>      end
>
>    And using MiddleMan proxy, you can keep queering status of your
> progress bar:
>
>        MiddleMan.ask_status(:worker => :progress_worker)
>
>    I would welcome, anyone who contributes more examples back. You can
> even use callbacks
>    or Deferable pattern for invoking callbacks.
>
> === Exciting new stuff
>   * Rock solid stable ( or will be , after few bug reports )
>   * Master Process is using a hash for storing status of each worker.
> I am thinking of adding something like this:
>
>         #backgroundrb.yml
>         status_storage:
>           storage: db
>           database: worker_status
>     Above code would fullfill thy wishes of persitent worker status
> storage. Any more ideas, welcome.
>   * Each worker comes with Event loop of its own and can potentially do
> lots of fancy stuff. Two noteworthy methods are:
>
>          connect(ip,port,Handler)
>          start_worker(ip,port,Handler)
>
>     If you are familiar with EventMachine or Twisted style of network
> programming, above methods allow you to
>     start tcp servers inside your workers or lets you connect to
> external tcp servers. For Each accepted client or
>     connected socket a instance of Handler class would be created and
> integrated with main event loop.
>     This can be used for worker to worker communication between
> backgroundrb servers running on two machines.
>
>     You are encouraged to look into framework directory, and see the
> code that implements all this stuff.Guts of
>     new bdrb is based on this library, which would be released soon as
> separately.
>
> == Online Resources
> - http://svn.devjavu.com/backgroundrb/branches/version10/
> - http://backgroundrb.devjavu.com (trac)
> - http://backgroundrb.rubyforge.org (rdoc)
>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed Nov 14 09:43:28 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 20:13:28 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
	<fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
	<1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>
Message-ID: <fb283d5e0711140643m14494b77qc225dcd097b4acc4@mail.gmail.com>

On Nov 14, 2007 7:57 PM, Brandon Keepers <brandon at opensoul.org> wrote:
>
>
> On Nov 14, 2007, at 8:55 AM, hemant wrote:
>
> > On Nov 14, 2007 6:50 PM, Wood, Peter <p.wood at lancaster.ac.uk> wrote:
> >> Heya all,
> >>
> >> A coworker and I have checked out the new version of BackgrounDRb in
> >> order to build one of our latest projects, so far we've just tried to
> >> replicate some of the functionality we had when testing the old trunk
> >> version.
> >>
> >> I may be missing something obvious, but from the code it doesn't
> >> appear
> >> that you can set off multiple copies of the same worker like you
> >> could
> >> in the old trunk version. We intend to use BackgrounDRb to set off
> >> snmp
> >> scans of various devices on our network, some of which take some
> >> time to
> >> complete. It is likely that this will happen more than once at any
> >> particular moment in time.
> >>
> >> At the moment, using the sample code we've moved the reply to the
> >> MiddleMan to before the work, therefore actually letting the calling
> >> application carry on (which is what we need). However a second call
> >> to
> >> the same worker obviously hangs until the first call has completed.
> >>
> >> I think what I'm essentially saying is that we want multiple
> >> instances
> >> of the same worker which can be identified by a key (as it was in the
> >> trunk). Is this possible with the new version of BackgrounDRb, or
> >> is it
> >> on the map?
> >>
> >
> > With new version you can't have two instances of a worker active at
> > the same time. Although, I do think of having that facility soon, but
> > for the time being you will have to put up with writing copies of
> > workers. BTW, how many copies do you need and how much time, does
> > it take to complete scan of devices on network?
>
> This is a show stopper for me.  The only thing I've ever used
> BackgrounDRb for it firing off workers at will to do various
> processing tasks, such as transferring large files or transcoding
> video.  If I have a quad core CPU, I want 4 instances of the same
> worker running in order to take full advantage of the CPU.
>

Ok, then folks, I will be implementing it. Will be available soonish.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed Nov 14 09:57:42 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 20:27:42 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <fb283d5e0711140643m14494b77qc225dcd097b4acc4@mail.gmail.com>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
	<fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
	<1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>
	<fb283d5e0711140643m14494b77qc225dcd097b4acc4@mail.gmail.com>
Message-ID: <fb283d5e0711140657x171527f2h6165e1f2f1c03f12@mail.gmail.com>

On Nov 14, 2007 8:13 PM, hemant <gethemant at gmail.com> wrote:
>
> On Nov 14, 2007 7:57 PM, Brandon Keepers <brandon at opensoul.org> wrote:
> >
> >
> > On Nov 14, 2007, at 8:55 AM, hemant wrote:
> >
> > > On Nov 14, 2007 6:50 PM, Wood, Peter <p.wood at lancaster.ac.uk> wrote:
> > >> Heya all,
> > >>
> > >> A coworker and I have checked out the new version of BackgrounDRb in
> > >> order to build one of our latest projects, so far we've just tried to
> > >> replicate some of the functionality we had when testing the old trunk
> > >> version.
> > >>
> > >> I may be missing something obvious, but from the code it doesn't
> > >> appear
> > >> that you can set off multiple copies of the same worker like you
> > >> could
> > >> in the old trunk version. We intend to use BackgrounDRb to set off
> > >> snmp
> > >> scans of various devices on our network, some of which take some
> > >> time to
> > >> complete. It is likely that this will happen more than once at any
> > >> particular moment in time.
> > >>
> > >> At the moment, using the sample code we've moved the reply to the
> > >> MiddleMan to before the work, therefore actually letting the calling
> > >> application carry on (which is what we need). However a second call
> > >> to
> > >> the same worker obviously hangs until the first call has completed.
> > >>
> > >> I think what I'm essentially saying is that we want multiple
> > >> instances
> > >> of the same worker which can be identified by a key (as it was in the
> > >> trunk). Is this possible with the new version of BackgrounDRb, or
> > >> is it
> > >> on the map?
> > >>
> > >
> > > With new version you can't have two instances of a worker active at
> > > the same time. Although, I do think of having that facility soon, but
> > > for the time being you will have to put up with writing copies of
> > > workers. BTW, how many copies do you need and how much time, does
> > > it take to complete scan of devices on network?
> >
> > This is a show stopper for me.  The only thing I've ever used
> > BackgrounDRb for it firing off workers at will to do various
> > processing tasks, such as transferring large files or transcoding
> > video.  If I have a quad core CPU, I want 4 instances of the same
> > worker running in order to take full advantage of the CPU.
> >
>
> Ok, then folks, I will be implementing it. Will be available soonish.

Ok, on second thoughts, I was curious about the fact, that can't you
have 4 different workers running at the same time?
Why you need copies of same workers? I know, you don't have to write
additional worker code, if you use the same worker, but
apart from that what?

You can very well have 4 four workers like:

video_encoder_worker.rb
transfer_data_worker.rb
snmp_scanner_worker.rb
port_scanner_worker.rb

You can very well do, whatever you want inside them. Another question
is, how do you schedule workers ( which are instance of same class?).
You
use cron based scheduling or Scheduling using MiddleMan  or what?

I am not trying to dodge, but just want to understand full use case.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From maze at strahlungsfrei.de  Wed Nov 14 09:40:10 2007
From: maze at strahlungsfrei.de (Martin Honermeyer)
Date: Wed, 14 Nov 2007 15:40:10 +0100
Subject: [Backgroundrb-devel] Worker status bug in new version
Message-ID: <200711141540.10201.maze@strahlungsfrei.de>

Hi all,

thanks to hemant for this nice new version of BackgrounDRB! I am trying this 
out in my current project, and it works nicely overall. 

I found a small typo which broke register_status. Attached is a patch to solve 
that.

Furthermore, +1 from me for re-implementing runtime scheduling in the 
MiddleMan! That feature will be needed in our project middle term. Let me 
know if I can be of any help.


Greetz,
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: worker_status_fix.diff
Type: text/x-diff
Size: 341 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071114/6d255e55/attachment.bin 

From brandon at opensoul.org  Wed Nov 14 09:27:22 2007
From: brandon at opensoul.org (Brandon Keepers)
Date: Wed, 14 Nov 2007 09:27:22 -0500
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
	<fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
Message-ID: <1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>


On Nov 14, 2007, at 8:55 AM, hemant wrote:

> On Nov 14, 2007 6:50 PM, Wood, Peter <p.wood at lancaster.ac.uk> wrote:
>> Heya all,
>>
>> A coworker and I have checked out the new version of BackgrounDRb in
>> order to build one of our latest projects, so far we've just tried to
>> replicate some of the functionality we had when testing the old trunk
>> version.
>>
>> I may be missing something obvious, but from the code it doesn't  
>> appear
>> that you can set off multiple copies of the same worker like you  
>> could
>> in the old trunk version. We intend to use BackgrounDRb to set off  
>> snmp
>> scans of various devices on our network, some of which take some  
>> time to
>> complete. It is likely that this will happen more than once at any
>> particular moment in time.
>>
>> At the moment, using the sample code we've moved the reply to the
>> MiddleMan to before the work, therefore actually letting the calling
>> application carry on (which is what we need). However a second call  
>> to
>> the same worker obviously hangs until the first call has completed.
>>
>> I think what I'm essentially saying is that we want multiple  
>> instances
>> of the same worker which can be identified by a key (as it was in the
>> trunk). Is this possible with the new version of BackgrounDRb, or  
>> is it
>> on the map?
>>
>
> With new version you can't have two instances of a worker active at
> the same time. Although, I do think of having that facility soon, but
> for the time being you will have to put up with writing copies of
> workers. BTW, how many copies do you need and how much time, does
> it take to complete scan of devices on network?

This is a show stopper for me.  The only thing I've ever used  
BackgrounDRb for it firing off workers at will to do various  
processing tasks, such as transferring large files or transcoding  
video.  If I have a quad core CPU, I want 4 instances of the same  
worker running in order to take full advantage of the CPU.

Brandon 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071114/c9476221/attachment.bin 

From gethemant at gmail.com  Wed Nov 14 10:02:35 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 14 Nov 2007 20:32:35 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <fb283d5e0711140657x171527f2h6165e1f2f1c03f12@mail.gmail.com>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
	<fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
	<1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>
	<fb283d5e0711140643m14494b77qc225dcd097b4acc4@mail.gmail.com>
	<fb283d5e0711140657x171527f2h6165e1f2f1c03f12@mail.gmail.com>
Message-ID: <fb283d5e0711140702y1a35089fy3cc7dbfed1832a94@mail.gmail.com>

Ok, whoever interested on discussing about future of backgroundrb is
welcome on #backgroundrb between 15.00 - 22.00 (UTC) on freenode.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From eduardodmz at gmail.com  Wed Nov 14 18:11:06 2007
From: eduardodmz at gmail.com (Eduardo Dominguez)
Date: Wed, 14 Nov 2007 17:11:06 -0600
Subject: [Backgroundrb-devel] couple of questions
Message-ID: <bc4bc5d00711141511t68f100d8n10766c470761d878@mail.gmail.com>

I'm using the latest backgroundrb. Do I need to setup a worker in
order to use it with Rails?

I  remember using "Worker::RailsBase" before.

Also, how do I access the logger from within the worker?

Thanks a lot!
-- 
Lalo

From gethemant at gmail.com  Wed Nov 14 23:26:46 2007
From: gethemant at gmail.com (hemant)
Date: Thu, 15 Nov 2007 09:56:46 +0530
Subject: [Backgroundrb-devel] couple of questions
In-Reply-To: <bc4bc5d00711141511t68f100d8n10766c470761d878@mail.gmail.com>
References: <bc4bc5d00711141511t68f100d8n10766c470761d878@mail.gmail.com>
Message-ID: <fb283d5e0711142026x59241d36w170493dfaf1e3487@mail.gmail.com>

On Nov 15, 2007 4:41 AM, Eduardo Dominguez <eduardodmz at gmail.com> wrote:
> I'm using the latest backgroundrb. Do I need to setup a worker in
> order to use it with Rails?
>
> I  remember using "Worker::RailsBase" before.

You do need at least one worker, which would do the processing of
requests from rails.

>
> Also, how do I access the logger from within the worker?

There is no logger. you can create worker specific loggers, in create
method of workers, although.

>
> Thanks a lot!
> --
> Lalo
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Thu Nov 15 01:37:22 2007
From: gethemant at gmail.com (hemant)
Date: Thu, 15 Nov 2007 12:07:22 +0530
Subject: [Backgroundrb-devel] BackgrounDRb version 1.0RC1 available now
In-Reply-To: <fb283d5e0711140702y1a35089fy3cc7dbfed1832a94@mail.gmail.com>
References: <fb283d5e0711132101t62f704c8mc4dfe4798fba8276@mail.gmail.com>
	<89C606D37C22D743B282A6B80CEE040203D8D79B@exchange-be3.lancs.local>
	<fb283d5e0711140555h1d1a8708r7f3d107ad6b9a9e6@mail.gmail.com>
	<1140D140-FAE5-41E0-87F5-CCE155B1582D@opensoul.org>
	<fb283d5e0711140643m14494b77qc225dcd097b4acc4@mail.gmail.com>
	<fb283d5e0711140657x171527f2h6165e1f2f1c03f12@mail.gmail.com>
	<fb283d5e0711140702y1a35089fy3cc7dbfed1832a94@mail.gmail.com>
Message-ID: <fb283d5e0711142237u4b368619m3bd45383ec98b40b@mail.gmail.com>

On Nov 14, 2007 8:32 PM, hemant <gethemant at gmail.com> wrote:
> Ok, whoever interested on discussing about future of backgroundrb is
> welcome on #backgroundrb between 15.00 - 22.00 (UTC) on freenode.

Ok, we did had a sort of discussion and decided to have following features:

1. Log worker.
2. Ability to start & stop workers at runtime.
3. Currently, waiting for long responses in rails is not supported.
People apparently need it.
4. Ability to cancel wrong running tasks.
5. Better examples demonstrating file uploads, progress bar and stuff.

Now, If anyone commits few patches and is ready to work on things, I
would be happy to give commit access.


>
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From emarceta at gmail.com  Fri Nov 16 07:46:25 2007
From: emarceta at gmail.com (Emil Marceta)
Date: Fri, 16 Nov 2007 13:46:25 +0100
Subject: [Backgroundrb-devel] Backgroundrb with Load Balancing Rails engines
Message-ID: <74b000990711160446u1a52f295gaed4f90498984e4f@mail.gmail.com>

Hi there,

We run several (3-4) Rails servers behind the reverse proxy / load
balancing web server serving the  single application. All Rails
instances share the same database and use the database as a session
storage. Memcached is also part of the picture.

I'm looking into using Backgroundrb for some large uploads / parsing
task that provide progress status updates (via ajax calls). Bdrb would
run on each Rails server using drbunix (domain sockets) and not on the
dedicated / shared Bdrb server.

In this scenario it is not known in which Rails engine will the status
update request land. It may or may not hit the engine where the Bdrb
worker has been created( we don't plan for session affinity nor
anything similar). As mentioned earlier the only shared component is
the database, and I was planning to introduce a 'background_jobs'
table - keyed with the worker key and storing the key in the session.
In addition to the worker key the table would have a blob data where
the updates are stored. Then the MiddleMan is basically used only for
starting the job and  the results could be read by any engine.

This looks to me as a fairly 'generic' requirement and I was wandering
whether someone had a similar requirement and what approach was taken,
whether there is something that is part of the framework already. Also
any comments / suggestions are much appreciated.

cheers,
emil

From gethemant at gmail.com  Sat Nov 17 07:43:55 2007
From: gethemant at gmail.com (hemant)
Date: Sat, 17 Nov 2007 18:13:55 +0530
Subject: [Backgroundrb-devel] Backgroundrb with Load Balancing Rails
	engines
In-Reply-To: <74b000990711160446u1a52f295gaed4f90498984e4f@mail.gmail.com>
References: <74b000990711160446u1a52f295gaed4f90498984e4f@mail.gmail.com>
Message-ID: <fb283d5e0711170443r71e289a4rf92901de8653b7c5@mail.gmail.com>

On Nov 16, 2007 6:16 PM, Emil Marceta <emarceta at gmail.com> wrote:
> Hi there,
>
> We run several (3-4) Rails servers behind the reverse proxy / load
> balancing web server serving the  single application. All Rails
> instances share the same database and use the database as a session
> storage. Memcached is also part of the picture.
>
> I'm looking into using Backgroundrb for some large uploads / parsing
> task that provide progress status updates (via ajax calls). Bdrb would
> run on each Rails server using drbunix (domain sockets) and not on the
> dedicated / shared Bdrb server.

Any reasons, to not run bdrb server centrally? Guaranteed, you get the
failover measures and stuff,
but any other reasons?


>
> In this scenario it is not known in which Rails engine will the status
> update request land. It may or may not hit the engine where the Bdrb
> worker has been created( we don't plan for session affinity nor
> anything similar). As mentioned earlier the only shared component is
> the database, and I was planning to introduce a 'background_jobs'
> table - keyed with the worker key and storing the key in the session.
> In addition to the worker key the table would have a blob data where
> the updates are stored. Then the MiddleMan is basically used only for
> starting the job and  the results could be read by any engine.
>
> This looks to me as a fairly 'generic' requirement and I was wandering
> whether someone had a similar requirement and what approach was taken,
> whether there is something that is part of the framework already. Also
> any comments / suggestions are much appreciated.

Ok, Are you aware of the new version of bdrb? Its a complete rewrite.
I would advise, you to checkout new version and implement your project on
top of it.

In  new bdrb, master process is nothing but a tcp server and hence you can
connect from any of the machines in your cluster.

checkout the code from here:

http://svn.devjavu.com/backgroundrb/branches/version10/

go through the read file, it should be suffice to get started.




-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From adamjroth at gmail.com  Tue Nov 20 19:06:34 2007
From: adamjroth at gmail.com (Adam Roth)
Date: Tue, 20 Nov 2007 16:06:34 -0800
Subject: [Backgroundrb-devel] Is fork { } safe inside of a worker? How else
	may this be done...
Message-ID: <390fc99d0711201606r4708f679l4420e1a1df0c1a5c@mail.gmail.com>

I have a BackgrounDRb worker running every 2 minutes. Here's the basic idea:

- events = Event.find_all_pending( 2.minutes )
- while( all events have yet to be processed )
---- if( Time.now <= event.time_to_process )
------- fork { process_event }
--- end
--- sleep( 0.25 )
- end
- puts "Done..."

Events that need to be processed within the next 2 minutes are loaded and
iterated over. If its time to process the event, fork and do so. Sleep for
0.25 seconds between loops until all events have been processed.

I have a few questions:

- Is this safe? Specifically calling fork within a BackgroundRb worker...
Some events may fire at the same exact time and cannot be delayed. This is a
very time sensitive application.
- Is there a better way to handle this scenario?

Thanks
Adam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071120/5199f61d/attachment.html 

From joevandyk at gmail.com  Tue Nov 20 19:29:46 2007
From: joevandyk at gmail.com (Joe Van Dyk)
Date: Tue, 20 Nov 2007 16:29:46 -0800
Subject: [Backgroundrb-devel] Is fork { } safe inside of a worker? How
	else may this be done...
In-Reply-To: <390fc99d0711201606r4708f679l4420e1a1df0c1a5c@mail.gmail.com>
References: <390fc99d0711201606r4708f679l4420e1a1df0c1a5c@mail.gmail.com>
Message-ID: <c715e640711201629t2784e4eci8f3eb8bb99e69faf@mail.gmail.com>

Possibly a related question, If I'm making activerecord updates inside
multiple brdb workers at the same time, that should be fine, right?
Wasn't sure how concurrency worked.

On Nov 20, 2007 4:06 PM, Adam Roth <adamjroth at gmail.com> wrote:
> I have a BackgrounDRb worker running every 2 minutes. Here's the basic idea:
>
> - events = Event.find_all_pending( 2.minutes )
> - while( all events have yet to be processed )
> ---- if( Time.now <= event.time_to_process )
> ------- fork { process_event }
> --- end
> --- sleep( 0.25 )
> - end
> - puts "Done..."
>
> Events that need to be processed within the next 2 minutes are loaded and
> iterated over. If its time to process the event, fork and do so. Sleep for
> 0.25 seconds between loops until all events have been processed.
>
> I have a few questions:
>
> - Is this safe? Specifically calling fork within a BackgroundRb worker...
> Some events may fire at the same exact time and cannot be delayed. This is a
> very time sensitive application.
> - Is there a better way to handle this scenario?
>
> Thanks
> Adam
>
>
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Tue Nov 20 20:59:15 2007
From: gethemant at gmail.com (hemant)
Date: Wed, 21 Nov 2007 07:29:15 +0530
Subject: [Backgroundrb-devel] Is fork { } safe inside of a worker? How
	else may this be done...
In-Reply-To: <c715e640711201629t2784e4eci8f3eb8bb99e69faf@mail.gmail.com>
References: <390fc99d0711201606r4708f679l4420e1a1df0c1a5c@mail.gmail.com>
	<c715e640711201629t2784e4eci8f3eb8bb99e69faf@mail.gmail.com>
Message-ID: <fb283d5e0711201759m752561a3p2931a80387f9c339@mail.gmail.com>

On Nov 21, 2007 5:59 AM, Joe Van Dyk <joevandyk at gmail.com> wrote:
> Possibly a related question, If I'm making activerecord updates inside
> multiple brdb workers at the same time, that should be fine, right?
> Wasn't sure how concurrency worked.

Normally your database handles such things. Its no problem at all.

>
>
> On Nov 20, 2007 4:06 PM, Adam Roth <adamjroth at gmail.com> wrote:
> > I have a BackgrounDRb worker running every 2 minutes. Here's the basic idea:
> >
> > - events = Event.find_all_pending( 2.minutes )
> > - while( all events have yet to be processed )
> > ---- if( Time.now <= event.time_to_process )
> > ------- fork { process_event }
> > --- end
> > --- sleep( 0.25 )
> > - end
> > - puts "Done..."
> >
> > Events that need to be processed within the next 2 minutes are loaded and
> > iterated over. If its time to process the event, fork and do so. Sleep for
> > 0.25 seconds between loops until all events have been processed.
> >
> > I have a few questions:
> >
> > - Is this safe? Specifically calling fork within a BackgroundRb worker...
> > Some events may fire at the same exact time and cannot be delayed. This is a
> > very time sensitive application.
> > - Is there a better way to handle this scenario?

Which version, are you using older bdrb version, which is still on
trunk or new version which is available here

http://svn.devjavu.com/backgroundrb/branches/version10/

Please try above new pre-release version, if it suits you. There are
some API differences, but i guess it will work for you.

Ok, back to original question. Its generally safe to fork within
workers, they are just another ruby programs, so shouldn't be
much of a problem there. But the question is, how much time, does
process_event takes?

If window period, between each fire is greater than, time taken by one
invocation of process_event, then you don't need fork.
With new version, each worker has a event loop of its own and hence
you can priortize, firing of events. In older version, it was the
master, who scheduled timer/events in workers. In new version, each
worker is responsible for its own scheduling.

If time window is long enough, you dont need fork. However, I am
working on couple of optimization with new version, so as we don't
search for timers, all the way down to last hash value.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From jazzyy at gmail.com  Thu Nov 22 09:40:06 2007
From: jazzyy at gmail.com (Nii Amon Dsane)
Date: Thu, 22 Nov 2007 14:40:06 +0000
Subject: [Backgroundrb-devel] How to get result from do_work in a worker
In-Reply-To: <7cf2a5480711220630m67ab5814n1ae9352c5fa35be3@mail.gmail.com>
References: <7cf2a5480711220630m67ab5814n1ae9352c5fa35be3@mail.gmail.com>
Message-ID: <7cf2a5480711220640k389d4aa2ke8fd144f869b9a9a@mail.gmail.com>

Hi guys,

I have a problem that I would some need help with. I have a long-running
task that blocks my server when it's processing. Now I refactored that into
a backgroundrb worker class that does the processing for the action. Now my
problem is how to retrieve the result for the task when the worker is done
processing since the MiddleMan.new_worker returns just the job_key.


Any ideas about how I could retrieve the result from the worker class?

Thanks.
Nii Amon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071122/400d5e7d/attachment.html 

From felixclack at gmail.com  Sat Nov 24 20:43:22 2007
From: felixclack at gmail.com (felix clack)
Date: Sun, 25 Nov 2007 12:43:22 +1100
Subject: [Backgroundrb-devel] Fwd: Render_to_string from a worker
In-Reply-To: <85dc64750711181935i1e95eefat6187c78625cc456@mail.gmail.com>
References: <85dc64750711181935i1e95eefat6187c78625cc456@mail.gmail.com>
Message-ID: <85dc64750711241743r17250d5as83093b887ce16892@mail.gmail.com>

Hi,
I am trying to generate a PDF from within a worker using PDF::Writer. In my
controller I use render_to_string to create the pdf file however from what I
understand render_to_string can only be used from a controller.

Does anybody have any suggestions how I might be able to replicate this
functionality in a worker?

-- 
thanks,
Felix
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071125/2a43cba6/attachment.html 

From gethemant at gmail.com  Sat Nov 24 22:16:30 2007
From: gethemant at gmail.com (hemant kumar)
Date: Sun, 25 Nov 2007 08:46:30 +0530
Subject: [Backgroundrb-devel] Fwd: Render_to_string from a worker
In-Reply-To: <85dc64750711241743r17250d5as83093b887ce16892@mail.gmail.com>
References: <85dc64750711181935i1e95eefat6187c78625cc456@mail.gmail.com>
	<85dc64750711241743r17250d5as83093b887ce16892@mail.gmail.com>
Message-ID: <1195960590.8358.34.camel@shire>

You are using render_to_string for generating pdf? You must be using one
of those groovy plugins i suppose.

Well if, yes... i suppose, it would be available in worker as well...
But can't say much unless i know, which plugin you are using and
stuff...


On Sun, 2007-11-25 at 12:43 +1100, felix clack wrote:
> Hi,
> 
> 
> I am trying to generate a PDF from within a worker using PDF::Writer.
> In my controller I use render_to_string to create the pdf file however
> from what I understand render_to_string can only be used from a
> controller.
> 
> 
> Does anybody have any suggestions how I might be able to replicate
> this functionality in a worker?
> 
> -- 
> thanks,
> Felix
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From felixclack at gmail.com  Tue Nov 27 23:40:57 2007
From: felixclack at gmail.com (felix clack)
Date: Wed, 28 Nov 2007 15:40:57 +1100
Subject: [Backgroundrb-devel]  Fwd: Render_to_string from a worker
In-Reply-To: <85dc64750711250033p2666d99bxefb18d4c1fe03bb@mail.gmail.com>
References: <85dc64750711181935i1e95eefat6187c78625cc456@mail.gmail.com>
	<85dc64750711241743r17250d5as83093b887ce16892@mail.gmail.com>
	<1195960590.8358.34.camel@shire>
	<85dc64750711250033p2666d99bxefb18d4c1fe03bb@mail.gmail.com>
Message-ID: <85dc64750711272040o2b00a8a5h292d31287fb947a3@mail.gmail.com>

Hi Hemant,

I found the plugin and explanation how to use it at
http://blog.nicksieger.com/articles/2006/05/26/rails-is-simpler-than-office


It registers the extension .rpdf to trigger code in the pdfrender plugin.
I use similar view code to produce a set of addresses for avery labels as in
the example blog post.

I have a controller action that prepares the data for the labels, I use
render_to_string to create and save the pdf based on the view code, and then
send an email with the newly created pdf attached.

This works fine when triggered manually from my app, however I came unstuck
when trying to implement this same process in a worker.

I hope this makes sense. Not sure if this is the best way to do what I'm
trying to achieve but pdfrender was the first solution where I could
understand how it all came together.

Any help on getting this running in a worker would be appreciated.

thanks
Felix

On 25/11/2007, hemant kumar <gethemant at gmail.com > wrote:
>
> You are using render_to_string for generating pdf? You must be using one
> of those groovy plugins i suppose.
>
> Well if, yes... i suppose, it would be available in worker as well...
> But can't say much unless i know, which plugin you are using and
> stuff...
>
>
> On Sun, 2007-11-25 at 12:43 +1100, felix clack wrote:
> > Hi,
> >
> >
> > I am trying to generate a PDF from within a worker using PDF::Writer.
> > In my controller I use render_to_string to create the pdf file however
> > from what I understand render_to_string can only be used from a
> > controller.
> >
> >
> > Does anybody have any suggestions how I might be able to replicate
> > this functionality in a worker?
> >
> > --
> > thanks,
> > Felix
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>


-- 
thanks,
Felix

-- 
thanks,
Felix
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071128/ebb24a9b/attachment-0001.html 

From peter at flippyhead.com  Fri Nov 30 13:57:12 2007
From: peter at flippyhead.com (Peter Brown)
Date: Fri, 30 Nov 2007 10:57:12 -0800
Subject: [Backgroundrb-devel] which version of backgroundrb to use
Message-ID: <c20d35f00711301057t3fa42306s1175c8b9eb4f9f9a@mail.gmail.com>

Hello

I've been using backgroundrb for a while now and love it.


I've notices some recent updates and am confused as to which branch/tag is
what.


What's the difference between these:


new_core/
version010/
version099/
version10/


and trunk, and where does "0.2.2" fit in?




Thanks!


-- 
Peter T. Brown
peter at wagglelabs.com
http://wagglelabs.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071130/cf3f9867/attachment.html 

From eimorton at gmail.com  Fri Nov 30 14:17:46 2007
From: eimorton at gmail.com (Erik Morton)
Date: Fri, 30 Nov 2007 14:17:46 -0500
Subject: [Backgroundrb-devel] backgroundrb not cleaning up tmp files?
Message-ID: <84F1E20A-7F11-4896-A054-F035A3B05C0C@gmail.com>

I just had an interesting issue on a production system that has been  
running nicely for about a year. There were approximately 10,000  
backgroundrb temp files in the /tmp directory. It looks like  
backgroundrb wasn't cleaning them up.

I noticed it with the following error:
DRb::DRbConnError: drbunix:///tmp/backgroundrb.20015/ 
backgroundrb_logger_0_0.101544829808665 - #<Errno::ENOENT: No such  
file or directory - ///tmp/backgroundrb.20015/ 
backgroundrb_logger_0_0.101544829808665>
  /usr/lib/ruby/1.8/drb/drb.rb:736:in `open'
  /usr/lib/ruby/1.8/drb/drb.rb:729:in `each'
  /usr/lib/ruby/1.8/drb/drb.rb:729:in `open'
  /usr/lib/ruby/1.8/drb/drb.rb:1189:in `initialize'
  /usr/lib/ruby/1.8/drb/drb.rb:1169:in `new'
  /usr/lib/ruby/1.8/drb/drb.rb:1169:in `open'
  /usr/lib/ruby/1.8/drb/drb.rb:1085:in `method_missing'
  /usr/lib/ruby/1.8/drb/drb.rb:1103:in `with_friend'
  /usr/lib/ruby/1.8/drb/drb.rb:1084:in `method_missing'
  /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ 
server/lib/backgroundrb/middleman.rb:42:in `initialize'
  /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ 
server/lib/backgroundrb/middleman.rb:420:in `new'
  /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ 
server/lib/backgroundrb/middleman.rb:420:in `worker'
  /apps/eisweb/releases/20071120022316/vendor/plugins/backgroundrb/ 
server/lib/backgroundrb/worker.rb:37:in `logger'
  #{RAILS_ROOT}/app/helpers/distribution_helper.rb:4:in  
`do_distribution'

Has anyone else seen this?