From normelton at gmail.com  Fri Feb  1 13:58:31 2008
From: normelton at gmail.com (Norman Elton)
Date: Fri, 1 Feb 2008 13:58:31 -0500
Subject: [Backgroundrb-devel] Job Queue
Message-ID: <0A5D4B92-7B0D-42CC-9F77-FA9ABD500BBC@gmail.com>

After some initial stumblings, I think I've got the hang of  
backgroundrb. It's great! I'd been thinking for many many months how  
cool something like this would be!

I'm trying to make a "job queue". That is, a pool of worker threads  
monitor a queue. When a job appears, one of the workers grabs it and  
executes the task. When complete, the worker returns to watching the  
queue. If all workers are busy, jobs simply sit in the queue until the  
resources are available.

I can see how to build an array containing jobs. And a mutex could be  
used to ensure thread-safe access to the array. But how can the  
workers "go to sleep" until a job arrives?

Any thoughts?

Thanks again!

Norman

From leavengood at gmail.com  Fri Feb  1 14:19:23 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Fri, 1 Feb 2008 14:19:23 -0500
Subject: [Backgroundrb-devel] Job Queue
In-Reply-To: <0A5D4B92-7B0D-42CC-9F77-FA9ABD500BBC@gmail.com>
References: <0A5D4B92-7B0D-42CC-9F77-FA9ABD500BBC@gmail.com>
Message-ID: <a4855ac50802011119p7f54ea06o74f884a143345865@mail.gmail.com>

On Feb 1, 2008 1:58 PM, Norman Elton <normelton at gmail.com> wrote:
>
> I can see how to build an array containing jobs. And a mutex could be
> used to ensure thread-safe access to the array. But how can the
> workers "go to sleep" until a job arrives?
>
> Any thoughts?

I had a similar requirement and ended up just using a single worker
and thread_pool.defer (the documentation talks about how to use this.)
I've looked at the BackgrounDRb code and thread_pool.defer makes use
of a queue which a configurable number of threads read from. You can
set the size of the pool by calling pool_size in your worker. The
default pool size is 20 threads.

If you need to know the status of each thread you need to use a mutex
to synchronize access to a hashtable member variable which can contain
the status for each thread (you would need to decide what the key
would be for each thread, maybe the database ID of the job the thread
is processing.) You would then pass this hashtable to the
register_status method. There is a post from hemant on this mailing
list from December which shows how to do this pretty well:

http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001170.html

The only caveat here is the green Ruby threads used in the thread_pool
may not play well with the job processing you are doing. But from the
testing I've done it seems pretty good, especially for something
involving the network which probably has a lot of latency.

The general architecture I would suggest would be to have a jobs table
in the database, and when jobs are added the Rails model
(after_create) can call MiddleMan.ask_work and pass the ID of the job
just created. The worker will pass that job_id into thread_pool.defer
which will then process the job. For my own work I tend to put all the
heavy processing into separate classes which I simply call from the
worker. So for you maybe something like JobProcessor.run(job_id) or
whatever.

Regards,
Ryan

From jason.lapier at gmail.com  Fri Feb  1 14:37:22 2008
From: jason.lapier at gmail.com (Jason LaPier)
Date: Fri, 1 Feb 2008 11:37:22 -0800
Subject: [Backgroundrb-devel] Job Queue
In-Reply-To: <a4855ac50802011119p7f54ea06o74f884a143345865@mail.gmail.com>
References: <0A5D4B92-7B0D-42CC-9F77-FA9ABD500BBC@gmail.com>
	<a4855ac50802011119p7f54ea06o74f884a143345865@mail.gmail.com>
Message-ID: <93b44c1f0802011137q6be9ed03l63354883329560b5@mail.gmail.com>

I was just about to reply, but you linked to my code sample already :)

The bugs we were discussing in that thread have been resolved and I've
been using thread pool in combination with mutex to save a hash of
"statuses" as I described in that post for a little over a month now
in production with no problems. Admittedly, the site in question is
not exactly high traffic (75k pageviews for the month of January
according to analytics), but I've had no stability issues whatsoever
in the last month. I run a pool of 10 threads currently, but the job
is fairly short so I doubt I'm ever using more than a couple threads
at once.

- Jason L.

-- 
My Rails and Linux Blog: http://offtheline.net


On Feb 1, 2008 11:19 AM, Ryan Leavengood <leavengood at gmail.com> wrote:
> On Feb 1, 2008 1:58 PM, Norman Elton <normelton at gmail.com> wrote:
> >
> > I can see how to build an array containing jobs. And a mutex could be
> > used to ensure thread-safe access to the array. But how can the
> > workers "go to sleep" until a job arrives?
> >
> > Any thoughts?
>
> I had a similar requirement and ended up just using a single worker
> and thread_pool.defer (the documentation talks about how to use this.)
> I've looked at the BackgrounDRb code and thread_pool.defer makes use
> of a queue which a configurable number of threads read from. You can
> set the size of the pool by calling pool_size in your worker. The
> default pool size is 20 threads.
>
> If you need to know the status of each thread you need to use a mutex
> to synchronize access to a hashtable member variable which can contain
> the status for each thread (you would need to decide what the key
> would be for each thread, maybe the database ID of the job the thread
> is processing.) You would then pass this hashtable to the
> register_status method. There is a post from hemant on this mailing
> list from December which shows how to do this pretty well:
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001170.html
>
> The only caveat here is the green Ruby threads used in the thread_pool
> may not play well with the job processing you are doing. But from the
> testing I've done it seems pretty good, especially for something
> involving the network which probably has a lot of latency.
>
> The general architecture I would suggest would be to have a jobs table
> in the database, and when jobs are added the Rails model
> (after_create) can call MiddleMan.ask_work and pass the ID of the job
> just created. The worker will pass that job_id into thread_pool.defer
> which will then process the job. For my own work I tend to put all the
> heavy processing into separate classes which I simply call from the
> worker. So for you maybe something like JobProcessor.run(job_id) or
> whatever.
>
> Regards,
> Ryan
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Fri Feb  1 14:51:05 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 2 Feb 2008 01:21:05 +0530
Subject: [Backgroundrb-devel] Job Queue
In-Reply-To: <93b44c1f0802011137q6be9ed03l63354883329560b5@mail.gmail.com>
References: <0A5D4B92-7B0D-42CC-9F77-FA9ABD500BBC@gmail.com>
	<a4855ac50802011119p7f54ea06o74f884a143345865@mail.gmail.com>
	<93b44c1f0802011137q6be9ed03l63354883329560b5@mail.gmail.com>
Message-ID: <fb283d5e0802011151o6d9a6f3bi39833db572763704@mail.gmail.com>

On Sat, Feb 2, 2008 at 1:07 AM, Jason LaPier <jason.lapier at gmail.com> wrote:
> I was just about to reply, but you linked to my code sample already :)
>
>  The bugs we were discussing in that thread have been resolved and I've
>  been using thread pool in combination with mutex to save a hash of
>  "statuses" as I described in that post for a little over a month now
>  in production with no problems. Admittedly, the site in question is
>  not exactly high traffic (75k pageviews for the month of January
>  according to analytics), but I've had no stability issues whatsoever
>  in the last month. I run a pool of 10 threads currently, but the job
>  is fairly short so I doubt I'm ever using more than a couple threads
>  at once.
>

I have been toying with some code that I picked other Ruby projects
for implementing a job queue based on database tables.
But i absolutely don't want to add any features without test cases in hand.

But yeah, as told by Ryan and Json, you should have no trouble in
using thread_pool feature.

From caifara.subscribe at gmail.com  Sat Feb  2 11:01:16 2008
From: caifara.subscribe at gmail.com (Ivo Dancet)
Date: Sat, 2 Feb 2008 17:01:16 +0100
Subject: [Backgroundrb-devel] Suspended start of task not suspended
Message-ID: <5F4F39FB-B9A8-4593-B9C6-D11AC7313DB4@gmail.com>

Hi

I thought this should suspend the task by 1 minute, but it starts  
immediately:

MiddleMan.ask_work( :worker => :bar_worker, :worker_method  
=> :test_method, :trigger_args => { :start => (Time.now + 1.minute)})

Is the start argument not allowed in the new backgroundrb (mine is at  
rev HEAD (=314))? What can I do about this?

I really need to suspend starting the task as I want the client to  
start the trigger but the task itself should wait till about midnight.

Thanks

From gethemant at gmail.com  Sat Feb  2 11:57:19 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 2 Feb 2008 22:27:19 +0530
Subject: [Backgroundrb-devel] Suspended start of task not suspended
In-Reply-To: <5F4F39FB-B9A8-4593-B9C6-D11AC7313DB4@gmail.com>
References: <5F4F39FB-B9A8-4593-B9C6-D11AC7313DB4@gmail.com>
Message-ID: <fb283d5e0802020857h2103a17bma7594586af54c9c7@mail.gmail.com>

On Feb 2, 2008 9:31 PM, Ivo Dancet <caifara.subscribe at gmail.com> wrote:
> Hi
>
> I thought this should suspend the task by 1 minute, but it starts
> immediately:
>
> MiddleMan.ask_work( :worker => :bar_worker, :worker_method
> => :test_method, :trigger_args => { :start => (Time.now + 1.minute)})
>

I do not think passing trigger_args like that is supported at all. No,
it won't work like that.

> Is the start argument not allowed in the new backgroundrb (mine is at
> rev HEAD (=314))? What can I do about this?


Yes it should be pretty simple to have an implementation like this:

# don't execute the task now, but delay it by half an hour
def test_method(args = nil)
  add_timer(30*60) { run_actual_method(args) }
end

def run_actual_method args
  # execute the delayed task
end

I know, its not probably very elegant, but should suffice.
Let us know, if we missed something in above simplistic implementation.
-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Sat Feb  2 12:00:15 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 2 Feb 2008 22:30:15 +0530
Subject: [Backgroundrb-devel] Suspended start of task not suspended
In-Reply-To: <fb283d5e0802020857h2103a17bma7594586af54c9c7@mail.gmail.com>
References: <5F4F39FB-B9A8-4593-B9C6-D11AC7313DB4@gmail.com>
	<fb283d5e0802020857h2103a17bma7594586af54c9c7@mail.gmail.com>
Message-ID: <fb283d5e0802020900q3df8c6d7r32b2f893fc29d42e@mail.gmail.com>

On Feb 2, 2008 10:27 PM, hemant <gethemant at gmail.com> wrote:
>
> Yes it should be pretty simple to have an implementation like this:
>
> # don't execute the task now, but delay it by half an hour
> def test_method(args = nil)
>   add_timer(30*60) { run_actual_method(args) }
> end
>
> def run_actual_method args
>   # execute the delayed task
> end
>

Also, if you need to set that time dynamically from rails, then you
can pass time as an argument to test_method,

MiddleMan.ask_work(:worker => :foo_worker,:worker_method =>
:test_method, :data => { :firetime => 60,:args => "bar"})

and in test_method

def test_method args
  add_timer(args[:firetime]) { fire_method(args[:args]) }
end

From caifara.subscribe at gmail.com  Sat Feb  2 14:20:12 2008
From: caifara.subscribe at gmail.com (Ivo Dancet)
Date: Sat, 2 Feb 2008 20:20:12 +0100
Subject: [Backgroundrb-devel] Suspended start of task not suspended
In-Reply-To: <fb283d5e0802020857h2103a17bma7594586af54c9c7@mail.gmail.com>
References: <5F4F39FB-B9A8-4593-B9C6-D11AC7313DB4@gmail.com>
	<fb283d5e0802020857h2103a17bma7594586af54c9c7@mail.gmail.com>
Message-ID: <5D74BD7C-492F-40EA-AA8A-F3EB1AF13E59@gmail.com>


Op 2-feb-08, om 17:57 heeft hemant het volgende geschreven:

> On Feb 2, 2008 9:31 PM, Ivo Dancet <caifara.subscribe at gmail.com>  
> wrote:
>> Hi
>>
>> I thought this should suspend the task by 1 minute, but it starts
>> immediately:
>>
>> MiddleMan.ask_work( :worker => :bar_worker, :worker_method
>> => :test_method, :trigger_args => { :start => (Time.now + 1.minute)})
>>
>
> I do not think passing trigger_args like that is supported at all. No,
> it won't work like that.


ok, hmmm, I'm going to have to look at it a bit closer. I thought it  
was going to work because it used to work in the previous  
backgroundrb. I relied heavily on that feature. Going to have to  
rewrite some code...

Thanks for the quick answer!


From iambenbryant at gmail.com  Sat Feb  2 23:56:49 2008
From: iambenbryant at gmail.com (Benjamin H. Bryant)
Date: Sat, 2 Feb 2008 23:56:49 -0500
Subject: [Backgroundrb-devel] Defer multiple methods within the same worker
	to the thread pool?
Message-ID: <ED7D2422-E0D2-4C0A-B41B-AFFB45C0E6D4@gmail.com>

Within a threaded worker, I would like to have multiple methods.  Will  
the following code work?

def method_1(args)

   thread_pool.defer(args) do |args|
     #work
   end

end

def method_2(args)

   thread_pool.defer(args) do |args|
    #work
   end

end

  . . . or, should I have one "meta"-method and pass which sub-method  
to perform via args?

def meta_method(args)

   thread_pool.defer(args) do |args|

    if args[:method] == "method_1"
      #do work
    end

    if args[:method] == "method_2"
      #do work
    end

   end

end




Thanks,
Ben

From leavengood at gmail.com  Sun Feb  3 00:02:44 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Sun, 3 Feb 2008 00:02:44 -0500
Subject: [Backgroundrb-devel] Defer multiple methods within the same
	worker to the thread pool?
In-Reply-To: <ED7D2422-E0D2-4C0A-B41B-AFFB45C0E6D4@gmail.com>
References: <ED7D2422-E0D2-4C0A-B41B-AFFB45C0E6D4@gmail.com>
Message-ID: <a4855ac50802022102n366ebe0bwe2b5b9e5b529757f@mail.gmail.com>

On Feb 2, 2008 11:56 PM, Benjamin H. Bryant <iambenbryant at gmail.com> wrote:
> Within a threaded worker, I would like to have multiple methods.  Will
> the following code work?
>
> def method_1(args)
>
>    thread_pool.defer(args) do |args|
>      #work
>    end
>
> end
>
> def method_2(args)
>
>    thread_pool.defer(args) do |args|
>     #work
>    end
>
> end

Yes that will work fine. The thread_pool.defer call just packages the
block and arguments into a class which is put into a queue. The
thread_pool pulls from that queue and executes the packaged code in
the order it was received.

Since I was making use of thread_pool.defer myself I read the source
to see what it was doing, which is nothing too crazy.

Regards,
Ryan

From mengkuan at gmail.com  Mon Feb  4 03:13:37 2008
From: mengkuan at gmail.com (Meng Kuan)
Date: Mon, 4 Feb 2008 16:13:37 +0800
Subject: [Backgroundrb-devel] Schedule parser
In-Reply-To: <fb283d5e0801310928o4731ae4ekba5ea2cd21e2eaaf@mail.gmail.com>
References: <20080130181117.EBDA718585FE@rubyforge.org>
	<38BC7A31-5E72-4579-B739-A1D4B794EE38@gmail.com>
	<74C80CDB-9F63-45AD-8C81-E46D33B02F90@gmail.com>
	<fb283d5e0801310928o4731ae4ekba5ea2cd21e2eaaf@mail.gmail.com>
Message-ID: <47753545-385D-492C-9C10-114DF8558956@gmail.com>


On 01 Feb 2008, at 1:28 AM, hemant wrote:

>>  The above still did not solve the problem. So I dug deeper and found
>>  a possible logic bug in cron_trigger.rb file. It appears that
>>  cron_trigger incremented the day value beyond the number of days of
>>  the current month (from 31 to 32) and this caused the Time.local  
>> call
>>  to abort.
>>
>>  Here's my patch to adjust for this.
>>
>>  $ diff -c3 cron_trigger.rb cron_trigger.rb.new
>>  *** cron_trigger.rb     Thu Dec 20 17:27:47 2007
>>  --- cron_trigger.rb.new Thu Jan 31 14:25:57 2008
>>  ***************
>>  *** 126,131 ****
>>  --- 126,135 ----
>>              if next_hour < hour
>>                hour = next_hour
>>                day += 1
>>  +             if day > month_days(year, month)
>>  +               day -= month_days(year, month)
>>  +               month += 1
>>  +             end
>>                retry
>>              end
>>              hour = next_hour
>>
>>
>>  This seems to finally fix the problem for me.
>
> Thanks Meng, We will have this patch in.
>


Found another bug in cron_trigger. When I specify the hour as a range  
like "8-17", the parse_part method returns an empty array. This is  
because Range.new handles string arguments and integer arguments  
differently. Compare

   irb(main):001:0> Range.new("8","17")
   => "8".."17"
   irb(main):002:0> Range.new("8","17").to_a
   => []

to this:

   irb(main):003:0> Range.new(8, 17).to_a
   => [8, 9, 10, 11, 12, 13, 14, 15, 16, 17]


The fix is to convert the argument to integer first before handing it  
to Range.new. See the following patch.


$ diff -u cron_trigger.rb cron_trigger.rb.new
--- cron_trigger.rb     2007-12-20 17:27:47.000000000 +0800
+++ cron_trigger.rb.new 2008-02-04 16:04:53.000000000 +0800
@@ -215,7 +215,7 @@
        r = Array.new
        part.split(',').each do |p|
          if p =~ /-/  # 0-5
-          r << Range.new(*p.scan(/\d+/)).to_a.map do |x| x.to_i end
+          r << Range.new(*p.scan(/\d+/).map {|x| x.to_i}).to_a
          elsif p =~ /(\*|\d+)\/(\d+)/ and not range.nil?  # */5, 2/10
            min = $1 == '*' ? 0 : $1.to_i
            inc = $2.to_i

From gethemant at gmail.com  Mon Feb  4 06:52:16 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 4 Feb 2008 17:22:16 +0530
Subject: [Backgroundrb-devel] Schedule parser
In-Reply-To: <47753545-385D-492C-9C10-114DF8558956@gmail.com>
References: <20080130181117.EBDA718585FE@rubyforge.org>
	<38BC7A31-5E72-4579-B739-A1D4B794EE38@gmail.com>
	<74C80CDB-9F63-45AD-8C81-E46D33B02F90@gmail.com>
	<fb283d5e0801310928o4731ae4ekba5ea2cd21e2eaaf@mail.gmail.com>
	<47753545-385D-492C-9C10-114DF8558956@gmail.com>
Message-ID: <fb283d5e0802040352q131b67f6h53350a188a52bbff@mail.gmail.com>

On Feb 4, 2008 1:43 PM, Meng Kuan <mengkuan at gmail.com> wrote:
>
>
> On 01 Feb 2008, at 1:28 AM, hemant wrote:
>
> >>  The above still did not solve the problem. So I dug deeper and found
> >>  a possible logic bug in cron_trigger.rb file. It appears that
> >>  cron_trigger incremented the day value beyond the number of days of
> >>  the current month (from 31 to 32) and this caused the Time.local
> >> call
> >>  to abort.
> >>
> >>  Here's my patch to adjust for this.
> >>
> >>  $ diff -c3 cron_trigger.rb cron_trigger.rb.new
> >>  *** cron_trigger.rb     Thu Dec 20 17:27:47 2007
> >>  --- cron_trigger.rb.new Thu Jan 31 14:25:57 2008
> >>  ***************
> >>  *** 126,131 ****
> >>  --- 126,135 ----
> >>              if next_hour < hour
> >>                hour = next_hour
> >>                day += 1
> >>  +             if day > month_days(year, month)
> >>  +               day -= month_days(year, month)
> >>  +               month += 1
> >>  +             end
> >>                retry
> >>              end
> >>              hour = next_hour
> >>
> >>
> >>  This seems to finally fix the problem for me.
> >
> > Thanks Meng, We will have this patch in.
> >
>
>
> Found another bug in cron_trigger. When I specify the hour as a range
> like "8-17", the parse_part method returns an empty array. This is
> because Range.new handles string arguments and integer arguments
> differently. Compare
>
>    irb(main):001:0> Range.new("8","17")
>    => "8".."17"
>    irb(main):002:0> Range.new("8","17").to_a
>    => []
>
> to this:
>
>    irb(main):003:0> Range.new(8, 17).to_a
>    => [8, 9, 10, 11, 12, 13, 14, 15, 16, 17]
>
>
> The fix is to convert the argument to integer first before handing it
> to Range.new. See the following patch.
>
>
> $ diff -u cron_trigger.rb cron_trigger.rb.new
> --- cron_trigger.rb     2007-12-20 17:27:47.000000000 +0800
> +++ cron_trigger.rb.new 2008-02-04 16:04:53.000000000 +0800
> @@ -215,7 +215,7 @@
>         r = Array.new
>         part.split(',').each do |p|
>           if p =~ /-/  # 0-5
> -          r << Range.new(*p.scan(/\d+/)).to_a.map do |x| x.to_i end
> +          r << Range.new(*p.scan(/\d+/).map {|x| x.to_i}).to_a
>           elsif p =~ /(\*|\d+)\/(\d+)/ and not range.nil?  # */5, 2/10
>             min = $1 == '*' ? 0 : $1.to_i
>             inc = $2.to_i
>

Send me your Devjavu and Rubyforge account logins, I will add you as a
developer, so that you can have commit access.

From scott at shefield.com  Mon Feb  4 14:07:20 2008
From: scott at shefield.com (Scott Ward)
Date: Mon, 4 Feb 2008 11:07:20 -0800
Subject: [Backgroundrb-devel] Best way to organize workers
In-Reply-To: <5D74BD7C-492F-40EA-AA8A-F3EB1AF13E59@gmail.com>
Message-ID: <20080204190812.1E56D18585CF@rubyforge.org>

Hello,

I have a few tasks that run every few minutes.  They definitely need their
own worker.

However, other tasks run once per day or even once per week.  Right now I am
creating a new worker for each of these tasks, but I wonder if that's
unnecessary for tasks that I know will never execute at the same time.  They
don't need to run concurrently.

Basically, I am wondering if I should combine a bunch of these periodic,
non-overlapping tasks into one worker called "general_maintenance_worker" or
something like that.  For each task, I could create a different method and
schedule them as needed in backgroundrb.yml.

That should work, but is it "right"?  What's the "best practice" is in this
situation?

Thanks,

Scott


From gethemant at gmail.com  Mon Feb  4 14:25:40 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 5 Feb 2008 00:55:40 +0530
Subject: [Backgroundrb-devel] Best way to organize workers
In-Reply-To: <20080204190812.1E56D18585CF@rubyforge.org>
References: <5D74BD7C-492F-40EA-AA8A-F3EB1AF13E59@gmail.com>
	<20080204190812.1E56D18585CF@rubyforge.org>
Message-ID: <fb283d5e0802041125r39982b7am18985df25c032c2f@mail.gmail.com>

On Feb 5, 2008 12:37 AM, Scott Ward <scott at shefield.com> wrote:
> Hello,
>
> I have a few tasks that run every few minutes.  They definitely need their
> own worker.
>
> However, other tasks run once per day or even once per week.  Right now I am
> creating a new worker for each of these tasks, but I wonder if that's
> unnecessary for tasks that I know will never execute at the same time.  They
> don't need to run concurrently.
>
> Basically, I am wondering if I should combine a bunch of these periodic,
> non-overlapping tasks into one worker called "general_maintenance_worker" or
> something like that.  For each task, I could create a different method and
> schedule them as needed in backgroundrb.yml.
>
> That should work, but is it "right"?  What's the "best practice" is in this
> situation?


Yup, if your tasks are non-overlapping then you can put them in the
same worker and through backgroundrb.yml
you can configure them to run at different intervals.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From leavengood at gmail.com  Mon Feb  4 16:54:03 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Mon, 4 Feb 2008 16:54:03 -0500
Subject: [Backgroundrb-devel] Best way to organize workers
In-Reply-To: <20080204190812.1E56D18585CF@rubyforge.org>
References: <5D74BD7C-492F-40EA-AA8A-F3EB1AF13E59@gmail.com>
	<20080204190812.1E56D18585CF@rubyforge.org>
Message-ID: <a4855ac50802041354v27caf34ahd6f9cc878960c0a6@mail.gmail.com>

On Feb 4, 2008 2:07 PM, Scott Ward <scott at shefield.com> wrote:
>
> Basically, I am wondering if I should combine a bunch of these periodic,
> non-overlapping tasks into one worker called "general_maintenance_worker" or
> something like that.  For each task, I could create a different method and
> schedule them as needed in backgroundrb.yml.
>
> That should work, but is it "right"?  What's the "best practice" is in this
> situation?

I am planning on doing the exact same thing for my periodic tasks. I
think it makes sense so let's make it into a BackgrounDRb best
practice :)

The thing to keep in mind is each worker gets a Ruby process, which
can consume 20-40 MB of memory. So if you can combine workers it will
definitely save resources. There really isn't a good reason I can
think of to have a bunch of workers sitting around just to be run one
or twice a day. So might as well put them all into one worker.
Especially if you can schedule tasks so they never overlap.

The way to keep this from getting too messy (as in a worker with tons
of unrelated code) is to extract the real work into separate ruby
modules which are required in the worker and then called from the
worker methods. Well that is how I have written things :)

Ryan

From heisters at greenriver.org  Mon Feb  4 20:27:02 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Mon, 4 Feb 2008 17:27:02 -0800
Subject: [Backgroundrb-devel] bdrb w/ functional tests?
Message-ID: <de819a5b0802041727w733c219ctb29a438b63b2c35f@mail.gmail.com>

Hi,

I was running into errors trying to run functional tests on
controllers that use MiddleMan, since my test environment doesn't have
a bdrb server running.

BackgrounDRb::BdrbConnError: Not able to connect
    vendor/plugins/backgroundrb/lib/backgroundrb.rb:39:in `ask_work'

The relevant lines of test/test_helper.rb looks like this:

  require File.expand_path(File.dirname(__FILE__) + "/bdrb_test_helper")
  require *Dir.glob(RAILS_ROOT + '/lib/workers/*.rb')

(I had to remove the "require 'test_helper'" line in bdrb_test_helper
for that to work).

Since I only use #ask_work, I stubbed this out in the bdrb_test_helper:

module BackgrounDRb
  # Added WorkerProxy stub so that functional tests will run
  class WorkerProxy
    def ask_work(arg)
      return true
    end
  end
end

which seems to work for the moment. It'd be nice if this stuff Just
Worked, though.

-Ian

From noah at rapouts.com  Mon Feb  4 20:30:11 2008
From: noah at rapouts.com (Noah Horton)
Date: Mon, 4 Feb 2008 17:30:11 -0800
Subject: [Backgroundrb-devel] 100meg logger worker?
Message-ID: <C3CCFC23.94D2%noah@rapouts.com>

Hey All,
  First off, thanks for everyone's work on backgroundrb.  My company just started using BackgroundRB for our site and I have a few questions that I could not find documented elsewhere.  Perhaps you can help me out with some info.

 1) I see two 'overhead' ruby processes; backgroundrb start and logger worker.  Both of these are big - 108 megs of memory for each.  Do I have something screwed up in my setup or is this normal?
 2) The old versions of BDRB seemed to have had (based on the docs) a vanilla ruby worker and a rails worker that one could extend from, whereas 1.0 has only metaworker.  Is there a way to get a vanilla ruby worker without loading my whole app?
 3) I seemed to have a memory leak the first time I deployed, where the processes kept growing in a stair-step pattern of 50 meg stairs.  My best hypothesis is that objects passed into ask_worker get kept in some datastructure that never goes away, and thus if you pass any objects there, things balloon; is that correct?  If not, do you have any thoughts on common 'gotchas' that could cause memory growth?

Thanks so much everyone.  I will try to blog the final solutions to the above to make this info available for others.

-Noah Horton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080204/802f4a71/attachment.html 

From heisters at greenriver.org  Tue Feb  5 00:12:35 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Mon, 4 Feb 2008 21:12:35 -0800
Subject: [Backgroundrb-devel] 100meg logger worker?
In-Reply-To: <C3CCFC23.94D2%noah@rapouts.com>
References: <C3CCFC23.94D2%noah@rapouts.com>
Message-ID: <de819a5b0802042112h3a45939fw62c3339dec598844@mail.gmail.com>

On 2/4/08, Noah Horton <noah at rapouts.com> wrote:
>   3) I seemed to have a memory leak the first time I deployed, where the
> processes kept growing in a stair-step pattern of 50 meg stairs.  My best
> hypothesis is that objects passed into ask_worker get kept in some
> datastructure that never goes away, and thus if you pass any objects there,
> things balloon; is that correct?  If not, do you have any thoughts on common
> 'gotchas' that could cause memory growth?

With an old version I found that I had to explicitly call #kill at the
end of the task, or the object would sit around forever. I don't think
there's an equivalent in the new version, or what version you're
running, so I've no idea if that will solve your question.

-Ian

From gethemant at gmail.com  Tue Feb  5 15:01:45 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 6 Feb 2008 01:31:45 +0530
Subject: [Backgroundrb-devel] bdrb w/ functional tests?
In-Reply-To: <de819a5b0802041727w733c219ctb29a438b63b2c35f@mail.gmail.com>
References: <de819a5b0802041727w733c219ctb29a438b63b2c35f@mail.gmail.com>
Message-ID: <fb283d5e0802051201p6cfecaepbe3c6df74edab874@mail.gmail.com>

On Feb 5, 2008 6:57 AM, Ian Smith-Heisters <heisters at greenriver.org> wrote:
> Hi,
>
> I was running into errors trying to run functional tests on
> controllers that use MiddleMan, since my test environment doesn't have
> a bdrb server running.
>
> BackgrounDRb::BdrbConnError: Not able to connect
>     vendor/plugins/backgroundrb/lib/backgroundrb.rb:39:in `ask_work'
>
> The relevant lines of test/test_helper.rb looks like this:
>
>   require File.expand_path(File.dirname(__FILE__) + "/bdrb_test_helper")
>   require *Dir.glob(RAILS_ROOT + '/lib/workers/*.rb')
>
> (I had to remove the "require 'test_helper'" line in bdrb_test_helper
> for that to work).
>
> Since I only use #ask_work, I stubbed this out in the bdrb_test_helper:
>
> module BackgrounDRb
>   # Added WorkerProxy stub so that functional tests will run
>   class WorkerProxy
>     def ask_work(arg)
>       return true
>     end
>   end
> end
>
> which seems to work for the moment. It'd be nice if this stuff Just
> Worked, though.

Cool, it will be commited.

From gethemant at gmail.com  Tue Feb  5 15:04:32 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 6 Feb 2008 01:34:32 +0530
Subject: [Backgroundrb-devel] 100meg logger worker?
In-Reply-To: <de819a5b0802042112h3a45939fw62c3339dec598844@mail.gmail.com>
References: <C3CCFC23.94D2%noah@rapouts.com>
	<de819a5b0802042112h3a45939fw62c3339dec598844@mail.gmail.com>
Message-ID: <fb283d5e0802051204m285e9a4bg6c7c982d40c749bd@mail.gmail.com>

On Feb 5, 2008 10:42 AM, Ian Smith-Heisters <heisters at greenriver.org> wrote:
> On 2/4/08, Noah Horton <noah at rapouts.com> wrote:
> >   3) I seemed to have a memory leak the first time I deployed, where the
> > processes kept growing in a stair-step pattern of 50 meg stairs.  My best
> > hypothesis is that objects passed into ask_worker get kept in some
> > datastructure that never goes away, and thus if you pass any objects there,
> > things balloon; is that correct?  If not, do you have any thoughts on common
> > 'gotchas' that could cause memory growth?

ask_work doesn't make objects stay around in a data structure.
However, you don't want to pass big objects anyways. Also, go light on
logging if possible.

>
> With an old version I found that I had to explicitly call #kill at the
> end of the task, or the object would sit around forever. I don't think
> there's an equivalent in the new version, or what version you're
> running, so I've no idea if that will solve your question.

New version has "exit". So, if you call exit from inside of your
worker, it will die.

From gfeil at realgirlsmedia.com  Wed Feb  6 19:39:08 2008
From: gfeil at realgirlsmedia.com (George Feil)
Date: Wed, 6 Feb 2008 16:39:08 -0800
Subject: [Backgroundrb-devel] configuration issues
Message-ID: <4BC9094459D5344490D03E5B38C3B89C017488E0@ex-be-005-sfo.shared.themessagecenter.com>

In the process of integrating BackgrounDRb into my company's CRM
application for our web site, I have run against two roadblocks in
configuration that required my actually modifying the BackgrounDRb
source code to work around:

 

1.	The MasterProxy class overrides the RAILS_ENV settings with the
settings in the backgroundrb.yml, or sets the environment to
"development" if not set. This makes it impossible to allow the server
to run in multiple configuration environments (e.g. staging and
production, for example). In my opinion, it would be best to have
MasterProxy rely solely on the current setting of RAILS_ENV, and never
override it under any circumstances.
2.	DebugMaster supports only logging options: log to files with
hard-coded names, or log to stdout. In our clustered production
environment, we use SyslogLogger to send all logging messages to a
remote server, in order to maintain logs for all our servers in one
place. There appears to be no way to do this in BackgrounDRb without
modifying the behavior of DebugMaster.

 

Apart from these two issues, this is a great tool for asynchronous
processing. Thanks!

 

 

George Feil | Senior Software Engineer

www.realgirlsmedia.com

www.divinecaroline.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080206/845e3fcc/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3238 bytes
Desc: image001.jpg
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080206/845e3fcc/attachment.jpe 

From gethemant at gmail.com  Wed Feb  6 21:51:45 2008
From: gethemant at gmail.com (hemant kumar)
Date: Thu, 07 Feb 2008 08:21:45 +0530
Subject: [Backgroundrb-devel] configuration issues
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017488E0@ex-be-005-sfo.shared.themessagecenter.com>
References: <4BC9094459D5344490D03E5B38C3B89C017488E0@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <1202352705.28594.6.camel@shire>

Hi George,

On Wed, 2008-02-06 at 16:39 -0800, George Feil wrote:
> In the process of integrating BackgrounDRb into my company?s CRM
> application for our web site, I have run against two roadblocks in
> configuration that required my actually modifying the BackgrounDRb
> source code to work around:
> 
>  
> 
>      1. The MasterProxy class overrides the RAILS_ENV settings with
>         the settings in the backgroundrb.yml, or sets the environment
>         to ?development? if not set. This makes it impossible to allow
>         the server to run in multiple configuration environments (e.g.
>         staging and production, for example). In my opinion, it would
>         be best to have MasterProxy rely solely on the current setting
>         of RAILS_ENV, and never override it under any circumstances.
>      2. DebugMaster supports only logging options: log to files with
>         hard-coded names, or log to stdout. In our clustered
>         production environment, we use SyslogLogger to send all
>         logging messages to a remote server, in order to maintain logs
>         for all our servers in one place. There appears to be no way
>         to do this in BackgrounDRb without modifying the behavior of
>         DebugMaster.
> 
>  
> 
> Apart from these two issues, this is a great tool for asynchronous
> processing. Thanks!
> 

It will be interesting to see your patches.



From heisters at greenriver.org  Wed Feb  6 22:02:29 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Wed, 6 Feb 2008 19:02:29 -0800
Subject: [Backgroundrb-devel] configuration issues
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017488E0@ex-be-005-sfo.shared.themessagecenter.com>
References: <4BC9094459D5344490D03E5B38C3B89C017488E0@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <de819a5b0802061902wc646eb5i7653df0c7c79aa7b@mail.gmail.com>

Hah! I've been hacking around this for hours today.

My old solution was something like this in config/backgroundrb.yml:
=============
<%
  ports = {
    'development' => 2000,
    'production' => 2001,
    'demo' => 2002,
    'sandbox' => 2003
  }
%>
---
:backgroundrb:
  :port: <%= ports[ENV['RAILS_ENV']] %>
  :environment: <%= ENV['RAILS_ENV'] %>
  :ip: 127.0.0.1
=============

But that failed when I started starting bdrb with capistrano because I
couldn't set RAILS_ENV (have to run bdrb sudo -u nobody). So my solution was
to create config files for each environment that are symlinked into the new
cap-created directory.

I'd agree that setting RAILS_ENV is unoptimal. It does really unexpected
things if you're relying on RAILS_ENV after starting bdrb.

On 2/6/08, George Feil <gfeil at realgirlsmedia.com> wrote:
>
>  In the process of integrating BackgrounDRb into my company's CRM
> application for our web site, I have run against two roadblocks in
> configuration that required my actually modifying the BackgrounDRb source
> code to work around:
>
>
>
>    1. The MasterProxy class overrides the RAILS_ENV settings with the
>    settings in the backgroundrb.yml, or sets the environment to
>    "development" if not set. This makes it impossible to allow the server to
>    run in multiple configuration environments (e.g. staging and
>    production, for example). In my opinion, it would be best to have
>    MasterProxy rely solely on the current setting of RAILS_ENV, and never
>    override it under any circumstances.
>    2. DebugMaster supports only logging options: log to files with
>    hard-coded names, or log to stdout. In our clustered production environment,
>    we use SyslogLogger to send all logging messages to a remote server, in
>    order to maintain logs for all our servers in one place. There appears to be
>    no way to do this in BackgrounDRb without modifying the behavior of
>    DebugMaster.
>
>
>
> Apart from these two issues, this is a great tool for asynchronous
> processing. Thanks!
>
>
>
> George Feil | Senior Software Engineer
>
> www.realgirlsmedia.com
>
> www.divinecaroline.com
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080206/5011aaab/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3238 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080206/5011aaab/attachment.jpe 

From thomas.a.wood at uconn.edu  Thu Feb  7 12:25:19 2008
From: thomas.a.wood at uconn.edu (Tom Wood)
Date: Thu, 7 Feb 2008 12:25:19 -0500
Subject: [Backgroundrb-devel] followup on another cron scheduling issue
Message-ID: <8C81AA7D3B12F4408C6B3359AEB001CC041388F4@LIB-EMarks.library.lib.uconn.edu>

About a week ago I mailed a question to the list about a cron scheduling
problem (worker running on an non-scheduled day) ... I have little bit
more information on this problem (but no patch, sorry! :-)

 

Am running 1.0.1 + Meng Kuan's recent patch to cron_trigger.rb.

 

I have a worker scheduled ala:

 

  :debugger_worker:

    :ping:

      :trigger_args: 0 0/5 09-17 * * 1,3,5 *

 

(in other words, run every 5 minutes between 9a - 5p on Mon/Wed/Fri.)

 

Here's what happened:

 

Correct: The worker ran correctly every 5 minutes on Friday, 9a - 5p.

Incorrect: It ran once on Saturday at 9a, but did not run again that
day.

Correct: It did not run on Sunday.

Correct: It ran every 5 minutes on Monday, 9a - 5p.

Incorrect: It ran once on Tuesday at 9a, and did not yet run again that
day.

Correct: Ran every 5 minutes on Wednesday, 9a - 5p

Incorrect: Ran once on Thursday at 9a, and did not run again that day.

 

Seems like the scheduler schedules one errant run on the first "off day"
after an "on day."

 

-Tom

 

Tom Wood

thomas.a.wood at uconn.edu <mailto:thomas.a.wood at uconn.edu> 

ITS Applications Developer

University of Connecticut Libraries

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080207/7dea1fb4/attachment.html 

From gethemant at gmail.com  Thu Feb  7 17:29:36 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 8 Feb 2008 03:59:36 +0530
Subject: [Backgroundrb-devel] followup on another cron scheduling issue
In-Reply-To: <8C81AA7D3B12F4408C6B3359AEB001CC041388F4@LIB-EMarks.library.lib.uconn.edu>
References: <8C81AA7D3B12F4408C6B3359AEB001CC041388F4@LIB-EMarks.library.lib.uconn.edu>
Message-ID: <fb283d5e0802071429r3ef2eae0je9f07d245c2119db@mail.gmail.com>

On Feb 7, 2008 10:55 PM, Tom Wood <thomas.a.wood at uconn.edu> wrote:
>
>
>
>
> About a week ago I mailed a question to the list about a cron scheduling
> problem (worker running on an non-scheduled day) ? I have little bit more
> information on this problem (but no patch, sorry! :-)
>
>
>
> Am running 1.0.1 + Meng Kuan's recent patch to cron_trigger.rb.
>
>
>
> I have a worker scheduled ala:
>
>
>
>   :debugger_worker:
>
>     :ping:
>
>       :trigger_args: 0 0/5 09-17 * * 1,3,5 *
>
>
>
> (in other words, run every 5 minutes between 9a ? 5p on Mon/Wed/Fri.)
>
>
>
> Here's what happened:
>
>
>
> Correct: The worker ran correctly every 5 minutes on Friday, 9a ? 5p.
>
> Incorrect: It ran once on Saturday at 9a, but did not run again that day.
>
> Correct: It did not run on Sunday.
>
> Correct: It ran every 5 minutes on Monday, 9a ? 5p.
>
> Incorrect: It ran once on Tuesday at 9a, and did not yet run again that day.
>
> Correct: Ran every 5 minutes on Wednesday, 9a ? 5p
>
> Incorrect: Ran once on Thursday at 9a, and did not run again that day.
>
>
>

Well, Cron Scheduler bugs are getting fixed. I am writing exhaustive
specs for cron trigger. Check out this pastie:

http://pastie.caboo.se/148945

From cosmin.radoi at gmail.com  Sun Feb 10 07:54:40 2008
From: cosmin.radoi at gmail.com (Cosmin Radoi)
Date: Sun, 10 Feb 2008 14:54:40 +0200
Subject: [Backgroundrb-devel] What happens if?
Message-ID: <d1497f5d0802100454h1f54987bha57779df0e9f6b68@mail.gmail.com>

Hello,

I'm new to the list so I'd like to first greet everybody and thank Ezra for
the nice solution to an old problem.

I also have a warm-up newbie question. What happens if I send a request to a
worker that in that moment is working on some other talk? Does it wait for
the worker to finish his current method and than evaluate my request, does
it interrupt the worker to handle the request or does it simply spawn
another thread for it?
Thanks.

Cosmin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080210/af843b9a/attachment-0001.html 

From gethemant at gmail.com  Sun Feb 10 09:29:34 2008
From: gethemant at gmail.com (hemant kumar)
Date: Sun, 10 Feb 2008 19:59:34 +0530
Subject: [Backgroundrb-devel] What happens if?
In-Reply-To: <d1497f5d0802100454h1f54987bha57779df0e9f6b68@mail.gmail.com>
References: <d1497f5d0802100454h1f54987bha57779df0e9f6b68@mail.gmail.com>
Message-ID: <1202653774.22147.14.camel@shire>


Hi
On Sun, 2008-02-10 at 14:54 +0200, Cosmin Radoi wrote:
> 
> Hello,
> 
> I'm new to the list so I'd like to first greet everybody and thank
> Ezra for the nice solution to an old problem.
> 
> I also have a warm-up newbie question. What happens if I send a
> request to a worker that in that moment is working on some other talk?
> Does it wait for the worker to finish his current method and than
> evaluate my request, does it interrupt the worker to handle the
> request or does it simply spawn another thread for it?
> Thanks.
> 

If you invoke ask_work on a worker thats already processing a task( in a
totally blocking manner ), then your request will be queued in master
process send_data buffer, until your worker is free and ready to remove
the data ready at the socket and start executing next task.

Since bdrb makes use of non blocking IO and select call to check for
availability of data at the socket, assuming your worker is doing some
processing because of which reactor loop is not able to run through next
iteration ( in each iteration it checks if there is data to be read from
the socket ), your request will be queued up in socket buffer. For
couple of requests this fine, but if you are making too many requests
and your requests are getting queued in socket buffer, its not a good.

However, if you are processing too many tasks and want to spawn a thread
for them, you can rather use inbuilt, "thread_pool" to defer execution
of tasks to a pool of threads. It works nicely and will probably work
better than roll your own thread solution.

For more information refer the README file or documentation here at:

http://backgroundrb.rubyforge.org

Finally in no case, a running tasks gets interrupted.





From sdsykes at gmail.com  Sun Feb 10 11:04:43 2008
From: sdsykes at gmail.com (Stephen Sykes)
Date: Sun, 10 Feb 2008 18:04:43 +0200
Subject: [Backgroundrb-devel] cron scheduling problem
In-Reply-To: <ce7b5df30802081424m2b093be4w9d2c68707d9dc2d2@mail.gmail.com>
References: <ce7b5df30802081424m2b093be4w9d2c68707d9dc2d2@mail.gmail.com>
Message-ID: <ce7b5df30802100804r4aef8958i3911b4c5a0e43768@mail.gmail.com>

> >It appears that cron_trigger incremented the day value beyond the
> >number of days of the current month (from 31 to 32) and this caused
> >the Time.local call to abort.

By the way I also provided a patch for this, here
http://backgroundrb.devjavu.com/ticket/78 which you may like to check
out.  It is slightly different from Meng's

BTW - I guess no-one looks at trac?

 - Stephen

From apsoto at gmail.com  Sun Feb 10 15:59:33 2008
From: apsoto at gmail.com (Alex Soto)
Date: Sun, 10 Feb 2008 12:59:33 -0800
Subject: [Backgroundrb-devel] What happens if?
In-Reply-To: <1202653774.22147.14.camel@shire>
References: <d1497f5d0802100454h1f54987bha57779df0e9f6b68@mail.gmail.com>
	<1202653774.22147.14.camel@shire>
Message-ID: <BFA0553C-3AF6-4419-BF3B-B2F1FF65E298@gmail.com>

Am I understanding correctly that  a task that is doing work would get  
interrupted and it's current job is effectively canceled?

Alex


On Feb 10, 2008, at 6:29 AM, hemant kumar wrote:

>
> Hi
> On Sun, 2008-02-10 at 14:54 +0200, Cosmin Radoi wrote:
>>
>> Hello,
>>
>> I'm new to the list so I'd like to first greet everybody and thank
>> Ezra for the nice solution to an old problem.
>>
>> I also have a warm-up newbie question. What happens if I send a
>> request to a worker that in that moment is working on some other  
>> talk?
>> Does it wait for the worker to finish his current method and than
>> evaluate my request, does it interrupt the worker to handle the
>> request or does it simply spawn another thread for it?
>> Thanks.
>>
>
> If you invoke ask_work on a worker thats already processing a  
> task( in a
> totally blocking manner ), then your request will be queued in master
> process send_data buffer, until your worker is free and ready to  
> remove
> the data ready at the socket and start executing next task.
>
> Since bdrb makes use of non blocking IO and select call to check for
> availability of data at the socket, assuming your worker is doing some
> processing because of which reactor loop is not able to run through  
> next
> iteration ( in each iteration it checks if there is data to be read  
> from
> the socket ), your request will be queued up in socket buffer. For
> couple of requests this fine, but if you are making too many requests
> and your requests are getting queued in socket buffer, its not a good.
>
> However, if you are processing too many tasks and want to spawn a  
> thread
> for them, you can rather use inbuilt, "thread_pool" to defer execution
> of tasks to a pool of threads. It works nicely and will probably work
> better than roll your own thread solution.
>
> For more information refer the README file or documentation here at:
>
> http://backgroundrb.rubyforge.org
>
> Finally in no case, a running tasks gets interrupted.
>
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From leavengood at gmail.com  Sun Feb 10 16:11:01 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Sun, 10 Feb 2008 16:11:01 -0500
Subject: [Backgroundrb-devel] What happens if?
In-Reply-To: <BFA0553C-3AF6-4419-BF3B-B2F1FF65E298@gmail.com>
References: <d1497f5d0802100454h1f54987bha57779df0e9f6b68@mail.gmail.com>
	<1202653774.22147.14.camel@shire>
	<BFA0553C-3AF6-4419-BF3B-B2F1FF65E298@gmail.com>
Message-ID: <a4855ac50802101311tc1bd277wf4d01bf2652caf2a@mail.gmail.com>

On Feb 10, 2008 3:59 PM, Alex Soto <apsoto at gmail.com> wrote:
> Am I understanding correctly that  a task that is doing work would get
> interrupted and it's current job is effectively canceled?

No, as Hemant said at the end of his email, no running job will ever
be interrupted. That would pretty much defeat the purpose of
BackgrounDRb.

Ryan

From lists at sourceillustrated.com  Mon Feb 11 01:11:01 2008
From: lists at sourceillustrated.com (John Wells)
Date: Mon, 11 Feb 2008 01:11:01 -0500
Subject: [Backgroundrb-devel] Problems running backgroundrb
Message-ID: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>

Hi guys,

New to backgroundrb, and I like it very much conceptually, but I'm
struggling to get it to run successfully. I started with trunk, but
was getting a very strange gem require error when starting a worker
(though it wasn't telling me which gem was missing)....see here:
http://pastie.caboo.se/150225

So I rolled back to the release 1.0.1 tag but had different problems
there. I have my own "acts_as"  plugin that's required in my
environment.rb....it's stored in lib/formatters.rb. However,
backgroundrb complains about it not being there in startup:

$ script/backgroundrb start

/home/jb/workspace/ruby/eqnow_dev/vendor/rails/activerecord/lib/active_record/base.rb:1238:in
`method_missing': undefined method `acts_as_phone_formatter' for
Event:Class (NoMethodError)
        from /home/jb/workspace/ruby/eqnow_dev/app/models/event.rb:45
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require'
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:495:in
`require'
        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:342:in
`new_constants_in'

It's definitely there in my config/environment.rb:

require 'formatters'

My backgroundrb.yml looks like:

---
:backgroundrb:
  :ip: localhost
  :port: 11006
  :environment: development

I'm completely at a loss on how to proceed. I'd really like to use
it... (I *need* it) but not sure how to get it running properly.

Can anyone help?

Thanks for any assistance you might provide!

John

From apsoto at gmail.com  Mon Feb 11 03:55:33 2008
From: apsoto at gmail.com (Alex Soto)
Date: Mon, 11 Feb 2008 00:55:33 -0800
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
Message-ID: <12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>

does require 'lib/formatters' work?


On Feb 10, 2008, at 10:11 PM, John Wells wrote:

> Hi guys,
>
> New to backgroundrb, and I like it very much conceptually, but I'm
> struggling to get it to run successfully. I started with trunk, but
> was getting a very strange gem require error when starting a worker
> (though it wasn't telling me which gem was missing)....see here:
> http://pastie.caboo.se/150225
>
> So I rolled back to the release 1.0.1 tag but had different problems
> there. I have my own "acts_as"  plugin that's required in my
> environment.rb....it's stored in lib/formatters.rb. However,
> backgroundrb complains about it not being there in startup:
>
> $ script/backgroundrb start
>
> /home/jb/workspace/ruby/eqnow_dev/vendor/rails/activerecord/lib/ 
> active_record/base.rb:1238:in
> `method_missing': undefined method `acts_as_phone_formatter' for
> Event:Class (NoMethodError)
>        from /home/jb/workspace/ruby/eqnow_dev/app/models/event.rb:45
>        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb: 
> 27:in
> `gem_original_require'
>        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb: 
> 27:in
> `require'
>        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/ 
> activerecord/lib/../../activesupport/lib/active_support/ 
> dependencies.rb:495:in
> `require'
>        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/ 
> activerecord/lib/../../activesupport/lib/active_support/ 
> dependencies.rb:342:in
> `new_constants_in'
>
> It's definitely there in my config/environment.rb:
>
> require 'formatters'
>
> My backgroundrb.yml looks like:
>
> ---
> :backgroundrb:
>  :ip: localhost
>  :port: 11006
>  :environment: development
>
> I'm completely at a loss on how to proceed. I'd really like to use
> it... (I *need* it) but not sure how to get it running properly.
>
> Can anyone help?
>
> Thanks for any assistance you might provide!
>
> John
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From gethemant at gmail.com  Mon Feb 11 05:03:35 2008
From: gethemant at gmail.com (hemant kumar)
Date: Mon, 11 Feb 2008 15:33:35 +0530
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
Message-ID: <1202724215.6982.8.camel@shire>


On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
> does require 'lib/formatters' work?
> 
> 
> On Feb 10, 2008, at 10:11 PM, John Wells wrote:
> 
> > Hi guys,
> >
> > New to backgroundrb, and I like it very much conceptually, but I'm
> > struggling to get it to run successfully. I started with trunk, but
> > was getting a very strange gem require error when starting a worker
> > (though it wasn't telling me which gem was missing)....see here:
> > http://pastie.caboo.se/150225

Its not complaining about any missing gems, rather its not able to load
workers defined inside lib/workers directory. You must define at least
one worker, before you can start bdrb. In case, you have already defined
a worker, Can we have a look at the code of your worker?

> >
> > So I rolled back to the release 1.0.1 tag but had different problems
> > there. I have my own "acts_as"  plugin that's required in my
> > environment.rb....it's stored in lib/formatters.rb. However,
> > backgroundrb complains about it not being there in startup:
> >
> > $ script/backgroundrb start
> >
> > /home/jb/workspace/ruby/eqnow_dev/vendor/rails/activerecord/lib/ 
> > active_record/base.rb:1238:in
> > `method_missing': undefined method `acts_as_phone_formatter' for
> > Event:Class (NoMethodError)
> >        from /home/jb/workspace/ruby/eqnow_dev/app/models/event.rb:45
> >        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb: 
> > 27:in
> > `gem_original_require'
> >        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb: 
> > 27:in
> > `require'
> >        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/ 
> > activerecord/lib/../../activesupport/lib/active_support/ 
> > dependencies.rb:495:in
> > `require'
> >        from /home/jb/workspace/ruby/eqnow_dev/vendor/rails/ 
> > activerecord/lib/../../activesupport/lib/active_support/ 
> > dependencies.rb:342:in
> > `new_constants_in'

Well, looks like you have a model Event defined in your app. 1.0.1
release tag has a bug where require statements in bdrb aren't relative
and since bdrb comes with its own Event class, its conflicting.

This has been fixed in trunk version of plugin and all the requires are
relative, hence, revert back to trunk and create few workers and see if
it works. Also, try creating a new rails project and try using bdrb
there, it will give you some further information.




> >
> > It's definitely there in my config/environment.rb:
> >
> > require 'formatters'
> >
> > My backgroundrb.yml looks like:
> >
> > ---
> > :backgroundrb:
> >  :ip: localhost
> >  :port: 11006
> >  :environment: development

That environment string is redundant since, by default bdrb will load
development environment anyways.

> >
> > I'm completely at a loss on how to proceed. I'd really like to use
> > it... (I *need* it) but not sure how to get it running properly.
> >
> > Can anyone help?
> >
> > Thanks for any assistance you might provide!
> >




From lists at sourceillustrated.com  Mon Feb 11 09:50:30 2008
From: lists at sourceillustrated.com (John Wells)
Date: Mon, 11 Feb 2008 09:50:30 -0500
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <1202724215.6982.8.camel@shire>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
Message-ID: <44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>

On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
>
> On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
> > does require 'lib/formatters' work?
> >
> >
> > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
> >
> > > Hi guys,
> > >
> > > New to backgroundrb, and I like it very much conceptually, but I'm
> > > struggling to get it to run successfully. I started with trunk, but
> > > was getting a very strange gem require error when starting a worker
> > > (though it wasn't telling me which gem was missing)....see here:
> > > http://pastie.caboo.se/150225
>
> Its not complaining about any missing gems, rather its not able to load
> workers defined inside lib/workers directory. You must define at least
> one worker, before you can start bdrb. In case, you have already defined
> a worker, Can we have a look at the code of your worker?

Thanks for your help. Sure thing...the code is below. However, I'm
very curious...how did you determine this was the problem? Just so I
know what to look for in the future.

===== controller that instantiates worker =====
class Admin::BatchController < Admin::ApplicationController

  def initialize
    super
    @formats = GW::BatchListing::ListingRow.formats
    @part_of = :admin
  end

  def get_progress
    if request.xhr?
      progress_percent = MiddleMan.get_worker(session[:job_key]).progress
      render :update do |page|
        page.call('progressPercent', 'progressbar', progress_percent)
        page.redirect_to( :action => 'done')   if progress_percent >= 100
      end
    else
      redirect_to :action => 'index'
    end
  end

  def done
    errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
    if errors.size > 0
      flash[:notice] = errors.join("<br/>")
    else
      flash[:notice] = "Successful load."
    end
    MiddleMan.delete_worker(session[:job_key])
  end

  def add
    @user = User.find(params[:id])
    @cats = Category.find(:all, :conditions=>"parent_id is null")
    if request.post?
      listings = params[:listings]
      content_type = listings.content_type.chomp
      if "application/zip"!=content_type and
         "application/x-zip-compressed"!=content_type
        flash[:notice] =
          "Only files of type application/zip " +
          "and application/x-zip-compressed " +
          "can be used. You uploaded #{content_type}."
      else
        # write out to tmp file
        tmpfile = "/tmp/#{rand 500000}.zip"
        open(tmpfile, "w") do |f|
          f << listings.read
        end
        parser =
          GW::BatchListing::ListingParser.new(params[:format],
                                                        params[:listing_type],
                                                        @user)
        session[:job_key] =
            MiddleMan.new_worker(:class => :foo_worker,
                                 :args => parser)
      end
    end
  end
end

===== and the worker itself =====
class BatchImportWorker < BackgrounDRb::MetaWorker
  attr_reader :progress
  attr_reader :errors
  set_worker_name :batch_import_worker

  def create(args = nil)
    # this method is called, when worker is loaded for the first time
  end

  def do_work(parser)
    @progress = 0
    @errors = parser.process(tmpfile)
  end
end

From gethemant at gmail.com  Mon Feb 11 13:28:35 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 11 Feb 2008 23:58:35 +0530
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
	<44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
Message-ID: <fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>

On Mon, Feb 11, 2008 at 8:20 PM, John Wells <lists at sourceillustrated.com> wrote:
> On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
>  >
>  > On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
>  > > does require 'lib/formatters' work?
>  > >
>  > >
>  > > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
>  > >
>  > > > Hi guys,
>  > > >
>  > > > New to backgroundrb, and I like it very much conceptually, but I'm
>  > > > struggling to get it to run successfully. I started with trunk, but
>  > > > was getting a very strange gem require error when starting a worker
>  > > > (though it wasn't telling me which gem was missing)....see here:
>  > > > http://pastie.caboo.se/150225
>  >
>  > Its not complaining about any missing gems, rather its not able to load
>  > workers defined inside lib/workers directory. You must define at least
>  > one worker, before you can start bdrb. In case, you have already defined
>  > a worker, Can we have a look at the code of your worker?
>
>  Thanks for your help. Sure thing...the code is below. However, I'm
>  very curious...how did you determine this was the problem? Just so I
>  know what to look for in the future.

Because, when I saw error at:

http://pastie.caboo.se/150225

It clearly says thats start_worker is failing on 114 line. Generally
this happens, when bdrb is not able to load a particular worker file.
Also, you see empty string "gem_original_require': no such file to
load --  (LoadError)" because bdrb is not able to get name of worker
file to load correctly.


>
>  ===== controller that instantiates worker =====
>  class Admin::BatchController < Admin::ApplicationController
>
>   def initialize
>     super
>     @formats = GW::BatchListing::ListingRow.formats
>     @part_of = :admin
>   end
>
>   def get_progress
>     if request.xhr?
>       progress_percent = MiddleMan.get_worker(session[:job_key]).progress
>       render :update do |page|
>         page.call('progressPercent', 'progressbar', progress_percent)
>         page.redirect_to( :action => 'done')   if progress_percent >= 100
>       end
>     else
>       redirect_to :action => 'index'
>     end
>   end
>
>   def done
>     errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
>     if errors.size > 0
>       flash[:notice] = errors.join("<br/>")
>     else
>       flash[:notice] = "Successful load."
>     end
>     MiddleMan.delete_worker(session[:job_key])
>   end
>
>   def add
>     @user = User.find(params[:id])
>     @cats = Category.find(:all, :conditions=>"parent_id is null")
>     if request.post?
>       listings = params[:listings]
>       content_type = listings.content_type.chomp
>       if "application/zip"!=content_type and
>          "application/x-zip-compressed"!=content_type
>         flash[:notice] =
>           "Only files of type application/zip " +
>           "and application/x-zip-compressed " +
>           "can be used. You uploaded #{content_type}."
>       else
>         # write out to tmp file
>         tmpfile = "/tmp/#{rand 500000}.zip"
>         open(tmpfile, "w") do |f|
>           f << listings.read
>         end
>         parser =
>           GW::BatchListing::ListingParser.new(params[:format],
>                                                         params[:listing_type],
>                                                         @user)
>         session[:job_key] =
>             MiddleMan.new_worker(:class => :foo_worker,
>                                  :args => parser)
>       end
>     end
>   end
>  end
>

There are couple of problems with above code, for example,
MiddleMan.get_worker is not defined. To get an idea about, which
methods are defined for MiddleMan, have a look at,

http://backgroundrb.rubyforge.org/classes/BackgrounDRb/WorkerProxy.html

Also, you should read README file that comes with plugin.

>  ===== and the worker itself =====
>  class BatchImportWorker < BackgrounDRb::MetaWorker
>   attr_reader :progress
>   attr_reader :errors
>   set_worker_name :batch_import_worker
>
>   def create(args = nil)
>     # this method is called, when worker is loaded for the first time
>   end
>
>   def do_work(parser)
>     @progress = 0
>     @errors = parser.process(tmpfile)
>   end
>  end

Well, I find  no problems with above worker code, and it should load.
Create a fresh rails app and see if bdrb works there.




-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From lists at sourceillustrated.com  Mon Feb 11 13:35:20 2008
From: lists at sourceillustrated.com (John Wells)
Date: Mon, 11 Feb 2008 13:35:20 -0500
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
	<44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
	<fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>
Message-ID: <44dddf400802111035h7f2fb34ajeda347a35235309c@mail.gmail.com>

On Feb 11, 2008 1:28 PM, hemant <gethemant at gmail.com> wrote:
> On Mon, Feb 11, 2008 at 8:20 PM, John Wells <lists at sourceillustrated.com> wrote:
> > On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
> >  >
> >  > On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
> >  > > does require 'lib/formatters' work?
> >  > >
> >  > >
> >  > > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
> >  > >
> >  > > > Hi guys,
> >  > > >
> >  > > > New to backgroundrb, and I like it very much conceptually, but I'm
> >  > > > struggling to get it to run successfully. I started with trunk, but
> >  > > > was getting a very strange gem require error when starting a worker
> >  > > > (though it wasn't telling me which gem was missing)....see here:
> >  > > > http://pastie.caboo.se/150225
> >  >
> >  > Its not complaining about any missing gems, rather its not able to load
> >  > workers defined inside lib/workers directory. You must define at least
> >  > one worker, before you can start bdrb. In case, you have already defined
> >  > a worker, Can we have a look at the code of your worker?
> >
> >  Thanks for your help. Sure thing...the code is below. However, I'm
> >  very curious...how did you determine this was the problem? Just so I
> >  know what to look for in the future.
>
> Because, when I saw error at:
>
> http://pastie.caboo.se/150225
>
> It clearly says thats start_worker is failing on 114 line. Generally
> this happens, when bdrb is not able to load a particular worker file.
> Also, you see empty string "gem_original_require': no such file to
> load --  (LoadError)" because bdrb is not able to get name of worker
> file to load correctly.
>
>
>
> >
> >  ===== controller that instantiates worker =====
> >  class Admin::BatchController < Admin::ApplicationController
> >
> >   def initialize
> >     super
> >     @formats = GW::BatchListing::ListingRow.formats
> >     @part_of = :admin
> >   end
> >
> >   def get_progress
> >     if request.xhr?
> >       progress_percent = MiddleMan.get_worker(session[:job_key]).progress
> >       render :update do |page|
> >         page.call('progressPercent', 'progressbar', progress_percent)
> >         page.redirect_to( :action => 'done')   if progress_percent >= 100
> >       end
> >     else
> >       redirect_to :action => 'index'
> >     end
> >   end
> >
> >   def done
> >     errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
> >     if errors.size > 0
> >       flash[:notice] = errors.join("<br/>")
> >     else
> >       flash[:notice] = "Successful load."
> >     end
> >     MiddleMan.delete_worker(session[:job_key])
> >   end
> >
> >   def add
> >     @user = User.find(params[:id])
> >     @cats = Category.find(:all, :conditions=>"parent_id is null")
> >     if request.post?
> >       listings = params[:listings]
> >       content_type = listings.content_type.chomp
> >       if "application/zip"!=content_type and
> >          "application/x-zip-compressed"!=content_type
> >         flash[:notice] =
> >           "Only files of type application/zip " +
> >           "and application/x-zip-compressed " +
> >           "can be used. You uploaded #{content_type}."
> >       else
> >         # write out to tmp file
> >         tmpfile = "/tmp/#{rand 500000}.zip"
> >         open(tmpfile, "w") do |f|
> >           f << listings.read
> >         end
> >         parser =
> >           GW::BatchListing::ListingParser.new(params[:format],
> >                                                         params[:listing_type],
> >                                                         @user)
> >         session[:job_key] =
> >             MiddleMan.new_worker(:class => :foo_worker,
> >                                  :args => parser)
> >       end
> >     end
> >   end
> >  end
> >
>
> There are couple of problems with above code, for example,
> MiddleMan.get_worker is not defined. To get an idea about, which
> methods are defined for MiddleMan, have a look at,
>
> http://backgroundrb.rubyforge.org/classes/BackgrounDRb/WorkerProxy.html
>
> Also, you should read README file that comes with plugin.
>
> >  ===== and the worker itself =====
> >  class BatchImportWorker < BackgrounDRb::MetaWorker
> >   attr_reader :progress
> >   attr_reader :errors
> >   set_worker_name :batch_import_worker
> >
> >   def create(args = nil)
> >     # this method is called, when worker is loaded for the first time
> >   end
> >
> >   def do_work(parser)
> >     @progress = 0
> >     @errors = parser.process(tmpfile)
> >   end
> >  end
>
> Well, I find  no problems with above worker code, and it should load.
> Create a fresh rails app and see if bdrb works there.

Ok, sorry... I was working from the old InfoQ article by Ezra and the
api docs. I probably made some bad assumptions there...I will take a
hard look and try it again.

Thanks for your help!

John

From gethemant at gmail.com  Mon Feb 11 13:41:33 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 12 Feb 2008 00:11:33 +0530
Subject: [Backgroundrb-devel] Updates and request for help
Message-ID: <fb283d5e0802111041s7e29335ck923e7548cd2bec6f@mail.gmail.com>

Hi Folks,

I am in process of moving bdrb code from devjavu to gitorious. New
development code is available here:

http://gitorious.org/projects/backgroundrb

I have done quite a bit of work on this. I have also removed framework
directory from plugin, since bdrb bundles packet gem within itself, it
was proving painful to maintain two separate versions of the same code
bases. So, bdrb has now added "packet" gem as a formal dependency. I
released new version of packet, which adds better support for writing
to socket and stuff.

New code is lot less and better organized and I am working towards
getting proper code coverage through test cases. Look into tests
directory.

Perhaps, git doesn't play so nicely with rails yet, so I want your
opinions on this. Dale Cook has worked on getting documentation and
new website done, which will be coming for new release.
I want your help and contribution. If you have submitted a patch to
bdrb in past and want to contribute. Please create an account with
gitorious and you will have commit access.

I am busy as ever with my day job and need your help. Also, users who
are running bdrb in production should keep using code at:

http://svn.devjavu.com/backgroundrb/trunk/






-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Mon Feb 11 22:27:03 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 12 Feb 2008 08:57:03 +0530
Subject: [Backgroundrb-devel] Updates and request for help
In-Reply-To: <009D7244-6B4A-4764-A27E-8AE9E32FF2F0@geoffgarside.co.uk>
References: <fb283d5e0802111041s7e29335ck923e7548cd2bec6f@mail.gmail.com>
	<009D7244-6B4A-4764-A27E-8AE9E32FF2F0@geoffgarside.co.uk>
Message-ID: <fb283d5e0802111927t5b56de5as5133e796ec5a615c@mail.gmail.com>

Hi

On Tue, Feb 12, 2008 at 4:15 AM, Geoff Garside <geoff at geoffgarside.co.uk> wrote:
>
>  Does gitorious have a subversion mirror service as well as straight
>
> git access?

No, I think.

>  Should we be using the deja vu subversion repository as the git mirror?
>
>  Thanks for all your incredible work Hemant.

Yup, thats what I thought, we will keep using devjavu svn repository
as a git mirror for end users but development will happen on git.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From rich.everhart at gmail.com  Tue Feb 12 17:06:07 2008
From: rich.everhart at gmail.com (Richard Everhart)
Date: Tue, 12 Feb 2008 14:06:07 -0800
Subject: [Backgroundrb-devel] Getting started
Message-ID: <a242781c0802121406o4800e3a1q187fb67a8bc3b711@mail.gmail.com>

I'm just getting started with Backgroundrb and I'm having a lot of
problems.  First of all, whenever I try to start or execute a worker I
get the following error:

BackgrounDRb::BdrbConnError (Not able to connect):
    .//vendor/plugins/backgroundrb/lib/backgroundrb.rb:47:in `new_worker'
    .//app/controllers/testdrb_controller.rb:13:in `do_fib'
    /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1095:in
`send'
[...snip...]

I'm using rails version 1.2.6 and ruby 1.8.6.  My backgroundrb.yaml
looks like this:
---
:backgroundrb:
  :port: 11006
  :ip: 0.0.0.0

I am starting the server for my app first and then running
'script/backgroundrb start' afterwards.  I'm just testing out
backgroundrb to get a feel of it so my app is pretty simple.  A number
is selected from a drop down and the user presses a 'go' button
causing the worker to calculate the selected number's fibonacci number
(just needed something long running).

Any help much appreciated.

Rich

From gethemant at gmail.com  Tue Feb 12 17:14:16 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 13 Feb 2008 03:44:16 +0530
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <a242781c0802121406o4800e3a1q187fb67a8bc3b711@mail.gmail.com>
References: <a242781c0802121406o4800e3a1q187fb67a8bc3b711@mail.gmail.com>
Message-ID: <fb283d5e0802121414o6c0863d4sf985cc732a72845c@mail.gmail.com>

On Wed, Feb 13, 2008 at 3:36 AM, Richard Everhart
<rich.everhart at gmail.com> wrote:
> I'm just getting started with Backgroundrb and I'm having a lot of
>  problems.  First of all, whenever I try to start or execute a worker I
>  get the following error:
>
>  BackgrounDRb::BdrbConnError (Not able to connect):
>     .//vendor/plugins/backgroundrb/lib/backgroundrb.rb:47:in `new_worker'
>     .//app/controllers/testdrb_controller.rb:13:in `do_fib'
>     /usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:1095:in
>  `send'
>  [...snip...]
>
>  I'm using rails version 1.2.6 and ruby 1.8.6.  My backgroundrb.yaml
>  looks like this:
>  ---
>  :backgroundrb:
>   :port: 11006
>   :ip: 0.0.0.0
>
>  I am starting the server for my app first and then running
>  'script/backgroundrb start' afterwards.  I'm just testing out
>  backgroundrb to get a feel of it so my app is pretty simple.  A number
>  is selected from a drop down and the user presses a 'go' button
>  causing the worker to calculate the selected number's fibonacci number
>  (just needed something long running).
>
>  Any help much appreciated.


You are running trunk version of library right? Also, probably bdrb
server is not starting up for you. So make sure, backgroundrb server
is running.
I have collected some best practices for getting up and running with bdrb.

http://gnufied.org/2008/02/12/backgroundrb-best-practises/

From rich.everhart at gmail.com  Tue Feb 12 20:49:16 2008
From: rich.everhart at gmail.com (Richard Everhart)
Date: Tue, 12 Feb 2008 17:49:16 -0800
Subject: [Backgroundrb-devel] Getting started
Message-ID: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>

I upgraded ruby right before I wrote my original message and that was
causing some problems.  That's fixed but I still get the 'Not able to
connect' error.  In backgroundrb_server.log there seem to really be
two errors:

/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load --  (LoadError)
       from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
       from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
`require'
       from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:342:in
`new_constants_in'
       from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
`require'
       from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:113:in
`start_worker'
[...snip...]
/home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:24:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
       from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/worker.rb:47:in
`handle_internal_messages'
       from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:154:in
`start_reactor'
[...snip...]

The first error seems to be due to a ruby and/or rails configuration
error.  However, I've seen this error before, and the '--' in the
error message is usually followed by the name of the file that can't
be loaded.

Thanks for your help!

Rich

From gethemant at gmail.com  Tue Feb 12 21:02:25 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 13 Feb 2008 07:32:25 +0530
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
Message-ID: <fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>

On Wed, Feb 13, 2008 at 7:19 AM, Richard Everhart
<rich.everhart at gmail.com> wrote:
>
> I upgraded ruby right before I wrote my original message and that was
>  causing some problems.  That's fixed but I still get the 'Not able to
>  connect' error.  In backgroundrb_server.log there seem to really be
>  two errors:
>
>  /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
>  `gem_original_require': no such file to load --  (LoadError)
>        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
>  `require'
>        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
>  `require'
>        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:342:in
>  `new_constants_in'
>        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
>  `require'
>        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:113:in
>  `start_worker'
>  [...snip...]
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:24:in
>  `read_data': Packet::DisconnectError (Packet::DisconnectError)
>        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/worker.rb:47:in
>  `handle_internal_messages'
>        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:154:in
>  `start_reactor'
>  [...snip...]
>
>  The first error seems to be due to a ruby and/or rails configuration
>  error.  However, I've seen this error before, and the '--' in the
>  error message is usually followed by the name of the file that can't
>  be loaded.
>

Did you define a worker before starting backgroundrb server? Above
error is because, bdrb is not able to load defined worker.
If you have the worker defined can we see the code of the worker.

From gethemant at gmail.com  Tue Feb 12 21:03:49 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 13 Feb 2008 07:33:49 +0530
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <44dddf400802111035h7f2fb34ajeda347a35235309c@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
	<44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
	<fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>
	<44dddf400802111035h7f2fb34ajeda347a35235309c@mail.gmail.com>
Message-ID: <fb283d5e0802121803i20d9804q1f9a18f7072dd14d@mail.gmail.com>

On Tue, Feb 12, 2008 at 12:05 AM, John Wells
<lists at sourceillustrated.com> wrote:
>
> On Feb 11, 2008 1:28 PM, hemant <gethemant at gmail.com> wrote:
>  > On Mon, Feb 11, 2008 at 8:20 PM, John Wells <lists at sourceillustrated.com> wrote:
>  > > On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
>  > >  >
>  > >  > On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
>  > >  > > does require 'lib/formatters' work?
>  > >  > >
>  > >  > >
>  > >  > > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
>  > >  > >
>  > >  > > > Hi guys,
>  > >  > > >
>  > >  > > > New to backgroundrb, and I like it very much conceptually, but I'm
>  > >  > > > struggling to get it to run successfully. I started with trunk, but
>  > >  > > > was getting a very strange gem require error when starting a worker
>  > >  > > > (though it wasn't telling me which gem was missing)....see here:
>  > >  > > > http://pastie.caboo.se/150225
>  > >  >
>  > >  > Its not complaining about any missing gems, rather its not able to load
>  > >  > workers defined inside lib/workers directory. You must define at least
>  > >  > one worker, before you can start bdrb. In case, you have already defined
>  > >  > a worker, Can we have a look at the code of your worker?
>  > >
>  > >  Thanks for your help. Sure thing...the code is below. However, I'm
>  > >  very curious...how did you determine this was the problem? Just so I
>  > >  know what to look for in the future.
>  >
>  > Because, when I saw error at:
>  >
>  > http://pastie.caboo.se/150225
>  >
>  > It clearly says thats start_worker is failing on 114 line. Generally
>  > this happens, when bdrb is not able to load a particular worker file.
>  > Also, you see empty string "gem_original_require': no such file to
>  > load --  (LoadError)" because bdrb is not able to get name of worker
>  > file to load correctly.
>  >
>  >
>  >
>  > >
>  > >  ===== controller that instantiates worker =====
>  > >  class Admin::BatchController < Admin::ApplicationController
>  > >
>  > >   def initialize
>  > >     super
>  > >     @formats = GW::BatchListing::ListingRow.formats
>  > >     @part_of = :admin
>  > >   end
>  > >
>  > >   def get_progress
>  > >     if request.xhr?
>  > >       progress_percent = MiddleMan.get_worker(session[:job_key]).progress
>  > >       render :update do |page|
>  > >         page.call('progressPercent', 'progressbar', progress_percent)
>  > >         page.redirect_to( :action => 'done')   if progress_percent >= 100
>  > >       end
>  > >     else
>  > >       redirect_to :action => 'index'
>  > >     end
>  > >   end
>  > >
>  > >   def done
>  > >     errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
>  > >     if errors.size > 0
>  > >       flash[:notice] = errors.join("<br/>")
>  > >     else
>  > >       flash[:notice] = "Successful load."
>  > >     end
>  > >     MiddleMan.delete_worker(session[:job_key])
>  > >   end
>  > >
>  > >   def add
>  > >     @user = User.find(params[:id])
>  > >     @cats = Category.find(:all, :conditions=>"parent_id is null")
>  > >     if request.post?
>  > >       listings = params[:listings]
>  > >       content_type = listings.content_type.chomp
>  > >       if "application/zip"!=content_type and
>  > >          "application/x-zip-compressed"!=content_type
>  > >         flash[:notice] =
>  > >           "Only files of type application/zip " +
>  > >           "and application/x-zip-compressed " +
>  > >           "can be used. You uploaded #{content_type}."
>  > >       else
>  > >         # write out to tmp file
>  > >         tmpfile = "/tmp/#{rand 500000}.zip"
>  > >         open(tmpfile, "w") do |f|
>  > >           f << listings.read
>  > >         end
>  > >         parser =
>  > >           GW::BatchListing::ListingParser.new(params[:format],
>  > >                                                         params[:listing_type],
>  > >                                                         @user)
>  > >         session[:job_key] =
>  > >             MiddleMan.new_worker(:class => :foo_worker,
>  > >                                  :args => parser)
>  > >       end
>  > >     end
>  > >   end
>  > >  end
>  > >
>  >
>  > There are couple of problems with above code, for example,
>  > MiddleMan.get_worker is not defined. To get an idea about, which
>  > methods are defined for MiddleMan, have a look at,
>  >
>  > http://backgroundrb.rubyforge.org/classes/BackgrounDRb/WorkerProxy.html
>  >
>  > Also, you should read README file that comes with plugin.
>  >
>  > >  ===== and the worker itself =====
>  > >  class BatchImportWorker < BackgrounDRb::MetaWorker
>  > >   attr_reader :progress
>  > >   attr_reader :errors
>  > >   set_worker_name :batch_import_worker
>  > >
>  > >   def create(args = nil)
>  > >     # this method is called, when worker is loaded for the first time
>  > >   end
>  > >
>  > >   def do_work(parser)
>  > >     @progress = 0
>  > >     @errors = parser.process(tmpfile)
>  > >   end
>  > >  end
>  >
>  > Well, I find  no problems with above worker code, and it should load.
>  > Create a fresh rails app and see if bdrb works there.
>
>
> Ok, sorry... I was working from the old InfoQ article by Ezra and the
>  api docs. I probably made some bad assumptions there...I will take a
>  hard look and try it again.
>
>  Thanks for your help!

Just catching up on this. So, did you manage to get it work after
reading latest docs?
If yes, can you tell us, why bdrb was not able to load your worker?

From rich.everhart at gmail.com  Wed Feb 13 01:47:42 2008
From: rich.everhart at gmail.com (Richard Everhart)
Date: Tue, 12 Feb 2008 22:47:42 -0800
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
Message-ID: <a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>

On Feb 12, 2008 6:02 PM, hemant <gethemant at gmail.com> wrote:
>
> On Wed, Feb 13, 2008 at 7:19 AM, Richard Everhart
> <rich.everhart at gmail.com> wrote:
> >
> > I upgraded ruby right before I wrote my original message and that was
> >  causing some problems.  That's fixed but I still get the 'Not able to
> >  connect' error.  In backgroundrb_server.log there seem to really be
> >  two errors:
> >
> >  /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
> >  `gem_original_require': no such file to load --  (LoadError)
> >        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
> >  `require'
> >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
> >  `require'
> >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:342:in
> >  `new_constants_in'
> >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
> >  `require'
> >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:113:in
> >  `start_worker'
> >  [...snip...]
> >  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:24:in
> >  `read_data': Packet::DisconnectError (Packet::DisconnectError)
> >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/worker.rb:47:in
> >  `handle_internal_messages'
> >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:154:in
> >  `start_reactor'
> >  [...snip...]
> >
> >  The first error seems to be due to a ruby and/or rails configuration
> >  error.  However, I've seen this error before, and the '--' in the
> >  error message is usually followed by the name of the file that can't
> >  be loaded.
> >
>
> Did you define a worker before starting backgroundrb server? Above
> error is because, bdrb is not able to load defined worker.
> If you have the worker defined can we see the code of the worker.
>

Here is my controller and worker:

class TestdrbController < ApplicationController
  def do_fib
    puts ">>> do_fib: #{params[:input]}"

    session[:job_key] = MiddleMan.new_worker(:class =>
:fibonacci_worker, :data => params[:input])

    puts "Job key: #{session[:job_key]}"

    MiddleMan.send_request(:worker => :fibonacci_worker,
:worker_method => :do_work, :data => params[:input])

    puts "<<< do_fib"

    render :action => 'result'
  end

  def result
    puts ">>> result"
    MiddleMan.delete_worker(:worker => :fibonacci_worker, :job_key =>
session[:job_key])
    puts "<<< result"
  end
end

class FibonacciWorker < BackgrounDRb::MetaWorker
  include Fibonacci

  set_worker_name :fibonacci_worker
  set_no_auto_load true

  def create(args = nil)
    logger.info("Worker create: '#{args}'")

    register_status("Processing started")

    n = args.to_i
    logger.info("Worker result: #{f(n)}")
  end

  def do_work(data)
    logger.info ">>> do_work"

    result = f(data.to_i)
    logger.info "Worker result: #{result}"

    return result
  end
end


Thanks again.

Richard

From gethemant at gmail.com  Wed Feb 13 04:46:23 2008
From: gethemant at gmail.com (hemant kumar)
Date: Wed, 13 Feb 2008 15:16:23 +0530
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
	<a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
Message-ID: <1202895983.6779.4.camel@shire>

Hi,

On Tue, 2008-02-12 at 22:47 -0800, Richard Everhart wrote:
> On Feb 12, 2008 6:02 PM, hemant <gethemant at gmail.com> wrote:
> >
> > On Wed, Feb 13, 2008 at 7:19 AM, Richard Everhart
> > <rich.everhart at gmail.com> wrote:
> > >
> > > I upgraded ruby right before I wrote my original message and that was
> > >  causing some problems.  That's fixed but I still get the 'Not able to
> > >  connect' error.  In backgroundrb_server.log there seem to really be
> > >  two errors:
> > >
> > >  /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
> > >  `gem_original_require': no such file to load --  (LoadError)
> > >        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
> > >  `require'
> > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
> > >  `require'
> > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:342:in
> > >  `new_constants_in'
> > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
> > >  `require'
> > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:113:in
> > >  `start_worker'
> > >  [...snip...]
> > >  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:24:in
> > >  `read_data': Packet::DisconnectError (Packet::DisconnectError)
> > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/worker.rb:47:in
> > >  `handle_internal_messages'
> > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:154:in
> > >  `start_reactor'
> > >  [...snip...]
> > >
> > >  The first error seems to be due to a ruby and/or rails configuration
> > >  error.  However, I've seen this error before, and the '--' in the
> > >  error message is usually followed by the name of the file that can't
> > >  be loaded.
> > >
> >
> > Did you define a worker before starting backgroundrb server? Above
> > error is because, bdrb is not able to load defined worker.
> > If you have the worker defined can we see the code of the worker.
> >
> 
> Here is my controller and worker:
> 
> class TestdrbController < ApplicationController
>   def do_fib
>     puts ">>> do_fib: #{params[:input]}"
> 
>     session[:job_key] = MiddleMan.new_worker(:class =>
> :fibonacci_worker, :data => params[:input])
> 
>     puts "Job key: #{session[:job_key]}"
> 
>     MiddleMan.send_request(:worker => :fibonacci_worker,
> :worker_method => :do_work, :data => params[:input])
> 
>     puts "<<< do_fib"
> 
>     render :action => 'result'
>   end
> 
>   def result
>     puts ">>> result"
>     MiddleMan.delete_worker(:worker => :fibonacci_worker, :job_key =>
> session[:job_key])
>     puts "<<< result"
>   end
> end

Although this is not related to your problem, but above controller code
has a bit of a problem, when you are not passing any job_key while
starting the worker via new_worker method, no job_key will be used and
while deleting the worker or invoking a task on the worker you should
not use job_key, if you haven't specified a job_key while creating the
worker.

> 
> class FibonacciWorker < BackgrounDRb::MetaWorker
>   include Fibonacci
> 
>   set_worker_name :fibonacci_worker
>   set_no_auto_load true
> 
>   def create(args = nil)
>     logger.info("Worker create: '#{args}'")
> 
>     register_status("Processing started")
> 
>     n = args.to_i
>     logger.info("Worker result: #{f(n)}")
>   end
> 
>   def do_work(data)
>     logger.info ">>> do_work"
> 
>     result = f(data.to_i)
>     logger.info "Worker result: #{result}"
> 
>     return result
>   end
> end

I took above worker code and created new rails application, checked out
bdrb code from trunk at devjavu and i was able to start bdrb without
problems. In a nutshell, I am unable to reproduce this problem. Now, can
you zip and send me your application through mail, so as I can see whats
wrong there?



From lists at sourceillustrated.com  Wed Feb 13 09:20:41 2008
From: lists at sourceillustrated.com (John Wells)
Date: Wed, 13 Feb 2008 09:20:41 -0500
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <fb283d5e0802121803i20d9804q1f9a18f7072dd14d@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
	<44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
	<fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>
	<44dddf400802111035h7f2fb34ajeda347a35235309c@mail.gmail.com>
	<fb283d5e0802121803i20d9804q1f9a18f7072dd14d@mail.gmail.com>
Message-ID: <44dddf400802130620w1b778e6fp2aaebcbdd1ac36c2@mail.gmail.com>

On Feb 12, 2008 9:03 PM, hemant <gethemant at gmail.com> wrote:
> On Tue, Feb 12, 2008 at 12:05 AM, John Wells
>
> <lists at sourceillustrated.com> wrote:
> >
> > On Feb 11, 2008 1:28 PM, hemant <gethemant at gmail.com> wrote:
> >  > On Mon, Feb 11, 2008 at 8:20 PM, John Wells <lists at sourceillustrated.com> wrote:
> >  > > On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
> >  > >  >
> >  > >  > On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
> >  > >  > > does require 'lib/formatters' work?
> >  > >  > >
> >  > >  > >
> >  > >  > > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
> >  > >  > >
> >  > >  > > > Hi guys,
> >  > >  > > >
> >  > >  > > > New to backgroundrb, and I like it very much conceptually, but I'm
> >  > >  > > > struggling to get it to run successfully. I started with trunk, but
> >  > >  > > > was getting a very strange gem require error when starting a worker
> >  > >  > > > (though it wasn't telling me which gem was missing)....see here:
> >  > >  > > > http://pastie.caboo.se/150225
> >  > >  >
> >  > >  > Its not complaining about any missing gems, rather its not able to load
> >  > >  > workers defined inside lib/workers directory. You must define at least
> >  > >  > one worker, before you can start bdrb. In case, you have already defined
> >  > >  > a worker, Can we have a look at the code of your worker?
> >  > >
> >  > >  Thanks for your help. Sure thing...the code is below. However, I'm
> >  > >  very curious...how did you determine this was the problem? Just so I
> >  > >  know what to look for in the future.
> >  >
> >  > Because, when I saw error at:
> >  >
> >  > http://pastie.caboo.se/150225
> >  >
> >  > It clearly says thats start_worker is failing on 114 line. Generally
> >  > this happens, when bdrb is not able to load a particular worker file.
> >  > Also, you see empty string "gem_original_require': no such file to
> >  > load --  (LoadError)" because bdrb is not able to get name of worker
> >  > file to load correctly.
> >  >
> >  >
> >  >
> >  > >
> >  > >  ===== controller that instantiates worker =====
> >  > >  class Admin::BatchController < Admin::ApplicationController
> >  > >
> >  > >   def initialize
> >  > >     super
> >  > >     @formats = GW::BatchListing::ListingRow.formats
> >  > >     @part_of = :admin
> >  > >   end
> >  > >
> >  > >   def get_progress
> >  > >     if request.xhr?
> >  > >       progress_percent = MiddleMan.get_worker(session[:job_key]).progress
> >  > >       render :update do |page|
> >  > >         page.call('progressPercent', 'progressbar', progress_percent)
> >  > >         page.redirect_to( :action => 'done')   if progress_percent >= 100
> >  > >       end
> >  > >     else
> >  > >       redirect_to :action => 'index'
> >  > >     end
> >  > >   end
> >  > >
> >  > >   def done
> >  > >     errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
> >  > >     if errors.size > 0
> >  > >       flash[:notice] = errors.join("<br/>")
> >  > >     else
> >  > >       flash[:notice] = "Successful load."
> >  > >     end
> >  > >     MiddleMan.delete_worker(session[:job_key])
> >  > >   end
> >  > >
> >  > >   def add
> >  > >     @user = User.find(params[:id])
> >  > >     @cats = Category.find(:all, :conditions=>"parent_id is null")
> >  > >     if request.post?
> >  > >       listings = params[:listings]
> >  > >       content_type = listings.content_type.chomp
> >  > >       if "application/zip"!=content_type and
> >  > >          "application/x-zip-compressed"!=content_type
> >  > >         flash[:notice] =
> >  > >           "Only files of type application/zip " +
> >  > >           "and application/x-zip-compressed " +
> >  > >           "can be used. You uploaded #{content_type}."
> >  > >       else
> >  > >         # write out to tmp file
> >  > >         tmpfile = "/tmp/#{rand 500000}.zip"
> >  > >         open(tmpfile, "w") do |f|
> >  > >           f << listings.read
> >  > >         end
> >  > >         parser =
> >  > >           GW::BatchListing::ListingParser.new(params[:format],
> >  > >                                                         params[:listing_type],
> >  > >                                                         @user)
> >  > >         session[:job_key] =
> >  > >             MiddleMan.new_worker(:class => :foo_worker,
> >  > >                                  :args => parser)
> >  > >       end
> >  > >     end
> >  > >   end
> >  > >  end
> >  > >
> >  >
> >  > There are couple of problems with above code, for example,
> >  > MiddleMan.get_worker is not defined. To get an idea about, which
> >  > methods are defined for MiddleMan, have a look at,
> >  >
> >  > http://backgroundrb.rubyforge.org/classes/BackgrounDRb/WorkerProxy.html
> >  >
> >  > Also, you should read README file that comes with plugin.
> >  >
> >  > >  ===== and the worker itself =====
> >  > >  class BatchImportWorker < BackgrounDRb::MetaWorker
> >  > >   attr_reader :progress
> >  > >   attr_reader :errors
> >  > >   set_worker_name :batch_import_worker
> >  > >
> >  > >   def create(args = nil)
> >  > >     # this method is called, when worker is loaded for the first time
> >  > >   end
> >  > >
> >  > >   def do_work(parser)
> >  > >     @progress = 0
> >  > >     @errors = parser.process(tmpfile)
> >  > >   end
> >  > >  end
> >  >
> >  > Well, I find  no problems with above worker code, and it should load.
> >  > Create a fresh rails app and see if bdrb works there.
> >
> >
> > Ok, sorry... I was working from the old InfoQ article by Ezra and the
> >  api docs. I probably made some bad assumptions there...I will take a
> >  hard look and try it again.
> >
> >  Thanks for your help!
>
> Just catching up on this. So, did you manage to get it work after
> reading latest docs?
> If yes, can you tell us, why bdrb was not able to load your worker?
>

It was a number of small little errors, and some big ones. Perhaps the
most damning was:

@errors = parser.process(tmpfile)

You'll note tmpfile doesn't exist in this context ;) That would cause a problem.

I have my solution working  now and it's working well. I really
appreciate your help.

My biggest hurdle with backgroundrb was the fact that I have some very
complicated business rules taking place on the brdb side of things,
and without an easy way to step through these with a debugger, lots of
things can go wrong and it makes life more difficult. Running
script/backgroundrb without arguments and using puts and inspect
helps, but hooking to it with a debugger so that one could inspect
variables, etc would be extremely useful. I started going down the
path of getting NetBeans' debugger to work with it, but quickly
realized it was going to take more time than I had given my current
project.

Anyway, thanks very much for your help!

John

From gethemant at gmail.com  Wed Feb 13 09:42:47 2008
From: gethemant at gmail.com (hemant kumar)
Date: Wed, 13 Feb 2008 20:12:47 +0530
Subject: [Backgroundrb-devel] Problems running backgroundrb
In-Reply-To: <44dddf400802130620w1b778e6fp2aaebcbdd1ac36c2@mail.gmail.com>
References: <44dddf400802102211u3827b5bar8b625cb800655886@mail.gmail.com>
	<12AACA96-EFEC-44C2-B743-1E0716100F45@gmail.com>
	<1202724215.6982.8.camel@shire>
	<44dddf400802110650m40cefcddj738644d848cc0fa6@mail.gmail.com>
	<fb283d5e0802111028y6e3a2eb2qf93165a45863f6ef@mail.gmail.com>
	<44dddf400802111035h7f2fb34ajeda347a35235309c@mail.gmail.com>
	<fb283d5e0802121803i20d9804q1f9a18f7072dd14d@mail.gmail.com>
	<44dddf400802130620w1b778e6fp2aaebcbdd1ac36c2@mail.gmail.com>
Message-ID: <1202913767.6779.17.camel@shire>


On Wed, 2008-02-13 at 09:20 -0500, John Wells wrote:
> On Feb 12, 2008 9:03 PM, hemant <gethemant at gmail.com> wrote:
> > On Tue, Feb 12, 2008 at 12:05 AM, John Wells
> >
> > <lists at sourceillustrated.com> wrote:
> > >
> > > On Feb 11, 2008 1:28 PM, hemant <gethemant at gmail.com> wrote:
> > >  > On Mon, Feb 11, 2008 at 8:20 PM, John Wells <lists at sourceillustrated.com> wrote:
> > >  > > On Feb 11, 2008 5:03 AM, hemant kumar <gethemant at gmail.com> wrote:
> > >  > >  >
> > >  > >  > On Mon, 2008-02-11 at 00:55 -0800, Alex Soto wrote:
> > >  > >  > > does require 'lib/formatters' work?
> > >  > >  > >
> > >  > >  > >
> > >  > >  > > On Feb 10, 2008, at 10:11 PM, John Wells wrote:
> > >  > >  > >
> > >  > >  > > > Hi guys,
> > >  > >  > > >
> > >  > >  > > > New to backgroundrb, and I like it very much conceptually, but I'm
> > >  > >  > > > struggling to get it to run successfully. I started with trunk, but
> > >  > >  > > > was getting a very strange gem require error when starting a worker
> > >  > >  > > > (though it wasn't telling me which gem was missing)....see here:
> > >  > >  > > > http://pastie.caboo.se/150225
> > >  > >  >
> > >  > >  > Its not complaining about any missing gems, rather its not able to load
> > >  > >  > workers defined inside lib/workers directory. You must define at least
> > >  > >  > one worker, before you can start bdrb. In case, you have already defined
> > >  > >  > a worker, Can we have a look at the code of your worker?
> > >  > >
> > >  > >  Thanks for your help. Sure thing...the code is below. However, I'm
> > >  > >  very curious...how did you determine this was the problem? Just so I
> > >  > >  know what to look for in the future.
> > >  >
> > >  > Because, when I saw error at:
> > >  >
> > >  > http://pastie.caboo.se/150225
> > >  >
> > >  > It clearly says thats start_worker is failing on 114 line. Generally
> > >  > this happens, when bdrb is not able to load a particular worker file.
> > >  > Also, you see empty string "gem_original_require': no such file to
> > >  > load --  (LoadError)" because bdrb is not able to get name of worker
> > >  > file to load correctly.
> > >  >
> > >  >
> > >  >
> > >  > >
> > >  > >  ===== controller that instantiates worker =====
> > >  > >  class Admin::BatchController < Admin::ApplicationController
> > >  > >
> > >  > >   def initialize
> > >  > >     super
> > >  > >     @formats = GW::BatchListing::ListingRow.formats
> > >  > >     @part_of = :admin
> > >  > >   end
> > >  > >
> > >  > >   def get_progress
> > >  > >     if request.xhr?
> > >  > >       progress_percent = MiddleMan.get_worker(session[:job_key]).progress
> > >  > >       render :update do |page|
> > >  > >         page.call('progressPercent', 'progressbar', progress_percent)
> > >  > >         page.redirect_to( :action => 'done')   if progress_percent >= 100
> > >  > >       end
> > >  > >     else
> > >  > >       redirect_to :action => 'index'
> > >  > >     end
> > >  > >   end
> > >  > >
> > >  > >   def done
> > >  > >     errors = "DONE " + MiddleMan.get_worker(session[:job_key]).errors
> > >  > >     if errors.size > 0
> > >  > >       flash[:notice] = errors.join("<br/>")
> > >  > >     else
> > >  > >       flash[:notice] = "Successful load."
> > >  > >     end
> > >  > >     MiddleMan.delete_worker(session[:job_key])
> > >  > >   end
> > >  > >
> > >  > >   def add
> > >  > >     @user = User.find(params[:id])
> > >  > >     @cats = Category.find(:all, :conditions=>"parent_id is null")
> > >  > >     if request.post?
> > >  > >       listings = params[:listings]
> > >  > >       content_type = listings.content_type.chomp
> > >  > >       if "application/zip"!=content_type and
> > >  > >          "application/x-zip-compressed"!=content_type
> > >  > >         flash[:notice] =
> > >  > >           "Only files of type application/zip " +
> > >  > >           "and application/x-zip-compressed " +
> > >  > >           "can be used. You uploaded #{content_type}."
> > >  > >       else
> > >  > >         # write out to tmp file
> > >  > >         tmpfile = "/tmp/#{rand 500000}.zip"
> > >  > >         open(tmpfile, "w") do |f|
> > >  > >           f << listings.read
> > >  > >         end
> > >  > >         parser =
> > >  > >           GW::BatchListing::ListingParser.new(params[:format],
> > >  > >                                                         params[:listing_type],
> > >  > >                                                         @user)
> > >  > >         session[:job_key] =
> > >  > >             MiddleMan.new_worker(:class => :foo_worker,
> > >  > >                                  :args => parser)
> > >  > >       end
> > >  > >     end
> > >  > >   end
> > >  > >  end
> > >  > >
> > >  >
> > >  > There are couple of problems with above code, for example,
> > >  > MiddleMan.get_worker is not defined. To get an idea about, which
> > >  > methods are defined for MiddleMan, have a look at,
> > >  >
> > >  > http://backgroundrb.rubyforge.org/classes/BackgrounDRb/WorkerProxy.html
> > >  >
> > >  > Also, you should read README file that comes with plugin.
> > >  >
> > >  > >  ===== and the worker itself =====
> > >  > >  class BatchImportWorker < BackgrounDRb::MetaWorker
> > >  > >   attr_reader :progress
> > >  > >   attr_reader :errors
> > >  > >   set_worker_name :batch_import_worker
> > >  > >
> > >  > >   def create(args = nil)
> > >  > >     # this method is called, when worker is loaded for the first time
> > >  > >   end
> > >  > >
> > >  > >   def do_work(parser)
> > >  > >     @progress = 0
> > >  > >     @errors = parser.process(tmpfile)
> > >  > >   end
> > >  > >  end
> > >  >
> > >  > Well, I find  no problems with above worker code, and it should load.
> > >  > Create a fresh rails app and see if bdrb works there.
> > >
> > >
> > > Ok, sorry... I was working from the old InfoQ article by Ezra and the
> > >  api docs. I probably made some bad assumptions there...I will take a
> > >  hard look and try it again.
> > >
> > >  Thanks for your help!
> >
> > Just catching up on this. So, did you manage to get it work after
> > reading latest docs?
> > If yes, can you tell us, why bdrb was not able to load your worker?
> >
> 
> It was a number of small little errors, and some big ones. Perhaps the
> most damning was:
> 
> @errors = parser.process(tmpfile)
> 
> You'll note tmpfile doesn't exist in this context ;) That would cause a problem.
> 
> I have my solution working  now and it's working well. I really
> appreciate your help.
> 
> My biggest hurdle with backgroundrb was the fact that I have some very
> complicated business rules taking place on the brdb side of things,
> and without an easy way to step through these with a debugger, lots of
> things can go wrong and it makes life more difficult. Running
> script/backgroundrb without arguments and using puts and inspect
> helps, but hooking to it with a debugger so that one could inspect
> variables, etc would be extremely useful. I started going down the
> path of getting NetBeans' debugger to work with it, but quickly
> realized it was going to take more time than I had given my current
> project.

I have similar project, where I have complicated set of business rules
in workers. Stepping through debugger is no solution, Please write test
cases for workers, thats the only way.


From gethemant at gmail.com  Wed Feb 13 14:45:41 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Feb 2008 01:15:41 +0530
Subject: [Backgroundrb-devel] BackgrounDRb prerelease for 1.0.3 version
Message-ID: <fb283d5e0802131145m28323b70ue739486d72b062a@mail.gmail.com>

Hullo there,

I have worker out quite some changes as promised in last mail. Here is
a brief ChangeLog:

	* Added TestCases for Cron Triggers, Meta Workers and stuff. We are
heading towards proper code coverage with specs.

	* Added preliminary support for starting a worker on demand through
scheduler. What it means is, when you have a worker which is getting
	scheduled very less frequently and you don't want the worker to
persist, you can ask BackgrounDRb to restart the worker on each
schedule.

	* Fixed some unreported issues with writing data to socket between
workers and stuff.

	* Fixed issues with too many open connections, because connections
were not getting closed. BackgrounDRb now opens only one connection,
which is
	reused throughout the lifecycle of rails application.

	* Fixed all outstanding issues with Cron triggers, BackgrounDRb now
explicitly depends on "chronic" gem.

	* Removed Framework directory and BackgrounDRb now explicitly depends
on packet gem.

To install:

gem install packet chronic

Checkout code from git repository:

git clone git at gitorious.org:backgroundrb/mainline.git backgroundrb

As usual, remove old backgroundrb script from script directory and
start bdrb server.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Thu Feb 14 12:52:36 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 14 Feb 2008 23:22:36 +0530
Subject: [Backgroundrb-devel] BackgrounDRb prerelease for 1.0.3 version
In-Reply-To: <fb283d5e0802131145m28323b70ue739486d72b062a@mail.gmail.com>
References: <fb283d5e0802131145m28323b70ue739486d72b062a@mail.gmail.com>
Message-ID: <fb283d5e0802140952t1f340783p98c2367be9fe2d7@mail.gmail.com>

On 2/14/08, hemant <gethemant at gmail.com> wrote:
> Hullo there,
>
>  I have worker out quite some changes as promised in last mail. Here is
>  a brief ChangeLog:
>
>         * Added TestCases for Cron Triggers, Meta Workers and stuff. We are
>  heading towards proper code coverage with specs.
>
>         * Added preliminary support for starting a worker on demand through
>  scheduler. What it means is, when you have a worker which is getting
>         scheduled very less frequently and you don't want the worker to
>  persist, you can ask BackgrounDRb to restart the worker on each
>  schedule.
>
>         * Fixed some unreported issues with writing data to socket between
>  workers and stuff.
>
>         * Fixed issues with too many open connections, because connections
>  were not getting closed. BackgrounDRb now opens only one connection,
>  which is
>         reused throughout the lifecycle of rails application.
>
>         * Fixed all outstanding issues with Cron triggers, BackgrounDRb now
>  explicitly depends on "chronic" gem.
>
>         * Removed Framework directory and BackgrounDRb now explicitly depends
>  on packet gem.
>
>  To install:
>
>  gem install packet chronic
>
>  Checkout code from git repository:
>
>  git clone git at gitorious.org:backgroundrb/mainline.git backgroundrb
>
>  As usual, remove old backgroundrb script from script directory and
>  start bdrb server.
>

I will really appreciate some feedback on above release, because those
Socket write issues I mentioned were serious ones and both "Packet"
library and bdrb suffered from them.  Let me explain what they were
and why they matter:

When reactor loop was writing data to sockets, it was not using
"writable" status on file descriptors.  For small amount of data being
written to the socket this is not a problem, but for larger objects it
could be a problem. Now, write happens in an evented manner and all
the data is "buffered" in memory until socket is ready for write. Bdrb
( or packet ) will write as much data as it can and will buffer the
remaining data for writing when socket is ready next time for write.

It was a known problem and I took some pain to fix this. So, Bdrb will
be much more robust in production environments now. Also, CronTrigger
code went through major cleanup and testing.

I implemented a function called next_turn in packet, which lets you
run random tasks at next turn of event loop. This will be used for
runtime loading of workers and task queues.

From gtcampbell at gmail.com  Thu Feb 14 14:12:46 2008
From: gtcampbell at gmail.com (Greg Campbell)
Date: Thu, 14 Feb 2008 11:12:46 -0800
Subject: [Backgroundrb-devel] Problem with new version from git repository
Message-ID: <ac6faba20802141112u6bbf3c3dsa101aac4624c2e57@mail.gmail.com>

Hello all,

I recently grabbed the 1.0.3 prerelease from gitorious as requested by
Hemant, and I had an odd issue - I'm wondering if anyone else is seeing the
same thing.  When I attempt to start the backgroundrb server, I get the
following error output.  Is the problem in the first line ("no such worker
log_worker"), or is there something else going on?

no such worker log_worker
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_meta_pimp.rb:48:in
`process_request': You have a nil object when you didn't expect it!
(NoMethodError)
The error occurred while evaluating nil.send_request    from
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_meta_pimp.rb:27:in
`handle_object'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_meta_pimp.rb:20:in
`receive_data'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_parser.rb:37:in
`call'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_parser.rb:37:in
`extract'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_meta_pimp.rb:18:in
`receive_data'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:64:in
`handle_internal_messages'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:171:in
`handle_read_event'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:169:in
`each'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:169:in
`handle_read_event'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:125:in
`start_reactor'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:120:in
`loop'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:120:in
`start_reactor'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:21:in
`run'
        from /(my site
folder)/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:161:in
`initialize'
        from script/backgroundrb:55:in `new'
        from script/backgroundrb:55
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:171:in
`handle_read_event'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:169:in
`each'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:169:in
`handle_read_event'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:125:in
`start_reactor'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:120:in
`loop'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_core.rb:120:in
`start_reactor'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_worker.rb:20:in
`start_worker'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:133:in
`fork_and_load'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:96:in
`load_workers'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:91:in
`each'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:91:in
`load_workers'
        from /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.4/lib/packet/packet_master.rb:20:in
`run'
        from /(my site
folder)/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:161:in
`initialize'
        from script/backgroundrb:55:in `new'
        from script/backgroundrb:55


Thanks,
Greg Campbell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080214/9b95ef98/attachment.html 

From gethemant at gmail.com  Thu Feb 14 14:32:48 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Feb 2008 01:02:48 +0530
Subject: [Backgroundrb-devel] Problem with new version from git
	repository
In-Reply-To: <ac6faba20802141112u6bbf3c3dsa101aac4624c2e57@mail.gmail.com>
References: <ac6faba20802141112u6bbf3c3dsa101aac4624c2e57@mail.gmail.com>
Message-ID: <fb283d5e0802141132l692d7a91i77726ed809a89288@mail.gmail.com>

On 2/15/08, Greg Campbell <gtcampbell at gmail.com> wrote:
> Hello all,
>
> I recently grabbed the 1.0.3 prerelease from gitorious as requested by
> Hemant, and I had an odd issue - I'm wondering if anyone else is seeing the
> same thing.  When I attempt to start the backgroundrb server, I get the
> following error output.  Is the problem in the first line ("no such worker
> log_worker"), or is there something else going on?
>

I think I forgot to commit this fix somehow. When I go home, I will
commit the fix, but currently open file "backgroundrb" present in
script directory of your rails root and:

remove this:
-  ["server","framework","lib"].each { |x|
$LOAD_PATH.unshift(PACKET_APP + "/#{x}")}

and add this:
+ ["server","server/lib","lib"].each { |x|
$LOAD_PATH.unshift(PACKET_APP + "/#{x}")}

This should fix your problem.

From rich.everhart at gmail.com  Thu Feb 14 17:45:17 2008
From: rich.everhart at gmail.com (Richard Everhart)
Date: Thu, 14 Feb 2008 14:45:17 -0800
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <1202895983.6779.4.camel@shire>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
	<a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
	<1202895983.6779.4.camel@shire>
Message-ID: <a242781c0802141445o38fd445cv6285bd9689551e6@mail.gmail.com>

Thanks, Hemant.  I made the change related to the job key and I'm not
getting a value back from new_worker.  However, the 'not able to
connect error' is on going.

My zipped up app is attached.  Thanks for helping me out.

Rich

On Wed, Feb 13, 2008 at 1:46 AM, hemant kumar <gethemant at gmail.com> wrote:
> Hi,
>
>
>
>  On Tue, 2008-02-12 at 22:47 -0800, Richard Everhart wrote:
>  > On Feb 12, 2008 6:02 PM, hemant <gethemant at gmail.com> wrote:
>  > >
>  > > On Wed, Feb 13, 2008 at 7:19 AM, Richard Everhart
>  > > <rich.everhart at gmail.com> wrote:
>  > > >
>  > > > I upgraded ruby right before I wrote my original message and that was
>  > > >  causing some problems.  That's fixed but I still get the 'Not able to
>  > > >  connect' error.  In backgroundrb_server.log there seem to really be
>  > > >  two errors:
>  > > >
>  > > >  /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
>  > > >  `gem_original_require': no such file to load --  (LoadError)
>  > > >        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
>  > > >  `require'
>  > > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
>  > > >  `require'
>  > > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:342:in
>  > > >  `new_constants_in'
>  > > >        from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in
>  > > >  `require'
>  > > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:113:in
>  > > >  `start_worker'
>  > > >  [...snip...]
>  > > >  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/nbio.rb:24:in
>  > > >  `read_data': Packet::DisconnectError (Packet::DisconnectError)
>  > > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/worker.rb:47:in
>  > > >  `handle_internal_messages'
>  > > >        from /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:154:in
>  > > >  `start_reactor'
>  > > >  [...snip...]
>  > > >
>  > > >  The first error seems to be due to a ruby and/or rails configuration
>  > > >  error.  However, I've seen this error before, and the '--' in the
>  > > >  error message is usually followed by the name of the file that can't
>  > > >  be loaded.
>  > > >
>  > >
>  > > Did you define a worker before starting backgroundrb server? Above
>  > > error is because, bdrb is not able to load defined worker.
>  > > If you have the worker defined can we see the code of the worker.
>  > >
>  >
>  > Here is my controller and worker:
>  >
>  > class TestdrbController < ApplicationController
>  >   def do_fib
>  >     puts ">>> do_fib: #{params[:input]}"
>  >
>  >     session[:job_key] = MiddleMan.new_worker(:class =>
>  > :fibonacci_worker, :data => params[:input])
>  >
>  >     puts "Job key: #{session[:job_key]}"
>  >
>  >     MiddleMan.send_request(:worker => :fibonacci_worker,
>  > :worker_method => :do_work, :data => params[:input])
>  >
>  >     puts "<<< do_fib"
>  >
>  >     render :action => 'result'
>  >   end
>  >
>  >   def result
>  >     puts ">>> result"
>  >     MiddleMan.delete_worker(:worker => :fibonacci_worker, :job_key =>
>  > session[:job_key])
>  >     puts "<<< result"
>  >   end
>  > end
>
>  Although this is not related to your problem, but above controller code
>  has a bit of a problem, when you are not passing any job_key while
>  starting the worker via new_worker method, no job_key will be used and
>  while deleting the worker or invoking a task on the worker you should
>  not use job_key, if you haven't specified a job_key while creating the
>  worker.
>
>
>  >
>  > class FibonacciWorker < BackgrounDRb::MetaWorker
>  >   include Fibonacci
>  >
>  >   set_worker_name :fibonacci_worker
>  >   set_no_auto_load true
>  >
>  >   def create(args = nil)
>  >     logger.info("Worker create: '#{args}'")
>  >
>  >     register_status("Processing started")
>  >
>  >     n = args.to_i
>  >     logger.info("Worker result: #{f(n)}")
>  >   end
>  >
>  >   def do_work(data)
>  >     logger.info ">>> do_work"
>  >
>  >     result = f(data.to_i)
>  >     logger.info "Worker result: #{result}"
>  >
>  >     return result
>  >   end
>  > end
>
>  I took above worker code and created new rails application, checked out
>  bdrb code from trunk at devjavu and i was able to start bdrb without
>  problems. In a nutshell, I am unable to reproduce this problem. Now, can
>  you zip and send me your application through mail, so as I can see whats
>  wrong there?
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: drb_test01.zip
Type: application/zip
Size: 385132 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080214/972cad18/attachment-0001.zip 

From gethemant at gmail.com  Thu Feb 14 18:24:39 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Feb 2008 04:54:39 +0530
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <a242781c0802141445o38fd445cv6285bd9689551e6@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
	<a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
	<1202895983.6779.4.camel@shire>
	<a242781c0802141445o38fd445cv6285bd9689551e6@mail.gmail.com>
Message-ID: <fb283d5e0802141524r5184cd98w7bfe66d2da40c778@mail.gmail.com>

On Fri, Feb 15, 2008 at 4:15 AM, Richard Everhart
<rich.everhart at gmail.com> wrote:
> Thanks, Hemant.  I made the change related to the job key and I'm not
>  getting a value back from new_worker.  However, the 'not able to
>  connect error' is on going.
>
>  My zipped up app is attached.  Thanks for helping me out.
>

First Problems with your controller code:

You have line:

session[:job_key] = MiddleMan.new_worker(:class => :fibonacci_worker,
:job_key => 'the_key', :data => params[:input])

Please note that, it should be :worker not :class, so please change the line to:
   session[:job_key] = MiddleMan.new_worker(:worker =>
:fibonacci_worker, :job_key => 'the_key', :data => params[:input])

Now, You have line:

   MiddleMan.send_request(:worker => :fibonacci_worker, :worker_method
=> :do_work, :data => params[:input])

Again, as i said earlier, when you are creating a worker with a
"job_key" you *must* always access that instance of worker with the
same job key and hence it should be:

   MiddleMan.send_request(:worker => :fibonacci_worker, :worker_method
=> :do_work, :data => params[:input], :job_key => session[:job_key])

Your worker code is perfectly fine and should work as it. Also, update
your plugin from trunk and remove "backgroundrb" script present in
script directory and restart the server and try now. It will work.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From rich.everhart at gmail.com  Thu Feb 14 20:41:34 2008
From: rich.everhart at gmail.com (Richard Everhart)
Date: Thu, 14 Feb 2008 17:41:34 -0800
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <fb283d5e0802141524r5184cd98w7bfe66d2da40c778@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
	<a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
	<1202895983.6779.4.camel@shire>
	<a242781c0802141445o38fd445cv6285bd9689551e6@mail.gmail.com>
	<fb283d5e0802141524r5184cd98w7bfe66d2da40c778@mail.gmail.com>
Message-ID: <a242781c0802141741s10d8bac4x3b3e7b8bd9a37154@mail.gmail.com>

Changing 'class' to 'worker' in new_worker seemed to get things going.
 I can see that after new_worker is called the worker's create method
is then called.  This only seems to work once. After the first time, I
get the following error:

You have a nil object when you didn't expect it!
The error occurred while evaluating nil.send_request
/home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in
`ask_worker'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:76:in
`process_work'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:34:in
`receive_data'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in
`call'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in
`extract'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:30:in
`receive_data'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:195:in
`read_external_socket'
/home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:187:in
`handle_external_messages'
[...snip...]

Eventually, what I would really like to do is return a value back to
the browser.  send_request is documented to do this but the browser
just hangs and I really can't tell if the the worker method (do_work
in this is being executed).  While the work is being executed by the
worker I'd like to display a progress bar or a busy indicator, both of
which I've seen examples.  But, it seems that the do_work method is
never called, even when calling ask_work, which is not supposed to
block.

I have not tried out the latest version so I'm not sure if that will
change things.  Thanks a lot!

Rich

On Thu, Feb 14, 2008 at 3:24 PM, hemant <gethemant at gmail.com> wrote:
> On Fri, Feb 15, 2008 at 4:15 AM, Richard Everhart
>
> <rich.everhart at gmail.com> wrote:
>
> > Thanks, Hemant.  I made the change related to the job key and I'm not
>  >  getting a value back from new_worker.  However, the 'not able to
>  >  connect error' is on going.
>  >
>  >  My zipped up app is attached.  Thanks for helping me out.
>  >
>
>  First Problems with your controller code:
>
>  You have line:
>
>
>  session[:job_key] = MiddleMan.new_worker(:class => :fibonacci_worker,
>  :job_key => 'the_key', :data => params[:input])
>
>  Please note that, it should be :worker not :class, so please change the line to:
>    session[:job_key] = MiddleMan.new_worker(:worker =>
>  :fibonacci_worker, :job_key => 'the_key', :data => params[:input])
>
>  Now, You have line:
>
>
>    MiddleMan.send_request(:worker => :fibonacci_worker, :worker_method
>  => :do_work, :data => params[:input])
>
>  Again, as i said earlier, when you are creating a worker with a
>  "job_key" you *must* always access that instance of worker with the
>  same job key and hence it should be:
>
>
>    MiddleMan.send_request(:worker => :fibonacci_worker, :worker_method
>  => :do_work, :data => params[:input], :job_key => session[:job_key])
>
>  Your worker code is perfectly fine and should work as it. Also, update
>  your plugin from trunk and remove "backgroundrb" script present in
>  script directory and restart the server and try now. It will work.
>
>
>  --
>  Let them talk of their oriental summer climes of everlasting
>  conservatories; give me the privilege of making my own summer with my
>  own coals.
>
>  http://gnufied.org
>

From stibrewal at gmail.com  Thu Feb 14 23:04:30 2008
From: stibrewal at gmail.com (Sanjay Tibrewal)
Date: Thu, 14 Feb 2008 20:04:30 -0800
Subject: [Backgroundrb-devel] BacgroundRB - Getting started question
Message-ID: <47b50f50.16078e0a.559f.ffff9dec@mx.google.com>

Resending this message - I sent it earlier in the day and I still don't see
it show up on the mailing list.

Thanks.

-----------------------------------------
Hello Folks,

I have been trying to get backgroundrb running on my Win XP box to
periodically wakeup and run a worker function. I wrote a test Worker class
as below and my backgroundrb.yml is included below. I can start backgroundrb
successfully through commondline (ruby script/backgroundrb/start) but I
don't see any output in my backgroundrb.log - I would expect the "Hello"
string in the log every 5 secs. 

The first question I have is, whether what I am doing even supported on
Windows? Reason I ask is because I saw a post from a few months back saying
something to the effect that forking doesn't happen on Windows (a reply
email from Hemant) so not sure how that affects what I am trying to do. 

The follow up question, if the answer to the first one is yes it is
supported, is what am I doing wrong here?

Appreciate any pointers/help.

Thanks,
Sanjay.

class NehronWorker < BackgrounDRb::Rails
  #set_worker_name :nehron_worker
  def create
    puts "Creating NehronWorker"
  end
  def do_work(args)
    # This method is called in it's own new thread when you
    # call new worker. args is set to :args
    puts "Hello"
  end

  def printhelloinloop(teststr="Hello2")
      puts "Hello "+ teststr
  end
end

--- 
port: "22222"
timer_sleep: 60
load_rails: true
environment: development
host: localhost
database_yml: config/database.yml
acl: 
  deny: all
  allow: localhost 127.0.0.1
  order: deny,allow
  
schedules:
  NehronWorker:
      printhelloinloop:
        trigger_args: */5 * * * * * *

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080214/92cb486a/attachment.html 

From orion at mh2o.com  Fri Feb 15 11:12:55 2008
From: orion at mh2o.com (Orion Delwaterman)
Date: Fri, 15 Feb 2008 11:12:55 -0500
Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21, Issue 7
Message-ID: <205576B004DDBD438F6560241E0751A01E000045@sbsexc02.mh2onyc.com>

Hello All,

I am using backgroundrb from the trunk of devjavu and I am unable to
start the script with Capistrano.  This is really annoying because it
means I have to ssh into the server everytime I deploy something instead
of letting Capistrano do the work me.  

I am starting the script using the command 'ruby script/backgroundrb
start'.  When I execute via Capistrano the pid file is created but the
process does not exist.  There are no errors in any of the backgroundrb
logs.  When I execute directly on the server it works fine.  If I rub
backgroundrb console via Capistrano it works properly

So my question is when backgroundrb starts up where is it supposed to
write any errors it may encounter during startup.  Otherwise it seems
impossible to debug the issue.

Orion

From gfeil at realgirlsmedia.com  Fri Feb 15 12:24:03 2008
From: gfeil at realgirlsmedia.com (George Feil)
Date: Fri, 15 Feb 2008 09:24:03 -0800
Subject: [Backgroundrb-devel] monitoring the server
Message-ID: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>

First, some good news: after a few fits, starts, and configuration
tweaks, we've succeeded in deploying BackgrounDRb onto our production
machines, where we're using itto perform cache expiration on our Rails
generated pages. [We're using v1.0.1 code at the moment, but plan to
upgrade to 1.0.3 in the next week or two.] We use an event driven model
for expiration, which is triggered by observers in our Rails
application.

 

However, we're concerned about robustness, and would like to set up some
monitoring tools. In particular, we'd like to keep tabs on things like:

 

*	total number of threads allocated
*	number of active worker threads
*	duration of currently active threads (to help identify hung
ones)

 

Are there any good examples of monitor workers that do this now? What
sort of statistics API's are available? I didn't see anything obvious in
the RDoc for this.

 

Thanks,

George

 

 

George Feil | Senior Software Engineer

office 415.295.8524 | fax 415.295.8580

www.realgirlsmedia.com

www.divinecaroline.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080215/45a6b735/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 3238 bytes
Desc: image001.jpg
Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080215/45a6b735/attachment-0001.jpe 

From gfeil at realgirlsmedia.com  Fri Feb 15 12:25:15 2008
From: gfeil at realgirlsmedia.com (George Feil)
Date: Fri, 15 Feb 2008 09:25:15 -0800
Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21, Issue 7
In-Reply-To: <205576B004DDBD438F6560241E0751A01E000045@sbsexc02.mh2onyc.com>
References: <205576B004DDBD438F6560241E0751A01E000045@sbsexc02.mh2onyc.com>
Message-ID: <4BC9094459D5344490D03E5B38C3B89C017B9583@ex-be-005-sfo.shared.themessagecenter.com>

Did you look in the log/backgroundrb_server.log file? I've noticed that
exceptions thrown by the server often wind up there.
 
George

> -----Original Message-----
> From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-
> bounces at rubyforge.org] On Behalf Of Orion Delwaterman
> Sent: Friday, February 15, 2008 8:13 AM
> To: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21,
Issue
> 7
> 
> Hello All,
> 
> I am using backgroundrb from the trunk of devjavu and I am unable to
> start the script with Capistrano.  This is really annoying because it
> means I have to ssh into the server everytime I deploy something
instead
> of letting Capistrano do the work me.
> 
> I am starting the script using the command 'ruby script/backgroundrb
> start'.  When I execute via Capistrano the pid file is created but the
> process does not exist.  There are no errors in any of the
backgroundrb
> logs.  When I execute directly on the server it works fine.  If I rub
> backgroundrb console via Capistrano it works properly
> 
> So my question is when backgroundrb starts up where is it supposed to
> write any errors it may encounter during startup.  Otherwise it seems
> impossible to debug the issue.
> 
> Orion
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From gethemant at gmail.com  Fri Feb 15 12:48:07 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Feb 2008 23:18:07 +0530
Subject: [Backgroundrb-devel] BacgroundRB - Getting started question
In-Reply-To: <47b50f50.16078e0a.559f.ffff9dec@mx.google.com>
References: <47b50f50.16078e0a.559f.ffff9dec@mx.google.com>
Message-ID: <fb283d5e0802150948w4ced7acn8f0ab5eb58cd66be@mail.gmail.com>

On Fri, Feb 15, 2008 at 9:34 AM, Sanjay Tibrewal <stibrewal at gmail.com> wrote:
>
>
>
> Resending this message ? I sent it earlier in the day and I still don't see
> it show up on the mailing list.
>
> Thanks.
>
> -----------------------------------------
>
> Hello Folks,
>
> I have been trying to get backgroundrb running on my Win XP box to
> periodically wakeup and run a worker function. I wrote a test Worker class
> as below and my backgroundrb.yml is included below. I can start backgroundrb
> successfully through commondline (ruby script/backgroundrb/start) but I
> don't see any output in my backgroundrb.log ? I would expect the "Hello"
> string in the log every 5 secs.
>
> The first question I have is, whether what I am doing even supported on
> Windows? Reason I ask is because I saw a post from a few months back saying
> something to the effect that forking doesn't happen on Windows (a reply
> email from Hemant) so not sure how that affects what I am trying to do.
>
> The follow up question, if the answer to the first one is yes it is
> supported, is what am I doing wrong here?
>
> Appreciate any pointers/help.
>
> Thanks,
>
> Sanjay.
>
> class NehronWorker < BackgrounDRb::Rails
>
>   #set_worker_name :nehron_worker
>
>   def create
>
>     puts "Creating NehronWorker"
>
>   end
>
>   def do_work(args)
>
>     # This method is called in it's own new thread when you
>
>     # call new worker. args is set to :args
>
>     puts "Hello"
>
>   end
>
>   def printhelloinloop(teststr="Hello2")
>
>       puts "Hello "+ teststr
>
>   end
>
> end
>
> ---
>
> port: "22222"
>
> timer_sleep: 60
>
> load_rails: true
>
> environment: development
>
> host: localhost
>
> database_yml: config/database.yml
>
> acl:
>
>   deny: all
>
>   allow: localhost 127.0.0.1
>
>   order: deny,allow
>
>
>
> schedules:
>
>   NehronWorker:
>
>       printhelloinloop:
>
>         trigger_args: */5 * * * * * *


Any version of BackgrounDRb > 0.2 is not supported/won't run on
Windows.  It may change in future, but its not a priority.

From gethemant at gmail.com  Fri Feb 15 12:58:33 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Feb 2008 23:28:33 +0530
Subject: [Backgroundrb-devel] Getting started
In-Reply-To: <a242781c0802141741s10d8bac4x3b3e7b8bd9a37154@mail.gmail.com>
References: <a242781c0802121749r4060d2ecrb758247a16adfc8@mail.gmail.com>
	<fb283d5e0802121802w43982238mb7e77a78be2f33e8@mail.gmail.com>
	<a242781c0802122247r36f9d2b7kf4e26deca578e50f@mail.gmail.com>
	<1202895983.6779.4.camel@shire>
	<a242781c0802141445o38fd445cv6285bd9689551e6@mail.gmail.com>
	<fb283d5e0802141524r5184cd98w7bfe66d2da40c778@mail.gmail.com>
	<a242781c0802141741s10d8bac4x3b3e7b8bd9a37154@mail.gmail.com>
Message-ID: <fb283d5e0802150958k4c1b9985hdebd9fcc584137cd@mail.gmail.com>

On Fri, Feb 15, 2008 at 7:11 AM, Richard Everhart
<rich.everhart at gmail.com> wrote:
> Changing 'class' to 'worker' in new_worker seemed to get things going.
>   I can see that after new_worker is called the worker's create method
>  is then called.  This only seems to work once. After the first time, I
>  get the following error:
>
>  You have a nil object when you didn't expect it!
>  The error occurred while evaluating nil.send_request
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/packet_master.rb:43:in
>  `ask_worker'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:76:in
>  `process_work'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:34:in
>  `receive_data'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in
>  `call'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/lib/../framework/bin_parser.rb:29:in
>  `extract'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/server/master_worker.rb:30:in
>  `receive_data'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:195:in
>  `read_external_socket'
>  /home/reverhart/drb_test/vendor/plugins/backgroundrb/framework/core.rb:187:in
>  `handle_external_messages'
>  [...snip...]
>
>  Eventually, what I would really like to do is return a value back to
>  the browser.  send_request is documented to do this but the browser
>  just hangs and I really can't tell if the the worker method (do_work
>  in this is being executed).  While the work is being executed by the
>  worker I'd like to display a progress bar or a busy indicator, both of
>  which I've seen examples.  But, it seems that the do_work method is
>  never called, even when calling ask_work, which is not supposed to
>  block.
>
>  I have not tried out the latest version so I'm not sure if that will
>  change things.  Thanks a lot!
>

I wanted to ask, are you sure you want to create new workers on each
request? Can't you handle all the requests in the same instance of
worker. Also, send_request blocks until bdrb returns a response and
thats why your browser is hung.

A better approach is to use register_status in your worker and invoke
ask_status in rails code. Again, please read the README carefully from
start to beginning.

Also, use following links for debugging:

http://gnufied.org/2008/02/12/backgroundrb-best-practises/
http://www.johnyerhot.com/2008/02/11/debugging-backgroundrb/

From gethemant at gmail.com  Fri Feb 15 13:00:08 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 15 Feb 2008 23:30:08 +0530
Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21, Issue 7
In-Reply-To: <205576B004DDBD438F6560241E0751A01E000045@sbsexc02.mh2onyc.com>
References: <205576B004DDBD438F6560241E0751A01E000045@sbsexc02.mh2onyc.com>
Message-ID: <fb283d5e0802151000y6f157ca5o6f8726f2d93a5d7c@mail.gmail.com>

On Fri, Feb 15, 2008 at 9:42 PM, Orion Delwaterman <orion at mh2o.com> wrote:
> Hello All,
>
>  I am using backgroundrb from the trunk of devjavu and I am unable to
>  start the script with Capistrano.  This is really annoying because it
>  means I have to ssh into the server everytime I deploy something instead
>  of letting Capistrano do the work me.
>
>  I am starting the script using the command 'ruby script/backgroundrb
>  start'.  When I execute via Capistrano the pid file is created but the
>  process does not exist.  There are no errors in any of the backgroundrb
>  logs.  When I execute directly on the server it works fine.  If I rub
>  backgroundrb console via Capistrano it works properly
>
>  So my question is when backgroundrb starts up where is it supposed to
>  write any errors it may encounter during startup.  Otherwise it seems
>  impossible to debug the issue.
>

Hmm, I will have a look.

From orion at mh2o.com  Fri Feb 15 13:13:19 2008
From: orion at mh2o.com (Orion Delwaterman)
Date: Fri, 15 Feb 2008 13:13:19 -0500
Subject: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21, Issue 7
Message-ID: <205576B004DDBD438F6560241E0751A01E000048@sbsexc02.mh2onyc.com>

Yeah the exception appears there is I start the server from the server.
If use Capistrano it does not.

So the good news is that I found the exception, and I can now track down
the problem (something with working directories in another gem).  The
bad news is that when the server is started via Capistrano it does not
appear in the backgroundrb_server.log


-----Original Message-----
From: George Feil [mailto:gfeil at realgirlsmedia.com] 
Sent: Friday, February 15, 2008 12:25 PM
To: Orion Delwaterman; backgroundrb-devel at rubyforge.org
Subject: RE: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21,
Issue 7

Did you look in the log/backgroundrb_server.log file? I've noticed that
exceptions thrown by the server often wind up there.
 
George

> -----Original Message-----
> From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-
> bounces at rubyforge.org] On Behalf Of Orion Delwaterman
> Sent: Friday, February 15, 2008 8:13 AM
> To: backgroundrb-devel at rubyforge.org
> Subject: Re: [Backgroundrb-devel] Backgroundrb-devel Digest, Vol 21,
Issue
> 7
> 
> Hello All,
> 
> I am using backgroundrb from the trunk of devjavu and I am unable to
> start the script with Capistrano.  This is really annoying because it
> means I have to ssh into the server everytime I deploy something
instead
> of letting Capistrano do the work me.
> 
> I am starting the script using the command 'ruby script/backgroundrb
> start'.  When I execute via Capistrano the pid file is created but the
> process does not exist.  There are no errors in any of the
backgroundrb
> logs.  When I execute directly on the server it works fine.  If I rub
> backgroundrb console via Capistrano it works properly
> 
> So my question is when backgroundrb starts up where is it supposed to
> write any errors it may encounter during startup.  Otherwise it seems
> impossible to debug the issue.
> 
> Orion
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From apsoto at gmail.com  Sat Feb 16 00:46:21 2008
From: apsoto at gmail.com (Alex Soto)
Date: Fri, 15 Feb 2008 21:46:21 -0800
Subject: [Backgroundrb-devel] monitoring the server
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
References: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <864E2D90-EF0C-4F03-811A-C457E7A5F43B@gmail.com>

I don't know if there are any API's, but you could create a monitoring/ 
stats worker that exposes whatever data you can gather.  If you write  
one, please consider submitting it for inclusion in the project.

Alex

On Feb 15, 2008, at 9:24 AM, George Feil wrote:

> First, some good news: after a few fits, starts, and configuration  
> tweaks, we?ve succeeded in deploying BackgrounDRb onto our  
> production machines, where we?re using itto perform cache expiration  
> on our Rails generated pages. [We?re using v1.0.1 code at the  
> moment, but plan to upgrade to 1.0.3 in the next week or two.] We  
> use an event driven model for expiration, which is triggered by  
> observers in our Rails application.
>
> However, we?re concerned about robustness, and would like to set up  
> some monitoring tools. In particular, we?d like to keep tabs on  
> things like:
>
> total number of threads allocated
> number of active worker threads
> duration of currently active threads (to help identify hung ones)
>
> Are there any good examples of monitor workers that do this now?  
> What sort of statistics API?s are available? I didn?t see anything  
> obvious in the RDoc for this.
>
> Thanks,
> George
>
> <image001.jpg>
> George Feil | Senior Software Engineer
> office 415.295.8524 | fax 415.295.8580
> www.realgirlsmedia.com
> www.divinecaroline.com<_______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080215/8f042d34/attachment-0001.html 

From lists at sourceillustrated.com  Sat Feb 16 22:20:22 2008
From: lists at sourceillustrated.com (John Wells)
Date: Sat, 16 Feb 2008 22:20:22 -0500
Subject: [Backgroundrb-devel] monitoring the server
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
References: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <44dddf400802161920t2b6df0a2vafddf295a5d56fe9@mail.gmail.com>

On Feb 15, 2008 12:24 PM, George Feil <gfeil at realgirlsmedia.com> wrote:

> Are there any good examples of monitor workers that do this now? What sort
> of statistics API's are available? I didn't see anything obvious in the RDoc
> for this.
>
George,

I have been using god  (god.rubyforge.org) for this purpose with great
success.

Good luck.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080216/561f4dab/attachment.html 

From lists at sourceillustrated.com  Sun Feb 17 12:38:00 2008
From: lists at sourceillustrated.com (John Wells)
Date: Sun, 17 Feb 2008 12:38:00 -0500
Subject: [Backgroundrb-devel] Canceling work...
Message-ID: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>

Guys,

Is there any sort of mechanism for canceling workers? I've looked through
the docs and don't see any but may be overlooking.

Thanks!
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080217/7b15c6bd/attachment.html 

From gethemant at gmail.com  Mon Feb 18 08:45:49 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 18 Feb 2008 19:15:49 +0530
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
Message-ID: <fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>

On Sun, Feb 17, 2008 at 11:08 PM, John Wells
<lists at sourceillustrated.com> wrote:
> Guys,
>
> Is there any sort of mechanism for canceling workers? I've looked through
> the docs and don't see any but may be overlooking.
>

What you mean by canceling the job? You mean the one scheduled or one
from queue?

From gethemant at gmail.com  Mon Feb 18 08:47:13 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 18 Feb 2008 19:17:13 +0530
Subject: [Backgroundrb-devel] monitoring the server
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
References: <Achv94+NzoFpzvsZTt6kcUD+TnrIKg==>
	<4BC9094459D5344490D03E5B38C3B89C017B9582@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <fb283d5e0802180547v23efb751hab21435216a15ed4@mail.gmail.com>

2008/2/15 George Feil <gfeil at realgirlsmedia.com>:
>
>
>
>
> First, some good news: after a few fits, starts, and configuration tweaks,
> we've succeeded in deploying BackgrounDRb onto our production machines,
> where we're using itto perform cache expiration on our Rails generated
> pages. [We're using v1.0.1 code at the moment, but plan to upgrade to 1.0.3
> in the next week or two.] We use an event driven model for expiration, which
> is triggered by observers in our Rails application.
>
>
>
> However, we're concerned about robustness, and would like to set up some
> monitoring tools. In particular, we'd like to keep tabs on things like:
>
>
>
> total number of threads allocated
> number of active worker threads
> duration of currently active threads (to help identify hung ones)
>
>
>
> Are there any good examples of monitor workers that do this now? What sort
> of statistics API's are available? I didn't see anything obvious in the RDoc
> for this.
>
>

Its not there, right now. but should be pretty easy to implement.

From gethemant at gmail.com  Mon Feb 18 12:20:54 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 18 Feb 2008 22:50:54 +0530
Subject: [Backgroundrb-devel] BackgrounDRb prerelease for 1.0.3 version
In-Reply-To: <bf159cd50802180912h75fade6sd2e9e4fc593a7d81@mail.gmail.com>
References: <fb283d5e0802131145m28323b70ue739486d72b062a@mail.gmail.com>
	<bf159cd50802180912h75fade6sd2e9e4fc593a7d81@mail.gmail.com>
Message-ID: <fb283d5e0802180920x220b34e8q179d5c36dc2a31df@mail.gmail.com>

On Mon, Feb 18, 2008 at 10:42 PM, Pieter Noordhuis
<p.c.noordhuis at gmail.com> wrote:
> Hi Hemant,
>
>  I'm following the BDrb mailing list because I had issues with too much
>  connections being open. Therefore I'd like to checkout your new
>  version. Unfortunately, trying to access the git repository, I'm asked
>  for a password which I can't seem to locate anywhere. Could you open
>  up the repository for anonymous access?
>

I think I posted the URL thats only open to comitters, well the
anonymous url for checking out backgroundrb code from git is:

git clone git://gitorious.org/backgroundrb/mainline.git



>  Thanks.
>
>  Pieter Noordhuis
>
>
>
>  On Feb 13, 2008 8:45 PM, hemant <gethemant at gmail.com> wrote:
>  > Hullo there,
>  >
>  > I have worker out quite some changes as promised in last mail. Here is
>  > a brief ChangeLog:
>  >
>  >         * Added TestCases for Cron Triggers, Meta Workers and stuff. We are
>  > heading towards proper code coverage with specs.
>  >
>  >         * Added preliminary support for starting a worker on demand through
>  > scheduler. What it means is, when you have a worker which is getting
>  >         scheduled very less frequently and you don't want the worker to
>  > persist, you can ask BackgrounDRb to restart the worker on each
>  > schedule.
>  >
>  >         * Fixed some unreported issues with writing data to socket between
>  > workers and stuff.
>  >
>  >         * Fixed issues with too many open connections, because connections
>  > were not getting closed. BackgrounDRb now opens only one connection,
>  > which is
>  >         reused throughout the lifecycle of rails application.
>  >
>  >         * Fixed all outstanding issues with Cron triggers, BackgrounDRb now
>  > explicitly depends on "chronic" gem.
>  >
>  >         * Removed Framework directory and BackgrounDRb now explicitly depends
>  > on packet gem.
>  >
>  > To install:
>  >
>  > gem install packet chronic
>  >
>  > Checkout code from git repository:
>  >
>  > git clone git at gitorious.org:backgroundrb/mainline.git backgroundrb
>  >
>  > As usual, remove old backgroundrb script from script directory and
>  > start bdrb server.
>  >
>  >
>  > --
>  > Let them talk of their oriental summer climes of everlasting
>  > conservatories; give me the privilege of making my own summer with my
>  > own coals.
>  >
>  > http://gnufied.org
>  > _______________________________________________
>  > Backgroundrb-devel mailing list
>  > Backgroundrb-devel at rubyforge.org
>  > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>  >
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From tli at jinnius.com  Mon Feb 18 12:49:09 2008
From: tli at jinnius.com (Tony Li)
Date: Mon, 18 Feb 2008 09:49:09 -0800
Subject: [Backgroundrb-devel] Results and Logger Workers
Message-ID: <482eec2c0802180949y664147cey594779103225a7f3@mail.gmail.com>

Hi all,

What's the cleanest way to not run the results and logger workers?

Thanks,
Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080218/f844526d/attachment.html 

From lists at sourceillustrated.com  Mon Feb 18 12:59:05 2008
From: lists at sourceillustrated.com (John Wells)
Date: Mon, 18 Feb 2008 12:59:05 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
Message-ID: <44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>

On Feb 18, 2008 8:45 AM, hemant <gethemant at gmail.com> wrote:

> On Sun, Feb 17, 2008 at 11:08 PM, John Wells
> <lists at sourceillustrated.com> wrote:
> > Guys,
> >
> > Is there any sort of mechanism for canceling workers? I've looked
> through
> > the docs and don't see any but may be overlooking.
> >
>
> What you mean by canceling the job? You mean the one scheduled or one
> from queue?


Well, when I sent this, it was to cancel a job that was started on an
autostarted worker with ask_work. However, I've since changed the worker to
not autostart. I then do call:

# controller action to start
def start_work
  MiddleMan.new_worker(:worker=>"my_worker")
  MiddleMan.ask_work(:worker=>"my_worker", :worker_method=>"my_method")
end

# controller action to stop
def stop_work
  MiddleMan.delete_worker(:worker=>"my_worker")
end

This seems to work, but there are a few undesired things left dangling, as I
haven't found a way to "catch" the delete request and do a cleanup. Assuming
this is impossible, is there any way to clear the last status registered by
the worker with register_status if that worker is deleted?

And, is the above the proper way of cancelling work?

Thanks very much for your help!

John

<http://www.greatworx.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080218/4d0b8380/attachment.html 

From noah at rapouts.com  Mon Feb 18 21:00:01 2008
From: noah at rapouts.com (Noah Horton)
Date: Mon, 18 Feb 2008 18:00:01 -0800
Subject: [Backgroundrb-devel] Set logger level
Message-ID: <C3DF7821.A3F4%noah@rapouts.com>

I can't find an example of setting the log level for the backgroundrb logger.  Anyone have a quick example?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080218/c3e17990/attachment.html 

From gethemant at gmail.com  Tue Feb 19 09:36:34 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 19 Feb 2008 20:06:34 +0530
Subject: [Backgroundrb-devel] Results and Logger Workers
In-Reply-To: <482eec2c0802180949y664147cey594779103225a7f3@mail.gmail.com>
References: <482eec2c0802180949y664147cey594779103225a7f3@mail.gmail.com>
Message-ID: <fb283d5e0802190636x2a97c86dtb9ee05cf277a161f@mail.gmail.com>

On Mon, Feb 18, 2008 at 11:19 PM, Tony Li <tli at jinnius.com> wrote:
> Hi all,
>
> What's the cleanest way to not run the results and logger workers?
>

There is no result worker, if you are running the latest version of
plugin. However, if you want to disable logger worker, then you can
simply edit master_worker.rb file and remove the line where logger
worker is being started. Probably in future versions, this can be made
configurable.

From gethemant at gmail.com  Tue Feb 19 09:38:09 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 19 Feb 2008 20:08:09 +0530
Subject: [Backgroundrb-devel] Set logger level
In-Reply-To: <C3DF7821.A3F4%noah@rapouts.com>
References: <C3DF7821.A3F4%noah@rapouts.com>
Message-ID: <fb283d5e0802190638te596b20q748c50b141e5c641@mail.gmail.com>

On Tue, Feb 19, 2008 at 7:30 AM, Noah Horton <noah at rapouts.com> wrote:
>
>  I can't find an example of setting the log level for the backgroundrb
> logger.  Anyone have a quick example?

Hmm, I think this feature is not there. Can you file a ticket on trac
and we will make sure it gets done.

From heisters at greenriver.org  Wed Feb 20 12:41:51 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Wed, 20 Feb 2008 09:41:51 -0800
Subject: [Backgroundrb-devel] query queued jobs
Message-ID: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>

Hi all,

I don't have any status logging setup for my workers, but some of them
have turned out to be rather longer jobs than I'd anticipated (like 24
hours and still going). So I was wondering if there's any way to query
the server for a list of queued jobs to see how much backup there is.
I'd think this could be independent of actual worker statuses, since
the server must be holding a queue regardless of the internals of the
workers themselves. FWIW, I've been queuing all my workers with
#ask_work.

Thanks,
Ian

From gfeil at realgirlsmedia.com  Wed Feb 20 12:58:13 2008
From: gfeil at realgirlsmedia.com (George Feil)
Date: Wed, 20 Feb 2008 09:58:13 -0800
Subject: [Backgroundrb-devel] query queued jobs
In-Reply-To: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
Message-ID: <4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>

Your timing is auspicious. I just coded up a status method for the cache
clearing worker I created for my company's CMS app. I received the green
light from my manager to donate the code to the project; however, I
still have a bit testing and perfecting to do. Expect a check-in by the
end of the month.

George Feil | Senior Software Engineer
office 415.295.8524 | fax 415.295.8580
www.realgirlsmedia.com
www.divinecaroline.com

> -----Original Message-----
> From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-
> bounces at rubyforge.org] On Behalf Of Ian Smith-Heisters
> Sent: Wednesday, February 20, 2008 9:42 AM
> To: backgroundrb-devel at rubyforge.org
> Subject: [Backgroundrb-devel] query queued jobs
> 
> Hi all,
> 
> I don't have any status logging setup for my workers, but some of them
> have turned out to be rather longer jobs than I'd anticipated (like 24
> hours and still going). So I was wondering if there's any way to query
> the server for a list of queued jobs to see how much backup there is.
> I'd think this could be independent of actual worker statuses, since
> the server must be holding a queue regardless of the internals of the
> workers themselves. FWIW, I've been queuing all my workers with
> #ask_work.
> 
> Thanks,
> Ian
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From lists at sourceillustrated.com  Wed Feb 20 13:20:08 2008
From: lists at sourceillustrated.com (John Wells)
Date: Wed, 20 Feb 2008 13:20:08 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
Message-ID: <44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>

On Feb 18, 2008 12:59 PM, John Wells <lists at sourceillustrated.com> wrote:

> This seems to work, but there are a few undesired things left dangling, as
> I haven't found a way to "catch" the delete request and do a cleanup.
> Assuming this is impossible, is there any way to clear the last status
> registered by the worker with register_status if that worker is deleted?
>
> And, is the above the proper way of cancelling work?
>
> Thanks very much for your help!
>

Sorry to bump my own message, but I'm still very interested in learning if
I'm approaching the above properly, and if there's a way to clear a worker's
status when that worker is deleted.

Thanks!
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080220/378d5470/attachment.html 

From heisters at greenriver.org  Wed Feb 20 13:21:10 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Wed, 20 Feb 2008 10:21:10 -0800
Subject: [Backgroundrb-devel] query queued jobs
In-Reply-To: <4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
Message-ID: <de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>

Cool, great to hear. Maybe my queue will have gotten through this job by then ;)

On 2/20/08, George Feil <gfeil at realgirlsmedia.com> wrote:
> Your timing is auspicious. I just coded up a status method for the cache
>  clearing worker I created for my company's CMS app. I received the green
>  light from my manager to donate the code to the project; however, I
>  still have a bit testing and perfecting to do. Expect a check-in by the
>  end of the month.
>
>  George Feil | Senior Software Engineer
>  office 415.295.8524 | fax 415.295.8580
>  www.realgirlsmedia.com
>  www.divinecaroline.com
>
>
>  > -----Original Message-----
>  > From: backgroundrb-devel-bounces at rubyforge.org
>  [mailto:backgroundrb-devel-
>  > bounces at rubyforge.org] On Behalf Of Ian Smith-Heisters
>  > Sent: Wednesday, February 20, 2008 9:42 AM
>  > To: backgroundrb-devel at rubyforge.org
>  > Subject: [Backgroundrb-devel] query queued jobs
>  >
>  > Hi all,
>  >
>  > I don't have any status logging setup for my workers, but some of them
>  > have turned out to be rather longer jobs than I'd anticipated (like 24
>  > hours and still going). So I was wondering if there's any way to query
>  > the server for a list of queued jobs to see how much backup there is.
>  > I'd think this could be independent of actual worker statuses, since
>  > the server must be holding a queue regardless of the internals of the
>  > workers themselves. FWIW, I've been queuing all my workers with
>  > #ask_work.
>  >
>  > Thanks,
>  > Ian
>
> > _______________________________________________
>  > Backgroundrb-devel mailing list
>  > Backgroundrb-devel at rubyforge.org
>  > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>

From gethemant at gmail.com  Wed Feb 20 14:22:45 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 21 Feb 2008 00:52:45 +0530
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
Message-ID: <fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>

On Wed, Feb 20, 2008 at 11:50 PM, John Wells
<lists at sourceillustrated.com> wrote:
>
>
>
> On Feb 18, 2008 12:59 PM, John Wells <lists at sourceillustrated.com> wrote:
> >
> >
> > This seems to work, but there are a few undesired things left dangling, as
> I haven't found a way to "catch" the delete request and do a cleanup.
> Assuming this is impossible, is there any way to clear the last status
> registered by the worker with register_status if that worker is deleted?
> >
> > And, is the above the proper way of cancelling work?
> >
> > Thanks very much for your help!
> >
>
> Sorry to bump my own message, but I'm still very interested in learning if
> I'm approaching the above properly, and if there's a way to clear a worker's
> status when that worker is deleted.
>

Hmm, No problems. The way you are deleting/canceling running workers
is correct. But apparently, you want a callback, which
should be invoked when worker dies, so you can cleanup stuff.

Now, unfortunately this means that like all requests, request to kill
a worker must be handled by the worker itself, but
if a worker is going to handle a request to kill itself ( in another
words calls "exit" from inside worker code ), then when you invoke:

MiddleMan.delete_worker

and worker is doing some processing then worker won't handle
delete_worker request until its done with processing whatever its
doing.
That was how "delete_worker" used to work earlier. I mean when you
invoke delete_worker it used to invoke "exit" method on worker and
worker used to exit as usual.

But later we changed the code and we thought when you invoke
delete_worker, your worker should exit immediately and request to kill
a worker shouldn't be queued.

In current code, apart from executing "delete_worker" another way to
kill workers is to call "exit" from inside of a worker. This will give
you opportunity to do proper cleanup and stuff.
I am not sure, if this is what you want, but "delete_worker" is like
"kill -9 " of unix.

So, if you want to handle graceful exit, call exit from worker code itself.

From gethemant at gmail.com  Wed Feb 20 14:26:26 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 21 Feb 2008 00:56:26 +0530
Subject: [Backgroundrb-devel] query queued jobs
In-Reply-To: <de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>
Message-ID: <fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>

Currently, we only have all_worker_info and worker_info for queering
runtime information of workers.
I think, we can pimp up information provided by these methods.



On Wed, Feb 20, 2008 at 11:51 PM, Ian Smith-Heisters
<heisters at greenriver.org> wrote:
> Cool, great to hear. Maybe my queue will have gotten through this job by then ;)
>
>
>
>  On 2/20/08, George Feil <gfeil at realgirlsmedia.com> wrote:
>  > Your timing is auspicious. I just coded up a status method for the cache
>  >  clearing worker I created for my company's CMS app. I received the green
>  >  light from my manager to donate the code to the project; however, I
>  >  still have a bit testing and perfecting to do. Expect a check-in by the
>  >  end of the month.

Thats cool. Looking forward.

From lists at sourceillustrated.com  Wed Feb 20 14:32:22 2008
From: lists at sourceillustrated.com (John Wells)
Date: Wed, 20 Feb 2008 14:32:22 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
Message-ID: <44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>

On Feb 20, 2008 2:22 PM, hemant <gethemant at gmail.com> wrote:

> But later we changed the code and we thought when you invoke
> delete_worker, your worker should exit immediately and request to kill
> a worker shouldn't be queued.
>
> In current code, apart from executing "delete_worker" another way to
> kill workers is to call "exit" from inside of a worker. This will give
> you opportunity to do proper cleanup and stuff.
> I am not sure, if this is what you want, but "delete_worker" is like
> "kill -9 " of unix.
>
> So, if you want to handle graceful exit, call exit from worker code
> itself.
>

Thanks for your reply.

So how would I be able to signal to the worker that it should call exit? I
guess that is the challenge. The worker is involved in a large batch job,
and I need to be able to send a message to it to tell it to kill itself. I
had hacked out an approach using both a database table in which it would
check for the existence of a certain row, and if there remove and kill
itself, but I wondered if the framework itself provided a better way?

Also, assuming I just continue to use delete_worker in the way I am today,
is there any way possible to delete the last status the worker reported?

Thanks for your help!
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080220/9604e712/attachment.html 

From gethemant at gmail.com  Wed Feb 20 14:39:19 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 21 Feb 2008 01:09:19 +0530
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
Message-ID: <fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>

On Thu, Feb 21, 2008 at 1:02 AM, John Wells <lists at sourceillustrated.com> wrote:
>
>
> On Feb 20, 2008 2:22 PM, hemant <gethemant at gmail.com> wrote:
> > But later we changed the code and we thought when you invoke
> > delete_worker, your worker should exit immediately and request to kill
> > a worker shouldn't be queued.
> >
> > In current code, apart from executing "delete_worker" another way to
> > kill workers is to call "exit" from inside of a worker. This will give
> > you opportunity to do proper cleanup and stuff.
> > I am not sure, if this is what you want, but "delete_worker" is like
> > "kill -9 " of unix.
> >
> > So, if you want to handle graceful exit, call exit from worker code
> itself.
> >
>
> Thanks for your reply.
>
> So how would I be able to signal to the worker that it should call exit? I
> guess that is the challenge. The worker is involved in a large batch job,
> and I need to be able to send a message to it to tell it to kill itself. I
> had hacked out an approach using both a database table in which it would
> check for the existence of a certain row, and if there remove and kill
> itself, but I wondered if the framework itself provided a better way?
>
> Also, assuming I just continue to use delete_worker in the way I am today,
> is there any way possible to delete the last status the worker reported?
>

Something like this:

class FooWorker < BackgrounDRb::MetaWorker
    def some_work(args = nil)
       # do some job processing from database
       exit
     end
end

From lists at sourceillustrated.com  Wed Feb 20 14:51:05 2008
From: lists at sourceillustrated.com (John Wells)
Date: Wed, 20 Feb 2008 14:51:05 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
Message-ID: <44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>

On Feb 20, 2008 2:39 PM, hemant <gethemant at gmail.com> wrote:

> On Thu, Feb 21, 2008 at 1:02 AM, John Wells <lists at sourceillustrated.com>
> wrote:
> >
> >
> > On Feb 20, 2008 2:22 PM, hemant <gethemant at gmail.com> wrote:
> > > But later we changed the code and we thought when you invoke
> > > delete_worker, your worker should exit immediately and request to kill
> > > a worker shouldn't be queued.
> > >
> > > In current code, apart from executing "delete_worker" another way to
> > > kill workers is to call "exit" from inside of a worker. This will give
> > > you opportunity to do proper cleanup and stuff.
> > > I am not sure, if this is what you want, but "delete_worker" is like
> > > "kill -9 " of unix.
> > >
> > > So, if you want to handle graceful exit, call exit from worker code
> > itself.
> > >
> >
> > Thanks for your reply.
> >
> > So how would I be able to signal to the worker that it should call exit?
> I
> > guess that is the challenge. The worker is involved in a large batch
> job,
> > and I need to be able to send a message to it to tell it to kill itself.
> I
> > had hacked out an approach using both a database table in which it would
> > check for the existence of a certain row, and if there remove and kill
> > itself, but I wondered if the framework itself provided a better way?
> >
> > Also, assuming I just continue to use delete_worker in the way I am
> today,
> > is there any way possible to delete the last status the worker reported?
> >
>
> Something like this:
>
> class FooWorker < BackgrounDRb::MetaWorker
>    def some_work(args = nil)
>       # do some job processing from database
>       exit
>     end
> end



Ok, but when would exit be called here? After all the work is done?

I guess what I want is a kill -9 functionality, but with the ability to at
the very least clear the worker's status...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080220/2526ffd7/attachment-0001.html 

From yem_backgroundrb-devel at filter.yve.net  Wed Feb 20 18:36:38 2008
From: yem_backgroundrb-devel at filter.yve.net (Yves-Eric Martin)
Date: Thu, 21 Feb 2008 08:36:38 +0900
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
Message-ID: <47BCB986.7070202@filter.yve.net>

Hi all,


Just and idea:

The normal way of doing cleanup in Ruby is to use a "begin...ensure" 
construct.
Maybe that will work here too? Try something like this:


class FooWorker < BackgrounDRb::MetaWorker
  def some_work(args=nil)
    begin
      # Large batch job here
      ...
    rescue # (optional)
      # Error processing here
      ...
    ensure
      # Cleanup code here
      ...
    end
  end
end


Then, just invoke delete_worker when you need to, and if all is well, it 
should interrupt the running batch job, but your cleanup code *should* 
get executed before the worker finally dies. If it is not, I'd call it a 
BackgrounDRb bug, as it would be breaking some basic Ruby functionality.

Let us know how it goes.


Cheers,

-- 
Yves-Eric Martin

John Wells wrote:
>
>     class FooWorker < BackgrounDRb::MetaWorker
>        def some_work(args = nil)
>           # do some job processing from database
>           exit
>         end
>     end
>
>
>
> Ok, but when would exit be called here? After all the work is done?
>
> I guess what I want is a kill -9 functionality, but with the ability 
> to at the very least clear the worker's status...
> ------------------------------------------------------------------------
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From gtcampbell at gmail.com  Wed Feb 20 20:03:19 2008
From: gtcampbell at gmail.com (Greg Campbell)
Date: Wed, 20 Feb 2008 17:03:19 -0800
Subject: [Backgroundrb-devel] Registering status for multithreaded worker?
Message-ID: <ac6faba20802201703y365c33c1gb9904894c76336d9@mail.gmail.com>

Hi all,

I'm using a backgroundrb worker for processing data reporting tasks which
can be initiated by users of my rails application, and which need to support
status monitoring.  I had been spawning a new instance with a new job_id for
each task, and reporting/requesting status via that job_id.  It appears that
this sort of thing may be better handled by thread_pool, but there seem to
be two ways of dealing with status reporting, and I'm curious whether people
have found one to be preferable over the other:

I could track status in the database, as I'm creating a new row for each
task anyway to store the results, or I could use register_status, with a
hash keyed on the equivalent of job_id (inside a mutex, as suggested in the
README).  Is there any reason to prefer the second over the first?
Alternately, am I incorrect in assuming that thread_pool is preferable to
spawning one worker per user request?

Thanks,
Greg Campbell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080220/b56af37e/attachment.html 

From heisters at greenriver.org  Wed Feb 20 21:45:24 2008
From: heisters at greenriver.org (Ian Smith-Heisters)
Date: Wed, 20 Feb 2008 18:45:24 -0800
Subject: [Backgroundrb-devel] query queued jobs
In-Reply-To: <fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>
	<fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>
Message-ID: <de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>

ah, cool, all_worker_info looks like it'll help out. This doesn't
appear to be mentioned anywhere on http://backgroundrb.rubyforge.org/

Thanks!

On 2/20/08, hemant <gethemant at gmail.com> wrote:
> Currently, we only have all_worker_info and worker_info for queering
>  runtime information of workers.
>  I think, we can pimp up information provided by these methods.
>
>
>
>
>  On Wed, Feb 20, 2008 at 11:51 PM, Ian Smith-Heisters
>  <heisters at greenriver.org> wrote:
>  > Cool, great to hear. Maybe my queue will have gotten through this job by then ;)
>  >
>  >
>  >
>  >  On 2/20/08, George Feil <gfeil at realgirlsmedia.com> wrote:
>  >  > Your timing is auspicious. I just coded up a status method for the cache
>  >  >  clearing worker I created for my company's CMS app. I received the green
>  >  >  light from my manager to donate the code to the project; however, I
>  >  >  still have a bit testing and perfecting to do. Expect a check-in by the
>  >  >  end of the month.
>
>
> Thats cool. Looking forward.
>

From tli at jinnius.com  Thu Feb 21 01:16:52 2008
From: tli at jinnius.com (Tony Li)
Date: Wed, 20 Feb 2008 22:16:52 -0800
Subject: [Backgroundrb-devel] Results and Logger Workers
In-Reply-To: <fb283d5e0802190636x2a97c86dtb9ee05cf277a161f@mail.gmail.com>
References: <482eec2c0802180949y664147cey594779103225a7f3@mail.gmail.com>
	<fb283d5e0802190636x2a97c86dtb9ee05cf277a161f@mail.gmail.com>
Message-ID: <482eec2c0802202216n14df4e69mdcdc87dbc249c748@mail.gmail.com>

Thanks for the response.  However, when I remove the line
"t_reactor.start_worker(:worker => :log_worker)", I see this in
backgroundrb_server.log on startup:

<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/meta_pimp.rb:48:in
`process_
request': undefined method `send_request' for nil:NilClass (NoMethodError)
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/meta_pimp.rb:27
:in `handle_object'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/meta_pimp.rb:20
:in `receive_data'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/lib/../framework/packet/bin_pars
er.rb:37:in `call'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/lib/../framework/packet/bin_pars
er.rb:37:in `extract'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/meta_pimp.rb:18
:in `receive_data'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:63:in `handle_internal_messages'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:158:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:156:in
`each'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:156:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:147:in
`loop'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:147:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:20:in `run'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/server/master_worker.rb:163:in `
initialize'
        from <rails_app_dir>/script/backgroundrb:38:in `new'
        from <rails_app_dir>/script/backgroundrb:38
<rails_app_dir>/vendor/plugins/backgroundrb/lib/../framework/packet/nbio.rb:23:in
`read_d
ata': Packet::DisconnectError (Packet::DisconnectError)
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/worker.rb:47:in
 `handle_internal_messages'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:158:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:156:in
`each'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:156:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:147:in
`loop'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/core.rb:147:in
`start_reactor'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/worker.rb:21:in
 `start_worker'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:139:in `fork_and_load'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:98:in `load_workers'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:93:in `each'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:93:in `load_workers'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/framework/packet/packet_master.r
b:19:in `run'
        from
<rails_app_dir>/vendor/plugins/backgroundrb/server/master_worker.rb:163:in `
initialize'
        from <rails_app_dir>/script/backgroundrb:38:in `new'
        from <rails_app_dir>/script/backgroundrb:38

On Tue, Feb 19, 2008 at 6:36 AM, hemant <gethemant at gmail.com> wrote:

> On Mon, Feb 18, 2008 at 11:19 PM, Tony Li <tli at jinnius.com> wrote:
> > Hi all,
> >
> > What's the cleanest way to not run the results and logger workers?
> >
>
> There is no result worker, if you are running the latest version of
> plugin. However, if you want to disable logger worker, then you can
> simply edit master_worker.rb file and remove the line where logger
> worker is being started. Probably in future versions, this can be made
> configurable.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080220/7e72f2bc/attachment.html 

From rubyonrails-ug at peuchert.de  Thu Feb 21 04:00:41 2008
From: rubyonrails-ug at peuchert.de (Alex Peuchert)
Date: Thu, 21 Feb 2008 10:00:41 +0100
Subject: [Backgroundrb-devel] [PATCH] allow different environments and move
 config into seperate class
In-Reply-To: <de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>	<fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>
	<de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>
Message-ID: <47BD3DB9.1050507@peuchert.de>

Hi, I had the problem of running different BackgrounDRB processes with 
different environments on the same box. BackgrounDRB does not support 
that, so I added a little bit of code. I opened ticket #80 on devjavu 
with the patch attached.

* script/backgroundrb now supports cli parameters

    * -h for help
    * -e to choose an environment

* moved configuration handling into lib/bdrb_config.rb

    if there is an entry in backgroundrb.yml with the environment name
    that entry is merged back into the config, eg:

---
:development:
  :backgroundrb:
    :port: 11111
:production:
  :backgroundrb:
    :port: 11113

:backgroundrb:
  :ip: localhost
  :port: 11006

        if you run this under development port will be 11111, under
        production port will be 11113 (this is pretty cool, you can add
        config instructions depending on the environment, even schedules:)

* the PID files goes into tmp/pids/

* PID and LOG file names now include the port for differentiation

Hope this improves BackgrounDRB a little bit.
-alex


From gethemant at gmail.com  Thu Feb 21 06:50:55 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 21 Feb 2008 17:20:55 +0530
Subject: [Backgroundrb-devel] [PATCH] allow different environments and
	move config into seperate class
In-Reply-To: <47BD3DB9.1050507@peuchert.de>
References: <de819a5b0802200941i2ecde1ffmd3e7a644c5ef7fd9@mail.gmail.com>
	<4BC9094459D5344490D03E5B38C3B89C017B9B2B@ex-be-005-sfo.shared.themessagecenter.com>
	<de819a5b0802201021i2959a432pd9e37a351fb2e5d9@mail.gmail.com>
	<fb283d5e0802201126w30ced88br71051174d629895c@mail.gmail.com>
	<de819a5b0802201845t37071a31s35d231b4f09d8594@mail.gmail.com>
	<47BD3DB9.1050507@peuchert.de>
Message-ID: <fb283d5e0802210350g4e868242uf3a02603ca953118@mail.gmail.com>

On Thu, Feb 21, 2008 at 2:30 PM, Alex Peuchert
<rubyonrails-ug at peuchert.de> wrote:
> Hi, I had the problem of running different BackgrounDRB processes with
>  different environments on the same box. BackgrounDRB does not support
>  that, so I added a little bit of code. I opened ticket #80 on devjavu
>  with the patch attached.
>
>  * script/backgroundrb now supports cli parameters
>
>     * -h for help
>     * -e to choose an environment
>
>  * moved configuration handling into lib/bdrb_config.rb
>
>     if there is an entry in backgroundrb.yml with the environment name
>     that entry is merged back into the config, eg:
>
>  ---
>  :development:
>   :backgroundrb:
>     :port: 11111
>  :production:
>   :backgroundrb:
>     :port: 11113
>
>  :backgroundrb:
>   :ip: localhost
>   :port: 11006
>
>         if you run this under development port will be 11111, under
>         production port will be 11113 (this is pretty cool, you can add
>         config instructions depending on the environment, even schedules:)
>
>  * the PID files goes into tmp/pids/
>
>  * PID and LOG file names now include the port for differentiation
>
>  Hope this improves BackgrounDRB a little bit.


Awesome. I am having a look and commit them accordingly. Also, as a
rule, you are qualified for commit access, so create an account on
gitorious and let me know your login handle.

http://gitorious.org/projects/backgroundrb

From gethemant at gmail.com  Thu Feb 21 06:54:53 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 21 Feb 2008 17:24:53 +0530
Subject: [Backgroundrb-devel] Registering status for multithreaded
	worker?
In-Reply-To: <ac6faba20802201703y365c33c1gb9904894c76336d9@mail.gmail.com>
References: <ac6faba20802201703y365c33c1gb9904894c76336d9@mail.gmail.com>
Message-ID: <fb283d5e0802210354u7946394frb3a49c6f4d3b473e@mail.gmail.com>

Hi Greg,

On Thu, Feb 21, 2008 at 6:33 AM, Greg Campbell <gtcampbell at gmail.com> wrote:
> Hi all,
>
> I'm using a backgroundrb worker for processing data reporting tasks which
> can be initiated by users of my rails application, and which need to support
> status monitoring.  I had been spawning a new instance with a new job_id for
> each task, and reporting/requesting status via that job_id.  It appears that
> this sort of thing may be better handled by thread_pool, but there seem to
> be two ways of dealing with status reporting, and I'm curious whether people
> have found one to be preferable over the other:
>
> I could track status in the database, as I'm creating a new row for each
> task anyway to store the results, or I could use register_status, with a
> hash keyed on the equivalent of job_id (inside a mutex, as suggested in the
> README).  Is there any reason to prefer the second over the first?
> Alternately, am I incorrect in assuming that thread_pool is preferable to
> spawning one worker per user request?
>

thread_pool is definitely preferable over one worker per request approach.

Also, usually register_status is faster than your hand rolled approach
of using databases. Also, if worker status results can be stored in
memcache clusters as well hence is preferable.

From lists at sourceillustrated.com  Thu Feb 21 10:23:56 2008
From: lists at sourceillustrated.com (John Wells)
Date: Thu, 21 Feb 2008 10:23:56 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <47BCB986.7070202@filter.yve.net>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<47BCB986.7070202@filter.yve.net>
Message-ID: <44dddf400802210723n4395f53qd502b3705913e884@mail.gmail.com>

On Wed, Feb 20, 2008 at 6:36 PM, Yves-Eric Martin <
yem_backgroundrb-devel at filter.yve.net> wrote:

> Then, just invoke delete_worker when you need to, and if all is well, it
> should interrupt the running batch job, but your cleanup code *should*
> get executed before the worker finally dies. If it is not, I'd call it a
> BackgrounDRb bug, as it would be breaking some basic Ruby functionality.
>

Thanks! It's a good idea, but unfortunately, I only want cleanup to happen
when the worker is forceably deleted...not when it exits on its own. Ensure
would cleanup in both cases.

Thanks for your suggestion regardless.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080221/141dc389/attachment.html 

From yem_backgroundrb-devel at filter.yve.net  Thu Feb 21 19:59:10 2008
From: yem_backgroundrb-devel at filter.yve.net (Yves-Eric Martin)
Date: Fri, 22 Feb 2008 09:59:10 +0900
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802210723n4395f53qd502b3705913e884@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>	
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>	
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>	
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>	
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>	
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>	
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>	
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>	
	<47BCB986.7070202@filter.yve.net>
	<44dddf400802210723n4395f53qd502b3705913e884@mail.gmail.com>
Message-ID: <47BE1E5E.1010904@filter.yve.net>

John Wells wrote:
> On Wed, Feb 20, 2008 at 6:36 PM, Yves-Eric Martin 
> <yem_backgroundrb-devel at filter.yve.net 
> <mailto:yem_backgroundrb-devel at filter.yve.net>> wrote:
>
> Thanks! It's a good idea, but unfortunately, I only want cleanup to 
> happen when the worker is forceably deleted...not when it exits on its 
> own. Ensure would cleanup in both cases.

Well that one is easily taken care of: just use a local variable to 
indicate whether the processing was interrupted or went to completion. 
You would probably simply use a boolean or constant integers to indicate 
the exit status, but I'll write it here with a string for simplicity:


class FooWorker < BackgrounDRb::MetaWorker
  def some_work(args=nil)
    job_exit_status = 'FORCEABLY_DELETED'
    begin
      # Large batch job here
      ...
    rescue
      job_exit_status = 'ERRORED_OUT'
      # Error processing here
      ...
    else
      job_exit_status = 'FINISHED_NORMALLY'
    ensure
      case job_exit_status
      when 'FINISHED_NORMALLY'
        # Your normal ending cleanup code here
        ...
      when 'ERRORED_OUT'
        # Your error ending cleanup code here
        ...
      else
        # Your "delete_worker" ending cleanup code here
        ...
      end
    end
  end
end


Of course this still all assumes that delete_worker is *not* the "kill 
-9" hemant suggested, and that it will let your ensure clause run. Have 
you tried it out yet? Let us know how it goes.


Cheers,

-- 
Yves-Eric


From todd at snappl.co.uk  Fri Feb 22 09:53:25 2008
From: todd at snappl.co.uk (Todd Tyree)
Date: Fri, 22 Feb 2008 14:53:25 +0000
Subject: [Backgroundrb-devel] ask_status returning 'running' even when the
	process has been killed
Message-ID: <44f20a950802220653w611aa5dei6e69b927479daad2@mail.gmail.com>

Just wondering if anyone else was seeing behaviour like this?

I've got a long running worker used to allow a cluster of mongrels to talk
to an xmpp server over a single connection using xmpp4r.  I've seen the
worker die a couple of times, so I set up a worker_alive? method that looks
like this:

def worker_alive?
  MiddleMan.ask_status(:worker => :xmpp_worker) == 'Running.'
end

The problem I'm having is that ask_status always returns 'Running', even if
the worker is dead and gone, so I can never get it to restart this way.
I've confirmed the worker is dead via top, ps and the xmpp admin console.

I've had a brief look at the ask_status code, but don't see anything
obvious.

MiddleMan#query_all_workers returns the correct information, however.

I'm running r315 of BDrb on OSX with 2.0.2.

Any thoughts or suggestions would be appreciated.

Cheers,
Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080222/0790c0ce/attachment.html 

From leavengood at gmail.com  Fri Feb 22 10:16:45 2008
From: leavengood at gmail.com (Ryan Leavengood)
Date: Fri, 22 Feb 2008 10:16:45 -0500
Subject: [Backgroundrb-devel] ask_status returning 'running' even when
	the process has been killed
In-Reply-To: <44f20a950802220653w611aa5dei6e69b927479daad2@mail.gmail.com>
References: <44f20a950802220653w611aa5dei6e69b927479daad2@mail.gmail.com>
Message-ID: <a4855ac50802220716t617deb7cid8533ee2e57f2652@mail.gmail.com>

On Fri, Feb 22, 2008 at 9:53 AM, Todd Tyree <todd at snappl.co.uk> wrote:
>
> The problem I'm having is that ask_status always returns 'Running', even if
> the worker is dead and gone, so I can never get it to restart this way.
> I've confirmed the worker is dead via top, ps and the xmpp admin console.

The master worker keeps the status for workers even after they are
dead, because in most cases it is useful to be able to query a
worker's status once it has died.

This is a little clumsy, but you can write your method like this instead:

def worker_alive?
  !(MiddleMan.all_worker_info.detect{|worker| worker[:status] ==
:running && worker[:worker] == :xmpp_worker }.nil?)
end

As far as I can see all_worker_info only returns currently running
workers, so probably the check for :status == :running is redundant.

Ryan

From lists at sourceillustrated.com  Sat Feb 23 16:31:00 2008
From: lists at sourceillustrated.com (John Wells)
Date: Sat, 23 Feb 2008 16:31:00 -0500
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
Message-ID: <44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>

On Wed, Feb 20, 2008 at 2:51 PM, John Wells <lists at sourceillustrated.com> wrote:
> Ok, but when would exit be called here? After all the work is done?
>
> I guess what I want is a kill -9 functionality, but with the ability to at
> the very least clear the worker's status...

Hemant,

Would you mind answering this? I'm just getting going again after a
nasty bout of the flu, but very interested in your answer.

Thanks!
John

From yem_backgroundrb-devel at filter.yve.net  Sat Feb 23 20:21:41 2008
From: yem_backgroundrb-devel at filter.yve.net (Yves-Eric Martin)
Date: Sun, 24 Feb 2008 10:21:41 +0900
Subject: [Backgroundrb-devel] Bug report: delete_worker breaks
 begin...ensure (was Re: Canceling work...)
In-Reply-To: <44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
Message-ID: <47C0C6A5.2010102@filter.yve.net>

Hi all, and welcome back John.


John, you may have missed my second reply to you:
http://rubyforge.org/pipermail/backgroundrb-devel/2008-February/001514.html

Anyway, the bad news is: I just tried it out and it does not seem to 
work. delete_worker *is* the "kill -9" hemant suggested.

Hemant, this is breaking some basic Ruby functionality, namely 
"begin...ensure". So can we please call this a bug? delete_worker should 
be a "kill", not a "kill -9", or at least not by default. We could add 
an argument to determine which behavior we want:

# normal "kill" behavior:
MiddleMan.delete_worker(:worker => :background_worker)

# brutal "kill -9" behavior:
MiddleMan.delete_worker(:worker => :background_worker, :kill => true) 



Here is first some code I used in the console to demonstrate in plain 
Ruby what should happen:

##########################################################
  def do_work(args=nil)
    job_exit_status = 'FORCED_EXIT'
    puts "#{Time.now} -- do_work entered"
    begin
      # Large batch job here
      sleep 30
    rescue
      job_exit_status = 'ERRORED_OUT'
      # Error processing here
    else
      job_exit_status = 'FINISHED_NORMALLY'
    ensure
      puts "#{Time.now} -- do_work exit status: #{job_exit_status}"
    end
  end
##########################################################


If Ctrl-C is pressed before completion, here is what we get:

 >> do_work
2008-02-24 10:12:03 -- do_work entered
2008-02-24 10:12:04 -- do_work exit status: FORCED_EXIT



Now here is the BackgrounDRb code used to show the bug:

##########################################################
class BackgroundWorker < BackgrounDRb::MetaWorker
  set_worker_name :background_worker
  def create(args = nil); end
  def do_work(args=nil)
    job_exit_status = 'FORCEABLY_DELETED'
    logger.info "#{Time.now} -- do_work entered"
    begin
      # Large batch job here
      sleep 30
    rescue
      job_exit_status = 'ERRORED_OUT'
      # Error processing here
    else
      job_exit_status = 'FINISHED_NORMALLY'
    ensure
      logger.info "#{Time.now} -- do_work exit status: #{job_exit_status}"
    end
  end
end
##########################################################

If allowed to finish, we get this in the logs:
2008-02-24 09:45:37 -- do_work entered
2008-02-24 09:45:47 -- do_work exit status: FINISHED_NORMALLY

However, if delete_worker is called before completion, we only get:
2008-02-24 09:46:03 -- do_work entered



Cheers,

-- 
Yves-Eric


From gethemant at gmail.com  Sun Feb 24 13:25:23 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 24 Feb 2008 23:55:23 +0530
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
Message-ID: <fb283d5e0802241025r2a969928la6826a17c520d7bb@mail.gmail.com>

Hi

On Sun, Feb 24, 2008 at 3:01 AM, John Wells <lists at sourceillustrated.com> wrote:
> On Wed, Feb 20, 2008 at 2:51 PM, John Wells <lists at sourceillustrated.com> wrote:
>  > Ok, but when would exit be called here? After all the work is done?
>  >
>  > I guess what I want is a kill -9 functionality, but with the ability to at
>  > the very least clear the worker's status...
>
>  Hemant,
>
>  Would you mind answering this? I'm just getting going again after a
>  nasty bout of the flu, but very interested in your answer.
>

John,

Sorry for later reply. Look into my reply in another thread by Eric.

From gethemant at gmail.com  Sun Feb 24 13:37:42 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 25 Feb 2008 00:07:42 +0530
Subject: [Backgroundrb-devel] Bug report: delete_worker breaks
	begin...ensure (was Re: Canceling work...)
In-Reply-To: <47C0C6A5.2010102@filter.yve.net>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
	<47C0C6A5.2010102@filter.yve.net>
Message-ID: <fb283d5e0802241037t4384e044ud5ba4bd502a0ef0e@mail.gmail.com>

Hi,

On Sun, Feb 24, 2008 at 6:51 AM, Yves-Eric Martin
<yem_backgroundrb-devel at filter.yve.net> wrote:
> Hi all, and welcome back John.
>
>
>  John, you may have missed my second reply to you:
>  http://rubyforge.org/pipermail/backgroundrb-devel/2008-February/001514.html
>
>  Anyway, the bad news is: I just tried it out and it does not seem to
>  work. delete_worker *is* the "kill -9" hemant suggested.
>
>  Hemant, this is breaking some basic Ruby functionality, namely
>  "begin...ensure". So can we please call this a bug? delete_worker should
>  be a "kill", not a "kill -9", or at least not by default. We could add
>  an argument to determine which behavior we want:
>
>  # normal "kill" behavior:
>  MiddleMan.delete_worker(:worker => :background_worker)
>
>  # brutal "kill -9" behavior:
>  MiddleMan.delete_worker(:worker => :background_worker, :kill => true)
>
>
>
>  Here is first some code I used in the console to demonstrate in plain
>  Ruby what should happen:
>
>  ##########################################################
>   def do_work(args=nil)
>     job_exit_status = 'FORCED_EXIT'
>     puts "#{Time.now} -- do_work entered"
>     begin
>       # Large batch job here
>       sleep 30
>     rescue
>       job_exit_status = 'ERRORED_OUT'
>       # Error processing here
>     else
>       job_exit_status = 'FINISHED_NORMALLY'
>     ensure
>       puts "#{Time.now} -- do_work exit status: #{job_exit_status}"
>     end
>   end
>  ##########################################################
>
>
>  If Ctrl-C is pressed before completion, here is what we get:
>
>   >> do_work
>  2008-02-24 10:12:03 -- do_work entered
>  2008-02-24 10:12:04 -- do_work exit status: FORCED_EXIT
>
>
>
>  Now here is the BackgrounDRb code used to show the bug:
>
>  ##########################################################
>  class BackgroundWorker < BackgrounDRb::MetaWorker
>   set_worker_name :background_worker
>   def create(args = nil); end
>   def do_work(args=nil)
>     job_exit_status = 'FORCEABLY_DELETED'
>     logger.info "#{Time.now} -- do_work entered"
>     begin
>       # Large batch job here
>       sleep 30
>     rescue
>       job_exit_status = 'ERRORED_OUT'
>       # Error processing here
>     else
>       job_exit_status = 'FINISHED_NORMALLY'
>     ensure
>       logger.info "#{Time.now} -- do_work exit status: #{job_exit_status}"
>     end
>   end
>  end
>  ##########################################################
>
>  If allowed to finish, we get this in the logs:
>  2008-02-24 09:45:37 -- do_work entered
>  2008-02-24 09:45:47 -- do_work exit status: FINISHED_NORMALLY
>
>  However, if delete_worker is called before completion, we only get:
>  2008-02-24 09:46:03 -- do_work entered

Eric,

Problem is if you look into code of delete_worker then, you will find
that we are sending TERM and KILL signals to the worker and hence its
not possible to use "ensure" there.

Why, we have both "TERM" and "KILL" there is because, signals to kill
a task is not cross platform across unixes.
But if we remove "KILL" and have "TERM" only, your worker will work as
you expected.

Along with Alex patches, I am trying to push this change in git, so
you can try. But, currently gitorious is down, sadly. I am literally
waiting for getting the damn thing up.

From gethemant at gmail.com  Sun Feb 24 13:39:46 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 25 Feb 2008 00:09:46 +0530
Subject: [Backgroundrb-devel] Canceling work...
In-Reply-To: <44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<fb283d5e0802180545j2fcb4c4n6a69cb8e5a56bea6@mail.gmail.com>
	<44dddf400802180959n73d59165n1644e6eaa6bdeffb@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
Message-ID: <fb283d5e0802241039h2738fb07l4fd50cb18ad29b47@mail.gmail.com>

On Sun, Feb 24, 2008 at 3:01 AM, John Wells <lists at sourceillustrated.com> wrote:
> On Wed, Feb 20, 2008 at 2:51 PM, John Wells <lists at sourceillustrated.com> wrote:
>  > Ok, but when would exit be called here? After all the work is done?
>  >
>  > I guess what I want is a kill -9 functionality, but with the ability to at
>  > the very least clear the worker's status...
>

And you shall have it. I am just not sure if it will work across all
*nixes. It works on Linux and thats the only OS I have. As suggested
by Eric, you can use "ensure" to do condition cleanup, if you don't
have "ensure" your worker die as usual.

From gethemant at gmail.com  Mon Feb 25 15:49:08 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 26 Feb 2008 02:19:08 +0530
Subject: [Backgroundrb-devel] Bug report: delete_worker breaks
	begin...ensure (was Re: Canceling work...)
In-Reply-To: <44dddf400802251136k32f88cacq2623da101fa5ae7b@mail.gmail.com>
References: <44dddf400802170938p348cacdape73a9bd68f0ff7fd@mail.gmail.com>
	<44dddf400802201020k2081f2n38500ceead872819@mail.gmail.com>
	<fb283d5e0802201122x186756e7vbad7730fb31e3f9@mail.gmail.com>
	<44dddf400802201132n24221bc8jf203b4f7b21c8c72@mail.gmail.com>
	<fb283d5e0802201139k4db563fcy13ed5161f74fde43@mail.gmail.com>
	<44dddf400802201151i1624fdfdka24ba1fc22678e1f@mail.gmail.com>
	<44dddf400802231331p6cc7d296i1486e904491c55ed@mail.gmail.com>
	<47C0C6A5.2010102@filter.yve.net>
	<fb283d5e0802241037t4384e044ud5ba4bd502a0ef0e@mail.gmail.com>
	<44dddf400802251136k32f88cacq2623da101fa5ae7b@mail.gmail.com>
Message-ID: <fb283d5e0802251249n7d48ca4axb7b1f3af69adc5b@mail.gmail.com>

On Tue, Feb 26, 2008 at 1:06 AM, John Wells <john.wells at greatworx.com> wrote:
>
> On Sun, Feb 24, 2008 at 1:37 PM, hemant <gethemant at gmail.com> wrote:
>  > Hi,
>  >
>  >
>  >
>  >  On Sun, Feb 24, 2008 at 6:51 AM, Yves-Eric Martin
>  >  <yem_backgroundrb-devel at filter.yve.net> wrote:
>  >  > Hi all, and welcome back John.
>  >  >
>  >  >
>  >  >  John, you may have missed my second reply to you:
>  >  >  http://rubyforge.org/pipermail/backgroundrb-devel/2008-February/001514.html
>  >  >
>  >  >  Anyway, the bad news is: I just tried it out and it does not seem to
>  >  >  work. delete_worker *is* the "kill -9" hemant suggested.
>  >  >
>  >  >  Hemant, this is breaking some basic Ruby functionality, namely
>  >  >  "begin...ensure". So can we please call this a bug? delete_worker should
>  >  >  be a "kill", not a "kill -9", or at least not by default. We could add
>  >  >  an argument to determine which behavior we want:
>  >  >
>  >  >  # normal "kill" behavior:
>  >  >  MiddleMan.delete_worker(:worker => :background_worker)
>  >  >
>  >  >  # brutal "kill -9" behavior:
>  >  >  MiddleMan.delete_worker(:worker => :background_worker, :kill => true)
>  >  >
>  >  >
>  >  >
>  >  >  Here is first some code I used in the console to demonstrate in plain
>  >  >  Ruby what should happen:
>  >  >
>  >  >  ##########################################################
>  >  >   def do_work(args=nil)
>  >  >     job_exit_status = 'FORCED_EXIT'
>  >  >     puts "#{Time.now} -- do_work entered"
>  >  >     begin
>  >  >       # Large batch job here
>  >  >       sleep 30
>  >  >     rescue
>  >  >       job_exit_status = 'ERRORED_OUT'
>  >  >       # Error processing here
>  >  >     else
>  >  >       job_exit_status = 'FINISHED_NORMALLY'
>  >  >     ensure
>  >  >       puts "#{Time.now} -- do_work exit status: #{job_exit_status}"
>  >  >     end
>  >  >   end
>  >  >  ##########################################################
>  >  >
>  >  >
>  >  >  If Ctrl-C is pressed before completion, here is what we get:
>  >  >
>  >  >   >> do_work
>  >  >  2008-02-24 10:12:03 -- do_work entered
>  >  >  2008-02-24 10:12:04 -- do_work exit status: FORCED_EXIT
>  >  >
>  >  >
>  >  >
>  >  >  Now here is the BackgrounDRb code used to show the bug:
>  >  >
>  >  >  ##########################################################
>  >  >  class BackgroundWorker < BackgrounDRb::MetaWorker
>  >  >   set_worker_name :background_worker
>  >  >   def create(args = nil); end
>  >  >   def do_work(args=nil)
>  >  >     job_exit_status = 'FORCEABLY_DELETED'
>  >  >     logger.info "#{Time.now} -- do_work entered"
>  >  >     begin
>  >  >       # Large batch job here
>  >  >       sleep 30
>  >  >     rescue
>  >  >       job_exit_status = 'ERRORED_OUT'
>  >  >       # Error processing here
>  >  >     else
>  >  >       job_exit_status = 'FINISHED_NORMALLY'
>  >  >     ensure
>  >  >       logger.info "#{Time.now} -- do_work exit status: #{job_exit_status}"
>  >  >     end
>  >  >   end
>  >  >  end
>  >  >  ##########################################################
>  >  >
>  >  >  If allowed to finish, we get this in the logs:
>  >  >  2008-02-24 09:45:37 -- do_work entered
>  >  >  2008-02-24 09:45:47 -- do_work exit status: FINISHED_NORMALLY
>  >  >
>  >  >  However, if delete_worker is called before completion, we only get:
>  >  >  2008-02-24 09:46:03 -- do_work entered
>  >
>  >  Eric,
>  >
>  >  Problem is if you look into code of delete_worker then, you will find
>  >  that we are sending TERM and KILL signals to the worker and hence its
>  >  not possible to use "ensure" there.
>  >
>  >  Why, we have both "TERM" and "KILL" there is because, signals to kill
>  >  a task is not cross platform across unixes.
>  >  But if we remove "KILL" and have "TERM" only, your worker will work as
>  >  you expected.
>  >
>  >  Along with Alex patches, I am trying to push this change in git, so
>  >  you can try. But, currently gitorious is down, sadly. I am literally
>  >  waiting for getting the damn thing up.
>
>  Thanks guys. I'm looking forward to seeing these patches.

They are there in the git repo, you can get latest code using:

git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb

Also, you need to do:

sudo gem install packet chronic

From gtcampbell at gmail.com  Tue Feb 26 15:27:09 2008
From: gtcampbell at gmail.com (Greg Campbell)
Date: Tue, 26 Feb 2008 12:27:09 -0800
Subject: [Backgroundrb-devel] Registering status for multithreaded
	worker?
In-Reply-To: <fb283d5e0802210354u7946394frb3a49c6f4d3b473e@mail.gmail.com>
References: <ac6faba20802201703y365c33c1gb9904894c76336d9@mail.gmail.com>
	<fb283d5e0802210354u7946394frb3a49c6f4d3b473e@mail.gmail.com>
Message-ID: <ac6faba20802261227q29420a55kaa91db81867fa725@mail.gmail.com>

One followup here:

On Thu, Feb 21, 2008 at 3:54 AM, hemant <gethemant at gmail.com> wrote:

> Hi Greg,
>
> On Thu, Feb 21, 2008 at 6:33 AM, Greg Campbell <gtcampbell at gmail.com>
> wrote:
> > Hi all,
> >
> > I'm using a backgroundrb worker for processing data reporting tasks
> which
> > can be initiated by users of my rails application, and which need to
> support
> > status monitoring.  I had been spawning a new instance with a new job_id
> for
> > each task, and reporting/requesting status via that job_id.  It appears
> that
> > this sort of thing may be better handled by thread_pool, but there seem
> to
> > be two ways of dealing with status reporting, and I'm curious whether
> people
> > have found one to be preferable over the other:
> >
> > I could track status in the database, as I'm creating a new row for each
> > task anyway to store the results, or I could use register_status, with a
> > hash keyed on the equivalent of job_id (inside a mutex, as suggested in
> the
> > README).  Is there any reason to prefer the second over the first?
> > Alternately, am I incorrect in assuming that thread_pool is preferable
> to
> > spawning one worker per user request?
> >
>
> thread_pool is definitely preferable over one worker per request approach.
>
> Also, usually register_status is faster than your hand rolled approach
> of using databases. Also, if worker status results can be stored in
> memcache clusters as well hence is preferable.
>

Things seem to be working with thread_pool, except for one issue -
ask_status returns something incorrect the first time it's called after an
ask_work call.  It looks like the first ask_status response is the return
value from the worker method that's calling thread_pool.defer, when I would
think that return value should be irrelevant (as the worker's being invoked
with the non-blocking ask_work).  Has anyone seen this behavior before?

  For reference, my code basically works this way (with all app-specific
stuff removed):

(controller)
def initiate_task
  @task = Task.create
  MiddleMan.ask_work(:worker => :threaded_worker, :worker_method =>
:process_task, :data => @task.id)
end

#called via AJAX polling to update progress bar
def task_status
  @task_id = params[:task_id].to_i
  status_hash = MiddleMan.ask_status(:worker => :threaded_worker)
  #do something with status_hash[@task_id]...
end

(worker)
def create
  @worker_status = {}
  @status_lock = Mutex.new
  register_status(@worker_status)
end

def process_task(task_id)
  thread_pool.defer(task_id) do |task_id|
    #do several things which call update_status...
  end
  return {:this_should_be => :irrelevant}
end

def update_status(task_id, status)
  @status_mutex.synchronize do
    @worker_status[task_id] = status
  end
  register_status(@worker_status)
end


Based on my logging in the controller, the first time task_status is called,
the status_hash retrieved looks something like this: {:type => :response,
:data => {:this_should_be => :irrelevant}, :client_signature => 11}.  The
next time, however, it looks correct: {(task_id_1) => (task_status_1),
(task_id_2) => (task_status_2)}, etc.  Again, has anyone seen this sort of
thing before?  Am I using the API incorrectly?

Thanks,
Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080226/2184fd67/attachment.html 

From gethemant at gmail.com  Tue Feb 26 16:41:43 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 27 Feb 2008 03:11:43 +0530
Subject: [Backgroundrb-devel] Registering status for multithreaded
	worker?
In-Reply-To: <ac6faba20802261227q29420a55kaa91db81867fa725@mail.gmail.com>
References: <ac6faba20802201703y365c33c1gb9904894c76336d9@mail.gmail.com>
	<fb283d5e0802210354u7946394frb3a49c6f4d3b473e@mail.gmail.com>
	<ac6faba20802261227q29420a55kaa91db81867fa725@mail.gmail.com>
Message-ID: <fb283d5e0802261341p66f8d25dmde8aa92b71ea73cd@mail.gmail.com>

Hi Greg,

On Wed, Feb 27, 2008 at 1:57 AM, Greg Campbell <gtcampbell at gmail.com> wrote:
> One followup here:
>
>
>
> On Thu, Feb 21, 2008 at 3:54 AM, hemant <gethemant at gmail.com> wrote:
> > Hi Greg,
> >
> >
> >
> >
> > On Thu, Feb 21, 2008 at 6:33 AM, Greg Campbell <gtcampbell at gmail.com>
> wrote:
> > > Hi all,
> > >
> > > I'm using a backgroundrb worker for processing data reporting tasks
> which
> > > can be initiated by users of my rails application, and which need to
> support
> > > status monitoring.  I had been spawning a new instance with a new job_id
> for
> > > each task, and reporting/requesting status via that job_id.  It appears
> that
> > > this sort of thing may be better handled by thread_pool, but there seem
> to
> > > be two ways of dealing with status reporting, and I'm curious whether
> people
> > > have found one to be preferable over the other:
> > >
> > > I could track status in the database, as I'm creating a new row for each
> > > task anyway to store the results, or I could use register_status, with a
> > > hash keyed on the equivalent of job_id (inside a mutex, as suggested in
> the
> > > README).  Is there any reason to prefer the second over the first?
> > > Alternately, am I incorrect in assuming that thread_pool is preferable
> to
> > > spawning one worker per user request?
> > >
> >
> > thread_pool is definitely preferable over one worker per request approach.
> >
> > Also, usually register_status is faster than your hand rolled approach
> > of using databases. Also, if worker status results can be stored in
> > memcache clusters as well hence is preferable.
> >
>
> Things seem to be working with thread_pool, except for one issue -
> ask_status returns something incorrect the first time it's called after an
> ask_work call.  It looks like the first ask_status response is the return
> value from the worker method that's calling thread_pool.defer, when I would
> think that return value should be irrelevant (as the worker's being invoked
> with the non-blocking ask_work).  Has anyone seen this behavior before?
>
>   For reference, my code basically works this way (with all app-specific
> stuff removed):
>
> (controller)
> def initiate_task
>   @task = Task.create
>   MiddleMan.ask_work(:worker => :threaded_worker, :worker_method =>
> :process_task, :data => @task.id)
>  end
>
> #called via AJAX polling to update progress bar
> def task_status
>   @task_id = params[:task_id].to_i
>   status_hash = MiddleMan.ask_status(:worker => :threaded_worker)
>   #do something with status_hash[@task_id]...
>  end
>
> (worker)
> def create
>   @worker_status = {}
>   @status_lock = Mutex.new
>   register_status(@worker_status)
> end
>
> def process_task(task_id)
>   thread_pool.defer(task_id) do |task_id|
>     #do several things which call update_status...
>    end
>   return {:this_should_be => :irrelevant}
> end
>
> def update_status(task_id, status)
>   @status_mutex.synchronize do
>     @worker_status[task_id] = status
>   end
>   register_status(@worker_status)
>  end
>
>
> Based on my logging in the controller, the first time task_status is called,
> the status_hash retrieved looks something like this: {:type => :response,
> :data => {:this_should_be => :irrelevant}, :client_signature => 11}.  The
> next time, however, it looks correct: {(task_id_1) => (task_status_1),
> (task_id_2) => (task_status_2)}, etc.  Again, has anyone seen this sort of
> thing before?  Am I using the API incorrectly?

Thanks for the bug report. I was able to reproduce this and hence I
fixed it in trunk ( thats been up on git for a while now ).

Get the code using:

   git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb

and follow the README as usual. You will need to install "packet" and
"chronic" gems.

From siebertm85 at googlemail.com  Wed Feb 27 08:11:39 2008
From: siebertm85 at googlemail.com (Michael Siebert)
Date: Wed, 27 Feb 2008 14:11:39 +0100
Subject: [Backgroundrb-devel] BgDRb initializes my models twice?
Message-ID: <FD77F181-E2CF-486D-95F2-D78ED6C32D05@googlemail.com>

Hello there,

I've stumbled into a phenomenon: my models get loaded twice (or more).  
Best thing would be describing my setup:

I have a Folder model using acts_as_taggable (from  
acts_as_taggable_on_steroids) in the acts_as_taggable class method  
being calledon initialization, it does the following:

> alias_method_chain :reload, :tag_list


Now, when something (namely better_nested_set) calls self.reload,  
reload_with_tag_list is called which in turn calls  
reload_without_tag_list (which is the original one). This is how it  
should work and how it works in rails context.

when i now start backgroundrb and self.reload is called, i get:

> #<SystemStackError: stack level too deep>
> /Users/micha/rails/dmt/trunk/vendor/plugins/ 
> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in  
> `reload_without_tag_list'
> /Users/micha/rails/dmt/trunk/vendor/plugins/ 
> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in `reload'
> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
> better_nested_set.rb:430:in `move_to'
> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
> connection_adapters/abstract/database_statements.rb:66:in  
> `transaction'
> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
> transactions.rb:80:in `transaction'
> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
> transactions.rb:100:in `transaction'
> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
> better_nested_set.rb:429:in `move_to'
> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
> better_nested_set.rb:421:in `move_to_child_of'
> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb: 
> 194:in `process_convert_pdf'
> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
> connection_adapters/abstract/database_statements.rb:66:in  
> `transaction'
> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
> transactions.rb:80:in `transaction'
> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb: 
> 191:in `process_convert_pdf'

with ruby-debug, i can see that acts_as_taggable is called more then  
once, doing the alias_method_chain more than once, too, so that  
reload_with_tag_list calls reload_with_taglist which calls  
reload_with_tag_list. if you don't understand that, try to sketch it :)

is this behaviour known, is it a bug or superman?

greetings,
Micha

---
Siebert WD
Michael Siebert
Trappenweg 6, 15749 Mittenwalde, Deutschland


From siebertm85 at googlemail.com  Wed Feb 27 08:29:26 2008
From: siebertm85 at googlemail.com (Michael Siebert)
Date: Wed, 27 Feb 2008 14:29:26 +0100
Subject: [Backgroundrb-devel] [SOLVED] Re: BgDRb initializes my models twice?
In-Reply-To: <FD77F181-E2CF-486D-95F2-D78ED6C32D05@googlemail.com>
References: <FD77F181-E2CF-486D-95F2-D78ED6C32D05@googlemail.com>
Message-ID: <95B2872E-0579-4528-A59B-D627CB6B659A@googlemail.com>

I got it. MasterProxy#load_rails_models loaded all models, which seems  
to be wrong. for our application, i commented out the call to  
load_rails_models and everything worked fine.

but we should ask ourselves if it is nesseccary to load the rails  
models for master_proxy. it doesnt use it and in the worker processes,  
rails autoloader could load them as well.

greetings,
micha
Am 27.02.2008 um 14:11 schrieb Michael Siebert:

> Hello there,
>
> I've stumbled into a phenomenon: my models get loaded twice (or  
> more). Best thing would be describing my setup:
>
> I have a Folder model using acts_as_taggable (from  
> acts_as_taggable_on_steroids) in the acts_as_taggable class method  
> being calledon initialization, it does the following:
>
>> alias_method_chain :reload, :tag_list
>
>
> Now, when something (namely better_nested_set) calls self.reload,  
> reload_with_tag_list is called which in turn calls  
> reload_without_tag_list (which is the original one). This is how it  
> should work and how it works in rails context.
>
> when i now start backgroundrb and self.reload is called, i get:
>
>> #<SystemStackError: stack level too deep>
>> /Users/micha/rails/dmt/trunk/vendor/plugins/ 
>> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in  
>> `reload_without_tag_list'
>> /Users/micha/rails/dmt/trunk/vendor/plugins/ 
>> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in `reload'
>> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
>> better_nested_set.rb:430:in `move_to'
>> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> connection_adapters/abstract/database_statements.rb:66:in  
>> `transaction'
>> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> transactions.rb:80:in `transaction'
>> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> transactions.rb:100:in `transaction'
>> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
>> better_nested_set.rb:429:in `move_to'
>> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/ 
>> better_nested_set.rb:421:in `move_to_child_of'
>> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb: 
>> 194:in `process_convert_pdf'
>> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> connection_adapters/abstract/database_statements.rb:66:in  
>> `transaction'
>> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/ 
>> transactions.rb:80:in `transaction'
>> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb: 
>> 191:in `process_convert_pdf'
>
> with ruby-debug, i can see that acts_as_taggable is called more then  
> once, doing the alias_method_chain more than once, too, so that  
> reload_with_tag_list calls reload_with_taglist which calls  
> reload_with_tag_list. if you don't understand that, try to sketch  
> it :)
>
> is this behaviour known, is it a bug or superman?
>
> greetings,
> Micha
>
> ---
> Siebert WD
> Michael Siebert
> Trappenweg 6, 15749 Mittenwalde, Deutschland
>

---
Siebert WD
Michael Siebert
Trappenweg 6, 15749 Mittenwalde, Deutschland


From epugh at opensourceconnections.com  Wed Feb 27 09:18:04 2008
From: epugh at opensourceconnections.com (Eric Pugh)
Date: Wed, 27 Feb 2008 09:18:04 -0500
Subject: [Backgroundrb-devel] Triggering a long running task twice while it
	is already running
Message-ID: <0E2E8A15-857E-495A-9A91-88C006825BB7@opensourceconnections.com>

Hi all,

I have a process that "clusters" tagged data using the Ruby Cluster  
Gem...  It takes ~3 minutes to doing the clustering process, so  
backgroundrb is a life saver!  However, I have found that if I attempt  
to ask for another run of the cluster worker while it is already  
running, I get a "Not able to connect" error message...

Does that sound like the correct behavior?  Do I need to manage when I  
call the ask_worker?

Eric

-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com




From clint at ctro.net  Wed Feb 27 12:46:53 2008
From: clint at ctro.net (Clint Troxel)
Date: Wed, 27 Feb 2008 10:46:53 -0700
Subject: [Backgroundrb-devel] backgroundrb current version, updates
Message-ID: <c9b2ca370802270946i5ec1beffg8ca5727b2483291f@mail.gmail.com>

Hi.

I'm using bdrb in a production environment but have been having
problems with reliability.  Hard to diagnose.  First:  I believe the
current version is 1.0.3 -- how do I find the version number of bdrb
in my app?  I can't remember if I'm at the latest version, and this
seems like the most likely place to start.

Thanks,
-clint

ps:  I also can't find an up-to-date changelog.  Is there such a thing?

From ian at telematter.com  Wed Feb 27 13:17:09 2008
From: ian at telematter.com (ian at telematter.com)
Date: Wed, 27 Feb 2008 11:17:09 -0700
Subject: [Backgroundrb-devel] multiple instances of workers?
Message-ID: <20080227111709.7c3536f0ffdc51a02ec2c9d1d72165d5.a1d4b4d451.wbe@email.secureserver.net>

Hi,
  I think I am misunderstanding what a 'worker' is.
Basically I call one up like so in rails:

session[:job_key] = MiddleMan.new_worker(:class => :slicer_worker, :args
=> game.id, :job_key => game.id)
MiddleMan.schedule_worker(
  :class => :slicer_worker,
  :args => game.id,
  :worker_method => :grab_slice,
  :job_key => session[:job_key],
  :worker_method_args => game.id.to_s,        # this HAS to be a string
for reasons beyond my knowledge
  :trigger_type => :trigger,
  :trigger_args => {
    :repeat_interval => 5.seconds
  }
)

I can call as many of these as I like and they do what they
are supposed to do (update some information for a game).
However if one of them needs to stop working because it is done
I am supposed to delete it right?

if blah != nil then
....
else
  logger.debug('should be deleting a worker')
  exit
  #worker.delete
  #MiddleMan.delete_worker(:class => slicer_worker, :job_key => args)

All three of those calls will stop ALL my workers. I just want to stop
the one
that hit that case. The rest can continue on working until they are
done.
Any suggestions?


Thanks,
ian


From clint at ctro.net  Wed Feb 27 18:26:39 2008
From: clint at ctro.net (Clint Troxel)
Date: Wed, 27 Feb 2008 16:26:39 -0700
Subject: [Backgroundrb-devel] worker process hanging around, spiking memory
Message-ID: <c9b2ca370802271526q195ac941yc7cfc1f4dee82c15@mail.gmail.com>

Hi. Posted a general question earlier today.  I have a specific
question this time -- hoping someone can help out.  I think I'm
probably doing something incorrectly.  Here goes:

I have a Worker method that sends a lot of emails.  The method looks like this:


def send_participant_reminders(args=nil)
    args[:emails].each do |email|

      unless email.notified?
        Emailer.deliver_survey_reminder(email)
      end

    end
 end


I call that worker from a controller like so:

    MiddleMan.new_worker(:worker => :participant_worker)
    MiddleMan.ask_work(:worker => :participant_worker,
      :worker_method => :send_participant_reminders,
      :data => {:emails => @emails})

----------------------------

My emails are being sent, but I'm seeing a new process created *each
time I invoke the code in the controller*.  That process starts out at
about 17m of memory then, once all work appears to have been
completed, jumps to about 50m.  The process stays around until
"script/backgroundrb stop".

Maybe I've been reading the examples incorrectly?

Any help would be appreciated.  Seeing this same problem (I think)
suck up 4G of ram on a production server.  yuck.

Thanks again,
clint

From gtcampbell at gmail.com  Wed Feb 27 19:07:12 2008
From: gtcampbell at gmail.com (Greg Campbell)
Date: Wed, 27 Feb 2008 16:07:12 -0800
Subject: [Backgroundrb-devel] worker process hanging around,
	spiking memory
In-Reply-To: <c9b2ca370802271526q195ac941yc7cfc1f4dee82c15@mail.gmail.com>
References: <c9b2ca370802271526q195ac941yc7cfc1f4dee82c15@mail.gmail.com>
Message-ID: <ac6faba20802271607p2b6782c3o75b219787af2044f@mail.gmail.com>

Hi Clint,

On Wed, Feb 27, 2008 at 3:26 PM, Clint Troxel <clint at ctro.net> wrote:

> Hi. Posted a general question earlier today.  I have a specific
> question this time -- hoping someone can help out.  I think I'm
> probably doing something incorrectly.  Here goes:
>
> I have a Worker method that sends a lot of emails.  The method looks like
> this:
>
>
> def send_participant_reminders(args=nil)
>    args[:emails].each do |email|
>
>      unless email.notified?
>        Emailer.deliver_survey_reminder(email)
>      end
>
>    end
>  end
>
>
> I call that worker from a controller like so:
>
>    MiddleMan.new_worker(:worker => :participant_worker)
>    MiddleMan.ask_work(:worker => :participant_worker,
>      :worker_method => :send_participant_reminders,
>      :data => {:emails => @emails})
>
> ----------------------------
>
> My emails are being sent, but I'm seeing a new process created *each
> time I invoke the code in the controller*.  That process starts out at
> about 17m of memory then, once all work appears to have been
> completed, jumps to about 50m.  The process stays around until
> "script/backgroundrb stop".
>
> Maybe I've been reading the examples incorrectly?
>
> Any help would be appreciated.  Seeing this same problem (I think)
> suck up 4G of ram on a production server.  yuck.
>
> Thanks again,
> clint


Calling new_worker does indeed create a new backgroundrb process (which may
or may not be necessary in your case, see below).  If you do in fact need to
create a new process for each task, you'll need to call either exit (from
within the worker) or MiddleMan.delete_worker (from the controller) once the
task is done. If you need to keep track of these worker tasks while they're
running (to call delete_worker from the controller, or to use ask_status),
you should also use a job_key when you create it.

You may not actually need the new process, however.  If the likely use case
is that there will only be one of these tasks running at a time, then you
can just omit the new_worker call entirely and direct all calls to a single
running instance of the worker process.  You may also want to investigate
using thread_pool.defer inside the worker to run the individual tasks in
threads (rather than spawning new worker processes for each one) if it seems
likely that there will be multiple requests running simultaneously.

Thanks,
Greg
p.s. If I've misrepresented anything, I'd like to invite Hemant or anyone
else more knowledgeable than I am to correct me, as I've only started
working on backgroundrb in the last few weeks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080227/aa903a9b/attachment.html 

From clint at ctro.net  Wed Feb 27 19:24:52 2008
From: clint at ctro.net (Clint Troxel)
Date: Wed, 27 Feb 2008 17:24:52 -0700
Subject: [Backgroundrb-devel] worker process hanging around,
	spiking memory
In-Reply-To: <ac6faba20802271607p2b6782c3o75b219787af2044f@mail.gmail.com>
References: <c9b2ca370802271526q195ac941yc7cfc1f4dee82c15@mail.gmail.com>
	<ac6faba20802271607p2b6782c3o75b219787af2044f@mail.gmail.com>
Message-ID: <c9b2ca370802271624j74325920k2f04005698ca9ee4@mail.gmail.com>

Greg,

that is helpful.  thanks!

calling exit() from the workers does indeed kill the process.

Two more questions:

1) before the process is killed by exit() I repeatedly see the memory
of the bdrb process spike from 18 to 45 M -- is there an explanation
for this?  Also, the process seems to take a long time to finish --
minutes just to send 4 emails.  Could some sort of configuration be
causing behavior like this?

2) More importantly, what is the recommended version to run?  And, how
do I check the version in my vendor dir? (I can't remember what
version you were at when I imported into my rails proj)

Thanks again,
-clint

On 2/27/08, Greg Campbell <gtcampbell at gmail.com> wrote:
> Hi Clint,
>
>
> On Wed, Feb 27, 2008 at 3:26 PM, Clint Troxel <clint at ctro.net> wrote:
> > Hi. Posted a general question earlier today.  I have a specific
> > question this time -- hoping someone can help out.  I think I'm
> > probably doing something incorrectly.  Here goes:
> >
> > I have a Worker method that sends a lot of emails.  The method looks like
> this:
> >
> >
> > def send_participant_reminders(args=nil)
> >    args[:emails].each do |email|
> >
> >      unless email.notified?
> >        Emailer.deliver_survey_reminder(email)
> >      end
> >
> >    end
> >  end
> >
> >
> > I call that worker from a controller like so:
> >
> >    MiddleMan.new_worker(:worker => :participant_worker)
> >    MiddleMan.ask_work(:worker => :participant_worker,
> >      :worker_method => :send_participant_reminders,
> >      :data => {:emails => @emails})
> >
> > ----------------------------
> >
> > My emails are being sent, but I'm seeing a new process created *each
> > time I invoke the code in the controller*.  That process starts out at
> > about 17m of memory then, once all work appears to have been
> > completed, jumps to about 50m.  The process stays around until
> > "script/backgroundrb stop".
> >
> > Maybe I've been reading the examples incorrectly?
> >
> > Any help would be appreciated.  Seeing this same problem (I think)
> > suck up 4G of ram on a production server.  yuck.
> >
> > Thanks again,
> > clint
>
> Calling new_worker does indeed create a new backgroundrb process (which may
> or may not be necessary in your case, see below).  If you do in fact need to
> create a new process for each task, you'll need to call either exit (from
> within the worker) or MiddleMan.delete_worker (from the controller) once the
> task is done. If you need to keep track of these worker tasks while they're
> running (to call delete_worker from the controller, or to use ask_status),
> you should also use a job_key when you create it.
>
> You may not actually need the new process, however.  If the likely use case
> is that there will only be one of these tasks running at a time, then you
> can just omit the new_worker call entirely and direct all calls to a single
> running instance of the worker process.  You may also want to investigate
> using thread_pool.defer inside the worker to run the individual tasks in
> threads (rather than spawning new worker processes for each one) if it seems
> likely that there will be multiple requests running simultaneously.
>
> Thanks,
> Greg
> p.s. If I've misrepresented anything, I'd like to invite Hemant or anyone
> else more knowledgeable than I am to correct me, as I've only started
> working on backgroundrb in the last few weeks.
>

From gethemant at gmail.com  Wed Feb 27 20:56:37 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:26:37 +0530
Subject: [Backgroundrb-devel] Release 1.0.3 RC2
Message-ID: <fb283d5e0802271756i2b8f6aecn5e4883192161d9de@mail.gmail.com>

Hi Folks,

I have been putting together new release of BackgrounDRb with lots of
bug fixes, API cleanup, Test Cases and stuff. I have been also working
on getting new documentation ready.

So here are the latest changes:

* Fixed few memory leaks related to workers.
* You can disable logging altogether now using :debug_log false option
in configuration files.
* New cleaner API for invoking tasks/collecting status results from rails:

You can use:

MiddleMan.worker(:foo_worker).some_method

rather than:

MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :some_method)

You can use:

MiddleMan.worker(:foo_worker).ask_status

*  Implemented reloadable workers. Now, you can reload/restart your
worker when its ready to be scheduled. So for workers which aren't
doing lots of acvitity, you
can configure them to reload when firetime has arrived.

I have also put together new documentation which is currently available at:

http://backgroundrb.gnufied.org

Also, whatever API changes that I have introduced won't break any
existing code, so you can upgrade without any problems.





-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Wed Feb 27 21:00:05 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:30:05 +0530
Subject: [Backgroundrb-devel] [SOLVED] Re: BgDRb initializes my models
	twice?
In-Reply-To: <95B2872E-0579-4528-A59B-D627CB6B659A@googlemail.com>
References: <FD77F181-E2CF-486D-95F2-D78ED6C32D05@googlemail.com>
	<95B2872E-0579-4528-A59B-D627CB6B659A@googlemail.com>
Message-ID: <fb283d5e0802271800n2ac5908dl9a99fe6e43d8ef8e@mail.gmail.com>

On Wed, Feb 27, 2008 at 6:59 PM, Michael Siebert
<siebertm85 at googlemail.com> wrote:
> I got it. MasterProxy#load_rails_models loaded all models, which seems
>  to be wrong. for our application, i commented out the call to
>  load_rails_models and everything worked fine.
>
>  but we should ask ourselves if it is nesseccary to load the rails
>  models for master_proxy. it doesnt use it and in the worker processes,
>  rails autoloader could load them as well.
>
>  greetings,
>  micha
>  Am 27.02.2008 um 14:11 schrieb Michael Siebert:
>
>
>
>  > Hello there,
>  >
>  > I've stumbled into a phenomenon: my models get loaded twice (or
>  > more). Best thing would be describing my setup:
>  >
>  > I have a Folder model using acts_as_taggable (from
>  > acts_as_taggable_on_steroids) in the acts_as_taggable class method
>  > being calledon initialization, it does the following:
>  >
>  >> alias_method_chain :reload, :tag_list
>  >
>  >
>  > Now, when something (namely better_nested_set) calls self.reload,
>  > reload_with_tag_list is called which in turn calls
>  > reload_without_tag_list (which is the original one). This is how it
>  > should work and how it works in rails context.
>  >
>  > when i now start backgroundrb and self.reload is called, i get:
>  >
>  >> #<SystemStackError: stack level too deep>
>  >> /Users/micha/rails/dmt/trunk/vendor/plugins/
>  >> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in
>  >> `reload_without_tag_list'
>  >> /Users/micha/rails/dmt/trunk/vendor/plugins/
>  >> acts_as_taggable_on_steroids/lib/acts_as_taggable.rb:192:in `reload'
>  >> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/
>  >> better_nested_set.rb:430:in `move_to'
>  >> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/
>  >> connection_adapters/abstract/database_statements.rb:66:in
>  >> `transaction'
>  >> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/
>  >> transactions.rb:80:in `transaction'
>  >> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/
>  >> transactions.rb:100:in `transaction'
>  >> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/
>  >> better_nested_set.rb:429:in `move_to'
>  >> /Users/micha/rails/dmt/trunk/vendor/plugins/betternestedset/lib/
>  >> better_nested_set.rb:421:in `move_to_child_of'
>  >> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb:
>  >> 194:in `process_convert_pdf'
>  >> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/
>  >> connection_adapters/abstract/database_statements.rb:66:in
>  >> `transaction'
>  >> /Library/Ruby/Gems/1.8/gems/activerecord-2.0.2/lib/active_record/
>  >> transactions.rb:80:in `transaction'
>  >> /Users/micha/rails/dmt/trunk/lib/workers/processing_queue_worker.rb:
>  >> 191:in `process_convert_pdf'
>  >
>  > with ruby-debug, i can see that acts_as_taggable is called more then
>  > once, doing the alias_method_chain more than once, too, so that
>  > reload_with_tag_list calls reload_with_taglist which calls
>  > reload_with_tag_list. if you don't understand that, try to sketch
>  > it :)
>  >
>  > is this behaviour known, is it a bug or superman?
>  >

Prolly superman, but this is a know problem and hence in version
available from git ( or svn trunk also ), has an option

:lazy_load false, which will disable loading of models in master process.

From gethemant at gmail.com  Wed Feb 27 21:03:48 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:33:48 +0530
Subject: [Backgroundrb-devel] Triggering a long running task twice while
	it is already running
In-Reply-To: <0E2E8A15-857E-495A-9A91-88C006825BB7@opensourceconnections.com>
References: <0E2E8A15-857E-495A-9A91-88C006825BB7@opensourceconnections.com>
Message-ID: <fb283d5e0802271803j3b0682c6g2ad7ae65ca7d5e7a@mail.gmail.com>

Hi

On Wed, Feb 27, 2008 at 7:48 PM, Eric Pugh
<epugh at opensourceconnections.com> wrote:
> Hi all,
>
>  I have a process that "clusters" tagged data using the Ruby Cluster
>  Gem...  It takes ~3 minutes to doing the clustering process, so
>  backgroundrb is a life saver!  However, I have found that if I attempt
>  to ask for another run of the cluster worker while it is already
>  running, I get a "Not able to connect" error message...
>
>  Does that sound like the correct behavior?  Do I need to manage when I
>  call the ask_worker?
>
>  Eric
>

Are you starting new workers for each task? You can perhaps use
inbuilt thread_pool for queuing up tasks.
Also, there were some problems while writing to the socket hence if
possible upgrade to latest version available from gitorious using:

git clone git://gitorious.org/backgroundrb/mainline.git backgroundrb

Once 1.0.3 release has been tested by users, I will update the SVN
mirrors to reflect latest code. Also, for using git version, I have
put together some documentation here on:

http://backgroundrb.gnufied.org/

From gethemant at gmail.com  Wed Feb 27 21:09:38 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:39:38 +0530
Subject: [Backgroundrb-devel] worker process hanging around,
	spiking memory
In-Reply-To: <c9b2ca370802271624j74325920k2f04005698ca9ee4@mail.gmail.com>
References: <c9b2ca370802271526q195ac941yc7cfc1f4dee82c15@mail.gmail.com>
	<ac6faba20802271607p2b6782c3o75b219787af2044f@mail.gmail.com>
	<c9b2ca370802271624j74325920k2f04005698ca9ee4@mail.gmail.com>
Message-ID: <fb283d5e0802271809t1473ef0eo1538b99150ff8c0a@mail.gmail.com>

On Thu, Feb 28, 2008 at 5:54 AM, Clint Troxel <clint at ctro.net> wrote:
> Greg,
>
>  that is helpful.  thanks!
>
>  calling exit() from the workers does indeed kill the process.
>
>  Two more questions:
>
>  1) before the process is killed by exit() I repeatedly see the memory
>  of the bdrb process spike from 18 to 45 M -- is there an explanation
>  for this?  Also, the process seems to take a long time to finish --
>  minutes just to send 4 emails.  Could some sort of configuration be
>  causing behavior like this?

I do not think so. But in each process full rails environment is
loaded and hence its costly.
Thats why, you should probably just use thread_pools rather than
starting new processes.

>
>  2) More importantly, what is the recommended version to run?  And, how
>  do I check the version in my vendor dir? (I can't remember what
>  version you were at when I imported into my rails proj)
>

Recommended version to run is one from git mainline repo:

http://gitorious.org/projects/backgroundrb/repos/mainline

We are getting ready for release 1.0.3 and I have put together new
documentation at:

http://backgroundrb.gnufied.org

Please note that above location is just temporary and in a day or two
when 1.0.3 comes out, main documentation at rubyforge will reflect the
same doco.

>
>
>  On 2/27/08, Greg Campbell <gtcampbell at gmail.com> wrote:
>  > Hi Clint,
>  >
>  >
>  > On Wed, Feb 27, 2008 at 3:26 PM, Clint Troxel <clint at ctro.net> wrote:
>  > > Hi. Posted a general question earlier today.  I have a specific
>  > > question this time -- hoping someone can help out.  I think I'm
>  > > probably doing something incorrectly.  Here goes:
>  > >
>  > > I have a Worker method that sends a lot of emails.  The method looks like
>  > this:
>  > >
>  > >
>  > > def send_participant_reminders(args=nil)
>  > >    args[:emails].each do |email|
>  > >
>  > >      unless email.notified?
>  > >        Emailer.deliver_survey_reminder(email)
>  > >      end
>  > >
>  > >    end
>  > >  end
>  > >
>  > >
>  > > I call that worker from a controller like so:
>  > >
>  > >    MiddleMan.new_worker(:worker => :participant_worker)
>  > >    MiddleMan.ask_work(:worker => :participant_worker,
>  > >      :worker_method => :send_participant_reminders,
>  > >      :data => {:emails => @emails})
>  > >
>  > > ----------------------------
>  > >
>  > > My emails are being sent, but I'm seeing a new process created *each
>  > > time I invoke the code in the controller*.  That process starts out at
>  > > about 17m of memory then, once all work appears to have been
>  > > completed, jumps to about 50m.  The process stays around until
>  > > "script/backgroundrb stop".
>  > >
>  > > Maybe I've been reading the examples incorrectly?
>  > >
>  > > Any help would be appreciated.  Seeing this same problem (I think)
>  > > suck up 4G of ram on a production server.  yuck.
>  > >
>  > > Thanks again,
>  > > clint
>  >
>  > Calling new_worker does indeed create a new backgroundrb process (which may
>  > or may not be necessary in your case, see below).  If you do in fact need to
>  > create a new process for each task, you'll need to call either exit (from
>  > within the worker) or MiddleMan.delete_worker (from the controller) once the
>  > task is done. If you need to keep track of these worker tasks while they're
>  > running (to call delete_worker from the controller, or to use ask_status),
>  > you should also use a job_key when you create it.
>  >
>  > You may not actually need the new process, however.  If the likely use case
>  > is that there will only be one of these tasks running at a time, then you
>  > can just omit the new_worker call entirely and direct all calls to a single
>  > running instance of the worker process.  You may also want to investigate
>  > using thread_pool.defer inside the worker to run the individual tasks in
>  > threads (rather than spawning new worker processes for each one) if it seems
>  > likely that there will be multiple requests running simultaneously.
>  >
>  > Thanks,
>  > Greg
>  > p.s. If I've misrepresented anything, I'd like to invite Hemant or anyone
>  > else more knowledgeable than I am to correct me, as I've only started
>  > working on backgroundrb in the last few weeks.
>  >

You have been absolutely correct!

From gethemant at gmail.com  Wed Feb 27 21:12:34 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:42:34 +0530
Subject: [Backgroundrb-devel] Release 1.0.3 RC2
In-Reply-To: <fb283d5e0802271756i2b8f6aecn5e4883192161d9de@mail.gmail.com>
References: <fb283d5e0802271756i2b8f6aecn5e4883192161d9de@mail.gmail.com>
Message-ID: <fb283d5e0802271812y514206b1jee7db75e1ea56f78@mail.gmail.com>

On Thu, Feb 28, 2008 at 7:26 AM, hemant <gethemant at gmail.com> wrote:
> Hi Folks,
>
>  I have been putting together new release of BackgrounDRb with lots of
>  bug fixes, API cleanup, Test Cases and stuff. I have been also working
>  on getting new documentation ready.
>
>  So here are the latest changes:
>
>  * Fixed few memory leaks related to workers.
>  * You can disable logging altogether now using :debug_log false option
>  in configuration files.
>  * New cleaner API for invoking tasks/collecting status results from rails:
>
>  You can use:
>
>  MiddleMan.worker(:foo_worker).some_method
>
>  rather than:
>
>  MiddleMan.ask_work(:worker => :foo_worker, :worker_method => :some_method)
>
>  You can use:
>
>  MiddleMan.worker(:foo_worker).ask_status
>
>  *  Implemented reloadable workers. Now, you can reload/restart your
>  worker when its ready to be scheduled. So for workers which aren't
>  doing lots of acvitity, you
>  can configure them to reload when firetime has arrived.
>
>  I have also put together new documentation which is currently available at:
>
>  http://backgroundrb.gnufied.org
>
>  Also, whatever API changes that I have introduced won't break any
>  existing code, so you can upgrade without any problems.
>
>

Also, above documentation location is totally temporary and just for
people who want to try out RC2 git release.
In a day or two, I will push final release of 1.0.3 and documentation
available at rubyforge will be updated with new content.

Also, when 1.0.3 will be released using git-svn I will update the svn
mirrors having backgroundrb code. At least thats the approach until
svn is deprecated fully.

From gethemant at gmail.com  Wed Feb 27 21:14:23 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 07:44:23 +0530
Subject: [Backgroundrb-devel] backgroundrb current version, updates
In-Reply-To: <c9b2ca370802270946i5ec1beffg8ca5727b2483291f@mail.gmail.com>
References: <c9b2ca370802270946i5ec1beffg8ca5727b2483291f@mail.gmail.com>
Message-ID: <fb283d5e0802271814o1e7e00c3g92c71410358cc6c3@mail.gmail.com>

On Wed, Feb 27, 2008 at 11:16 PM, Clint Troxel <clint at ctro.net> wrote:
> Hi.
>
>  I'm using bdrb in a production environment but have been having
>  problems with reliability.  Hard to diagnose.  First:  I believe the
>  current version is 1.0.3 -- how do I find the version number of bdrb
>  in my app?  I can't remember if I'm at the latest version, and this
>  seems like the most likely place to start.
>

Hi Clint,

Your best option is to run version from git:

http://gitorious.org/projects/backgroundrb/repos/mainline

From epugh at opensourceconnections.com  Thu Feb 28 09:22:25 2008
From: epugh at opensourceconnections.com (Eric Pugh)
Date: Thu, 28 Feb 2008 09:22:25 -0500
Subject: [Backgroundrb-devel] Release 1.0.3 RC2
In-Reply-To: <fb283d5e0802271812y514206b1jee7db75e1ea56f78@mail.gmail.com>
References: <fb283d5e0802271756i2b8f6aecn5e4883192161d9de@mail.gmail.com>
	<fb283d5e0802271812y514206b1jee7db75e1ea56f78@mail.gmail.com>
Message-ID: <9108ED24-9937-4E0E-82E4-8D87E5C905EF@opensourceconnections.com>

Hemant,

I appreciate your pushing stuff out using SVN!  For all us Piston  
users, it makes it much easier to update!

One thought that comes to mind is it would be nice to either have a  
CHANGES or VERSION file in the plugin.  I know I've seen this  
suggested by other users as well.  Real supporting for knowing what  
version of a plugin you had was one of the things I had been hoping  
Rails2 would have had :-(

I like the simpler API as well!

Eric




On Feb 27, 2008, at 9:12 PM, hemant wrote:

> On Thu, Feb 28, 2008 at 7:26 AM, hemant <gethemant at gmail.com> wrote:
>> Hi Folks,
>>
>> I have been putting together new release of BackgrounDRb with lots of
>> bug fixes, API cleanup, Test Cases and stuff. I have been also  
>> working
>> on getting new documentation ready.
>>
>> So here are the latest changes:
>>
>> * Fixed few memory leaks related to workers.
>> * You can disable logging altogether now using :debug_log false  
>> option
>> in configuration files.
>> * New cleaner API for invoking tasks/collecting status results from  
>> rails:
>>
>> You can use:
>>
>> MiddleMan.worker(:foo_worker).some_method
>>
>> rather than:
>>
>> MiddleMan.ask_work(:worker => :foo_worker, :worker_method  
>> => :some_method)
>>
>> You can use:
>>
>> MiddleMan.worker(:foo_worker).ask_status
>>
>> *  Implemented reloadable workers. Now, you can reload/restart your
>> worker when its ready to be scheduled. So for workers which aren't
>> doing lots of acvitity, you
>> can configure them to reload when firetime has arrived.
>>
>> I have also put together new documentation which is currently  
>> available at:
>>
>> http://backgroundrb.gnufied.org
>>
>> Also, whatever API changes that I have introduced won't break any
>> existing code, so you can upgrade without any problems.
>>
>>
>
> Also, above documentation location is totally temporary and just for
> people who want to try out RC2 git release.
> In a day or two, I will push final release of 1.0.3 and documentation
> available at rubyforge will be updated with new content.
>
> Also, when 1.0.3 will be released using git-svn I will update the svn
> mirrors having backgroundrb code. At least thats the approach until
> svn is deprecated fully.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com




From gethemant at gmail.com  Thu Feb 28 11:48:47 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 28 Feb 2008 22:18:47 +0530
Subject: [Backgroundrb-devel] Release 1.0.3 RC2
In-Reply-To: <9108ED24-9937-4E0E-82E4-8D87E5C905EF@opensourceconnections.com>
References: <fb283d5e0802271756i2b8f6aecn5e4883192161d9de@mail.gmail.com>
	<fb283d5e0802271812y514206b1jee7db75e1ea56f78@mail.gmail.com>
	<9108ED24-9937-4E0E-82E4-8D87E5C905EF@opensourceconnections.com>
Message-ID: <fb283d5e0802280848l3391ea34t9464b8129217b214@mail.gmail.com>

On Thu, Feb 28, 2008 at 7:52 PM, Eric Pugh
<epugh at opensourceconnections.com> wrote:
> Hemant,
>
>  I appreciate your pushing stuff out using SVN!  For all us Piston
>  users, it makes it much easier to update!
>
>  One thought that comes to mind is it would be nice to either have a
>  CHANGES or VERSION file in the plugin.  I know I've seen this
>  suggested by other users as well.  Real supporting for knowing what
>  version of a plugin you had was one of the things I had been hoping
>  Rails2 would have had :-(
>

Cool, we will I will add it no problems. Also, I will request people
to try this code.