From bslayerw at gmail.com  Tue Jul  1 13:42:19 2008
From: bslayerw at gmail.com (Byron Wright)
Date: Tue, 1 Jul 2008 10:42:19 -0700
Subject: [Backgroundrb-devel] Cron scheduling for week days
Message-ID: <FE52B645-E38D-42C8-850F-8778B250990E@gmail.com>

Hi,

I'm attempting to setup a cron style scheduling for a method and  
cannot seem to get this to work as expected. I'm trying to call this  
method every morning at 2:00 am week days only (not on sat and sun).  
This is the config that never triggers.

:schedules:
:scheduled_worker:
:do_at_two_weekdays_only:
   :trigger_args: 0 0 2 * * 1-5 *

thanks,
Byron

From gethemant at gmail.com  Tue Jul  1 14:33:31 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 2 Jul 2008 00:03:31 +0530
Subject: [Backgroundrb-devel] Cron scheduling for week days
In-Reply-To: <FE52B645-E38D-42C8-850F-8778B250990E@gmail.com>
References: <FE52B645-E38D-42C8-850F-8778B250990E@gmail.com>
Message-ID: <fb283d5e0807011133w3984a62bna1116fcb3d1a96c1@mail.gmail.com>

On Tue, Jul 1, 2008 at 11:12 PM, Byron Wright <bslayerw at gmail.com> wrote:
> Hi,
>
> I'm attempting to setup a cron style scheduling for a method and cannot seem
> to get this to work as expected. I'm trying to call this method every
> morning at 2:00 am week days only (not on sat and sun). This is the config
> that never triggers.
>
> :schedules:
> :scheduled_worker:
> :do_at_two_weekdays_only:
>  :trigger_args: 0 0 2 * * 1-5 *
>

Well, I was trying to fix these things and reproduce the problem, but
i do not see the problem.I have added these test cases for checking
out issues:

  specify "should run for weekdays " do
    t_time = Time.parse("Wed Feb 13 20:53:43 +0530 2008")
    a = BackgrounDRb::CronTrigger.new("0 0 2 * * 1-5 *")
    firetime = a.fire_after_time(t_time)
    firetime.hour.should == 2
    firetime.min.should == 0
    firetime.sec.should == 0
    firetime.wday.should == 4

    t_time = Time.parse("Fri June 6 20:53:43 +0530 2008")
    a = BackgrounDRb::CronTrigger.new("0 0 2 * * 1-5 *")
    firetime = a.fire_after_time(t_time)
    firetime.hour.should == 2
    firetime.min.should == 0
    firetime.sec.should == 0
    firetime.wday.should == 1
  end


It works as expected. Can i see your worker code?

From bob at kludgebox.com  Wed Jul  2 15:30:15 2008
From: bob at kludgebox.com (Bob Zoller)
Date: Wed, 2 Jul 2008 12:30:15 -0700
Subject: [Backgroundrb-devel] launch worker_a#foo from worker_b#bar
Message-ID: <4ac1582e0807021230v3a241cd7h8f449830c161b3af@mail.gmail.com>

I'm trying to kick off a job from within another worker using
MiddleMan, but it doesn't appear to actually start.  I see the job
request logged though... is this not possible, or am I doing something
wrong?

(backgroundrb from git, rails 2.1.0)

Thanks,
--Bob

From raghu.srinivasan at gmail.com  Thu Jul  3 12:52:20 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Thu, 3 Jul 2008 09:52:20 -0700
Subject: [Backgroundrb-devel] Couple questions on BDRb
Message-ID: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>

Following Hemant's suggestion, I upgraded to the git version of BDRb as per
http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb
-for-better-memory-usage/ and yes, it is more stable and doesn't have the
limit of a maximum of 256 jobs that can be queued. So that's great. But one
thing I've noticed is that my mongrels which start off at about 50MB of
memory slowly start to swell and in about 12-24 hours end up at 200MB or
even 300MB at times. I can restart rails in a couple seconds and the problem
goes away (but eventually returns) but I wanted to know if anyone else is
seeing this and what can be done to avoid it altogether. I am not using any
other gems/plugins except googlecharts. I haven't had this problem before.

Another question - is there a way to roll over the background*.log files
once they hit a certain size like 50 or 100M. Eventually isn't that going to
slow things down - having to write to a ginormous log file? Sort of like how
we can set, in Rails, both a max log file size and the number of log files
to be stored?

Thanks,

Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080703/b1040441/attachment.html>

From gethemant at gmail.com  Thu Jul  3 13:26:38 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 3 Jul 2008 22:56:38 +0530
Subject: [Backgroundrb-devel] Couple questions on BDRb
In-Reply-To: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
References: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
Message-ID: <fb283d5e0807031026x27651afbv5733cf9e69c2701e@mail.gmail.com>

On Thu, Jul 3, 2008 at 10:22 PM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> Following Hemant's suggestion, I upgraded to the git version of BDRb as per
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
> and yes, it is more stable and doesn't have the limit of a maximum of 256
> jobs that can be queued. So that's great. But one thing I've noticed is that
> my mongrels which start off at about 50MB of memory slowly start to swell
> and in about 12-24 hours end up at 200MB or even 300MB at times. I can
> restart rails in a couple seconds and the problem goes away (but eventually
> returns) but I wanted to know if anyone else is seeing this and what can be
> done to avoid it altogether. I am not using any other gems/plugins except
> googlecharts. I haven't had this problem before.

Hmm, you figure this could be a problem with BackgrounDRb client
library, since server runs totally separately from rails and shouldn't
have anything to do with rails?  Anyway, try to run your app under
bleakhouse and see from where this memory issue is coming up. I
haven't seen this kind of problem in our production environment.


>
> Another question - is there a way to roll over the background*.log files
> once they hit a certain size like 50 or 100M. Eventually isn't that going to
> slow things down - having to write to a ginormous log file? Sort of like how
> we can set, in Rails, both a max log file size and the number of log files
> to be stored?
>

Another good idea would be disable logging altogether with following
option in config file:

:backgroundrb:
  :debug_log: false

From gethemant at gmail.com  Thu Jul  3 13:28:42 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 3 Jul 2008 22:58:42 +0530
Subject: [Backgroundrb-devel] launch worker_a#foo from worker_b#bar
In-Reply-To: <4ac1582e0807021230v3a241cd7h8f449830c161b3af@mail.gmail.com>
References: <4ac1582e0807021230v3a241cd7h8f449830c161b3af@mail.gmail.com>
Message-ID: <fb283d5e0807031028t6d887d76kcb725889862921b8@mail.gmail.com>

On Thu, Jul 3, 2008 at 1:00 AM, Bob Zoller <bob at kludgebox.com> wrote:
> I'm trying to kick off a job from within another worker using
> MiddleMan, but it doesn't appear to actually start.  I see the job
> request logged though... is this not possible, or am I doing something
> wrong?
>

This should be perfectly possible. I will look into the issue and
submit a patch asap.

From raghu.srinivasan at gmail.com  Thu Jul  3 13:35:10 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Thu, 3 Jul 2008 10:35:10 -0700
Subject: [Backgroundrb-devel] Couple questions on BDRb
In-Reply-To: <fb283d5e0807031026x27651afbv5733cf9e69c2701e@mail.gmail.com>
References: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
	<fb283d5e0807031026x27651afbv5733cf9e69c2701e@mail.gmail.com>
Message-ID: <6482c06a0807031035l39c8c6g13cb067f0459876a@mail.gmail.com>

1.  I see it came out that way but I didn't mean to say that I was sure that
it was the new BDRb version that was causing my mongrel bloat. Sorry about
that. The only change I made was the switch to this version and the 'packet'
gem upgrade. Are you saying that BDRb should never affect Rails?
2. No, I actually find logging very valuable and don't want to disable it. I
just don't want the log sizes to grow too big.

On Thu, Jul 3, 2008 at 10:26 AM, hemant <gethemant at gmail.com> wrote:

> On Thu, Jul 3, 2008 at 10:22 PM, Raghu Srinivasan
> <raghu.srinivasan at gmail.com> wrote:
> > Following Hemant's suggestion, I upgraded to the git version of BDRb as
> per
> >
> http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
> > and yes, it is more stable and doesn't have the limit of a maximum of 256
> > jobs that can be queued. So that's great. But one thing I've noticed is
> that
> > my mongrels which start off at about 50MB of memory slowly start to swell
> > and in about 12-24 hours end up at 200MB or even 300MB at times. I can
> > restart rails in a couple seconds and the problem goes away (but
> eventually
> > returns) but I wanted to know if anyone else is seeing this and what can
> be
> > done to avoid it altogether. I am not using any other gems/plugins except
> > googlecharts. I haven't had this problem before.
>
> Hmm, you figure this could be a problem with BackgrounDRb client
> library, since server runs totally separately from rails and shouldn't
> have anything to do with rails?  Anyway, try to run your app under
> bleakhouse and see from where this memory issue is coming up. I
> haven't seen this kind of problem in our production environment.
>
>
> >
> > Another question - is there a way to roll over the background*.log files
> > once they hit a certain size like 50 or 100M. Eventually isn't that going
> to
> > slow things down - having to write to a ginormous log file? Sort of like
> how
> > we can set, in Rails, both a max log file size and the number of log
> files
> > to be stored?
> >
>
> Another good idea would be disable logging altogether with following
> option in config file:
>
> :backgroundrb:
>  :debug_log: false
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080703/ed155da1/attachment.html>

From gethemant at gmail.com  Thu Jul  3 13:50:13 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 3 Jul 2008 23:20:13 +0530
Subject: [Backgroundrb-devel] Couple questions on BDRb
In-Reply-To: <6482c06a0807031035l39c8c6g13cb067f0459876a@mail.gmail.com>
References: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
	<fb283d5e0807031026x27651afbv5733cf9e69c2701e@mail.gmail.com>
	<6482c06a0807031035l39c8c6g13cb067f0459876a@mail.gmail.com>
Message-ID: <fb283d5e0807031050x69bc8d6ej19c98bee95e89554@mail.gmail.com>

On Thu, Jul 3, 2008 at 11:05 PM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> 1.  I see it came out that way but I didn't mean to say that I was sure that
> it was the new BDRb version that was causing my mongrel bloat. Sorry about
> that. The only change I made was the switch to this version and the 'packet'
> gem upgrade. Are you saying that BDRb should never affect Rails?

Yes, bdrb should never affect rails. In your mongrel, only client
libraries of BackgrounDRb will be available which holds the socket
connection to the server. Its a very thin layer and shouldn't affect
rails. But anyways, I will try a hard look on this.

> 2. No, I actually find logging very valuable and don't want to disable it. I
> just don't want the log sizes to grow too big.

There is a solution, you can monkey patch log_worker.rb,

@log_file = Logger.new("#{RAILS_HOME}/log/backgroundrb_#{CONFIG_FILE[:backgroundrb][:port]}.log")

and change above line to:

@log_file = Logger.new("#{RAILS_HOME}/log/backgroundrb_#{CONFIG_FILE[:backgroundrb][:port]}.log",'daily')


>
> On Thu, Jul 3, 2008 at 10:26 AM, hemant <gethemant at gmail.com> wrote:
>>
>> On Thu, Jul 3, 2008 at 10:22 PM, Raghu Srinivasan
>> <raghu.srinivasan at gmail.com> wrote:
>> > Following Hemant's suggestion, I upgraded to the git version of BDRb as
>> > per
>> >
>> > http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/
>> > and yes, it is more stable and doesn't have the limit of a maximum of
>> > 256
>> > jobs that can be queued. So that's great. But one thing I've noticed is
>> > that
>> > my mongrels which start off at about 50MB of memory slowly start to
>> > swell
>> > and in about 12-24 hours end up at 200MB or even 300MB at times. I can
>> > restart rails in a couple seconds and the problem goes away (but
>> > eventually
>> > returns) but I wanted to know if anyone else is seeing this and what can
>> > be
>> > done to avoid it altogether. I am not using any other gems/plugins
>> > except
>> > googlecharts. I haven't had this problem before.
>>
>> Hmm, you figure this could be a problem with BackgrounDRb client
>> library, since server runs totally separately from rails and shouldn't
>> have anything to do with rails?  Anyway, try to run your app under
>> bleakhouse and see from where this memory issue is coming up. I
>> haven't seen this kind of problem in our production environment.
>>
>>
>> >
>> > Another question - is there a way to roll over the background*.log files
>> > once they hit a certain size like 50 or 100M. Eventually isn't that
>> > going to
>> > slow things down - having to write to a ginormous log file? Sort of like
>> > how
>> > we can set, in Rails, both a max log file size and the number of log
>> > files
>> > to be stored?
>> >
>>
>> Another good idea would be disable logging altogether with following
>> option in config file:
>>
>> :backgroundrb:
>>  :debug_log: false
>
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From ian.lesperance at gmail.com  Thu Jul  3 16:33:11 2008
From: ian.lesperance at gmail.com (Ian Lesperance)
Date: Thu, 3 Jul 2008 13:33:11 -0700
Subject: [Backgroundrb-devel] PID File Overwritten on Failed Start
Message-ID: <8375e09b0807031333vdb6effcq7811a08f536a3354@mail.gmail.com>

I have monitoring in place for BackgrounDRb to ensure that it stays
up, but I've been getting some false alarms lately.  I've realized
that it has to do with the way BackrounDRb daemonizes.  If you attempt
to start BackgrounDRb while it's already running, it's going to (1)
write its new PID to the file then (2) fail with Errno::EADDRINUSE
upon attempting to establish a socket connection.

Because of a small deployment race condition, sometimes my monitoring
software attempts to start BackgrounDRb along with my actual
deployment scripts.  This causes an invalid PID to get written to the
file.  Since my monitoring software uses this PID file to determine
the status of BackgrounDRb, it keeps sending out false outage alerts
and attempting (and failing) to restart BackgrounDRb.

Now, one quick and simple solution to this is to store the old PID in
a variable, rescue the Errno::EADDRINUSE, and restore the old PID.
I've already written a patch that does just that.

However, is there any reason it should even attempt to start in the
presence of an existing process?  If not, then I could just use
something like Process.getpgid() to check if the old process still
exists and abort the start before.

Thoughts?

Ian

From ian.lesperance at gmail.com  Thu Jul  3 18:40:52 2008
From: ian.lesperance at gmail.com (Ian Lesperance)
Date: Thu, 3 Jul 2008 15:40:52 -0700
Subject: [Backgroundrb-devel] PID File Overwritten on Failed Start
In-Reply-To: <8375e09b0807031333vdb6effcq7811a08f536a3354@mail.gmail.com>
References: <8375e09b0807031333vdb6effcq7811a08f536a3354@mail.gmail.com>
Message-ID: <8375e09b0807031540t58c9a78fv5def2f095c845f19@mail.gmail.com>

Actually, I realized there's still a race condition here, albeit much
smaller than before.

If two fresh starts are attempted at the same time, neither would see
a PID in the file, which means the problem would still exist if the
one that fails was also the one that wrote to the file last, since it
would not be able to revert the PID.

Since the window for that to happen is extremely narrow, I would argue
that that particular race is not worth fretting over.  But if any of
you disagree on the severity, then I'd be happy to continue discussing
it further.  As it stands, though, I think checking process existence
and/or catching EADDRINUSE will be enough for my needs.

Ian

On Thu, Jul 3, 2008 at 1:33 PM, Ian Lesperance <ian.lesperance at gmail.com> wrote:
> I have monitoring in place for BackgrounDRb to ensure that it stays
> up, but I've been getting some false alarms lately.  I've realized
> that it has to do with the way BackrounDRb daemonizes.  If you attempt
> to start BackgrounDRb while it's already running, it's going to (1)
> write its new PID to the file then (2) fail with Errno::EADDRINUSE
> upon attempting to establish a socket connection.
>
> Because of a small deployment race condition, sometimes my monitoring
> software attempts to start BackgrounDRb along with my actual
> deployment scripts.  This causes an invalid PID to get written to the
> file.  Since my monitoring software uses this PID file to determine
> the status of BackgrounDRb, it keeps sending out false outage alerts
> and attempting (and failing) to restart BackgrounDRb.
>
> Now, one quick and simple solution to this is to store the old PID in
> a variable, rescue the Errno::EADDRINUSE, and restore the old PID.
> I've already written a patch that does just that.
>
> However, is there any reason it should even attempt to start in the
> presence of an existing process?  If not, then I could just use
> something like Process.getpgid() to check if the old process still
> exists and abort the start before.
>
> Thoughts?
>
> Ian
>

From sdsykes at gmail.com  Fri Jul  4 08:52:59 2008
From: sdsykes at gmail.com (Stephen Sykes)
Date: Fri, 4 Jul 2008 15:52:59 +0300
Subject: [Backgroundrb-devel] Couple questions on BDRb
In-Reply-To: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
References: <6482c06a0807030952y10d7b059h7b768d05aabfd57a@mail.gmail.com>
Message-ID: <ce7b5df30807040552w5a33a5fdt402af2e01658a976@mail.gmail.com>

On Thu, Jul 3, 2008 at 7:52 PM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> ... But one thing I've noticed is that
> my mongrels which start off at about 50MB of memory slowly start to swell
> and in about 12-24 hours end up at 200MB or even 300MB at times. I can
> restart rails in a couple seconds and the problem goes away (but eventually
> returns) but I wanted to know if anyone else is seeing this and what can be
> done to avoid it altogether. I am not using any other gems/plugins except
> googlecharts. I haven't had this problem before.
>

This is not related to backgroundrb, but yes, we see and have seen
process growth like this.  After very long investigations, we believe
most of our problems to have been attributable to Ruby memory leaks.
See here:

http://pennysmalls.com/2008/03/23/ruby-leaks-memory/

I recommend using Ruby 1.8.7 if you are able.  See here:

http://pennysmalls.com/2008/06/29/rails-on-ruby-187/

Regards,
Stephen

From raghu.srinivasan at gmail.com  Sat Jul  5 02:53:48 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 4 Jul 2008 23:53:48 -0700
Subject: [Backgroundrb-devel] Question on number of processes engendered by
	BDRb
Message-ID: <6482c06a0807042353w24038740v98b28f5a7bb5f024@mail.gmail.com>

I have two workers (ie two files in my RAILS_ROOT/lib/workers directory) on
my Rails app. When I run a top on my box, I see *three* processes called
packer_worker_r. Shouldn't I just be having two? What is the third process
for?

Here is an extract of the output from my top command. The PIDs are 7084,
7085 and 7083.
============================================
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10961 raghus    15   0  193m 110m 3508 S    0 10.8   0:52.87 mongrel_rails
10971 raghus    15   0  188m 107m 3440 S    0 10.5   0:50.61 mongrel_rails
11013 raghus    15   0  179m 103m 3348 S    0 10.1   0:45.18 mongrel_rails
* 7084 raghus    15   0  152m  73m 2036 S   11  7.2 116:31.68
packet_worker_r*
11129 raghus    15   0  134m  58m 3336 S    0  5.7   0:05.20 mongrel_rails
* 7085 raghus    15   0  131m  53m 2020 S    0  5.2   2:23.61
packet_worker_r*
 5094 mysql     15   0  215m  38m 3272 S    0  3.7  44:13.99 mysqld
* 7083 raghus    15   0 97.9m  36m 1192 S    0  3.5   2:28.98
packet_worker_r*
 7081 raghus    15   0 98.3m  34m 1036 S    0  3.4   3:21.40 ruby
10996 raghus    15   0 55820  12m 1340 S    0  1.2   0:06.16 god
11091 raghus    15   0 19748 3728 1384 S    0  0.4   0:00.07 bash
11088 root      16   0 51376 2532 1976 S    0  0.2   0:00.01 sshd
11330 postfix   16   0 38660 1992 1572 S    0  0.2   0:00.00 pickup
 7110 www-data  15   0 28568 1476  568 S    0  0.1   0:03.19 nginx
11
..............
============================================

Thanks,

Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080704/84585011/attachment-0001.html>

From tsanders at hiddenjester.com  Sun Jul  6 12:50:40 2008
From: tsanders at hiddenjester.com (Timothy Sanders)
Date: Sun, 6 Jul 2008 09:50:40 -0700
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <fb283d5e0806181331x749b9b50v1fc5cc920f4c0520@mail.gmail.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
	<fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
	<8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>
	<fb283d5e0806181331x749b9b50v1fc5cc920f4c0520@mail.gmail.com>
Message-ID: <29581764-4345-4352-A00A-16D494BBAD77@hiddenjester.com>

Any update on this new tagged version? I need a new stable version  
before I'll be allowed to deploy an update, and at the moment we've  
got a whole set of things that are waiting for a stable version of  
backgroundrb to deploy.

I understand that it's difficult to have a firm timeline, but it's  
also been a couple of weeks since the last thing we've heard on  
tagging a new version.

On Jun 18, 2008, at 1:31 PM, hemant wrote:

> On Thu, Jun 19, 2008 at 1:39 AM, Ian Lesperance
> <ian.lesperance at gmail.com> wrote:
>> I am hesitant to jump onto the edge, since this is being used in a
>> production application.  Also, since I believe I've managed to  
>> eliminate our
>> reason for needing to restart BackgrounDRb, it's more of an  
>> inconvenience at
>> this point.
>>
>> Do you have plans to tag a release anytime soon, be it through  
>> Subversion or
>> Git?
>
> I am not committing a time line, but I intend to release new version
> this weekend if everything I planned goes well. Also, I hear from
> folks( and in my own experience), git version is a _lot_ more stable.
> If you don't want to change in production at least try in production.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4619 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080706/93b686c4/attachment.bin>

From gethemant at gmail.com  Sun Jul  6 15:17:10 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 7 Jul 2008 00:47:10 +0530
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <29581764-4345-4352-A00A-16D494BBAD77@hiddenjester.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
	<fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
	<8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>
	<fb283d5e0806181331x749b9b50v1fc5cc920f4c0520@mail.gmail.com>
	<29581764-4345-4352-A00A-16D494BBAD77@hiddenjester.com>
Message-ID: <fb283d5e0807061217t7ed2a60cq6db1293fe83b399c@mail.gmail.com>

On Sun, Jul 6, 2008 at 10:20 PM, Timothy Sanders
<tsanders at hiddenjester.com> wrote:
> Any update on this new tagged version? I need a new stable version before
> I'll be allowed to deploy an update, and at the moment we've got a whole set
> of things that are waiting for a stable version of backgroundrb to deploy.
>
> I understand that it's difficult to have a firm timeline, but it's also been
> a couple of weeks since the last thing we've heard on tagging a new version.
>

I can use some help in making new tagged release. Basically, master
branch of github will be merged with testcase branch. I am exclusively
working on testcase branch.

In past, I had some bad experience with hurried releases and don't
want to repeat the same mistake, in the meanwhile, you can happily use
git master, it works pretty rock solid and whenever required i roll
out new fixes.

BUT, next release will be a major release and I don't want to mess it
up. Here are set of things, that I am working on:

1. Get close to 100% test coverage and make tests runnable by end users.
2. Implement support for database based persistent worker queues. Done
in testcase branch.
3. Implement support for clustering of BackgrounDRb servers and load
balancing. This is again done.
4. Support for multi-machine setup and let user run tasks on specified
hosts.Done
5. make result storing thread safe and hence can be called from within threads.
6. Support memcache based result storage properly.
7. Cleanup job_key confusion.
8. Fix some memory leaks introduced by upstream stupid ruby tricks.

You can see this change sheet is huge. There have been some minor API
changes as well, but I want this release to be stable and scalable
within production environments.

If you want to accelerate this process, then checkout code from
testcase branch of github backgroundrb repo and give it a try.Thats
the future. Otherwise, use master branch, there is no problem there.

From tsanders at hiddenjester.com  Sun Jul  6 19:05:34 2008
From: tsanders at hiddenjester.com (Timothy Sanders)
Date: Sun, 6 Jul 2008 16:05:34 -0700
Subject: [Backgroundrb-devel] Restarting BackgrounDRb Means Restarting
	Mongrel?
In-Reply-To: <fb283d5e0807061217t7ed2a60cq6db1293fe83b399c@mail.gmail.com>
References: <8375e09b0806161128l479064a7ufd04f8f9f0050b82@mail.gmail.com>
	<fb283d5e0806161834p17cd032bp6de3f458ad11097b@mail.gmail.com>
	<8375e09b0806181309i4c00f1bbu24c06739919c5440@mail.gmail.com>
	<fb283d5e0806181331x749b9b50v1fc5cc920f4c0520@mail.gmail.com>
	<29581764-4345-4352-A00A-16D494BBAD77@hiddenjester.com>
	<fb283d5e0807061217t7ed2a60cq6db1293fe83b399c@mail.gmail.com>
Message-ID: <6047316B-9330-4B01-B3CB-B4A7BB902825@hiddenjester.com>

Indeed. I wasn't trying to hurry you, I was just curious what the  
status was. Having said that, there's big perceived difference in  
stability between saying "We're using the 1.0.3 release" and "We're  
using the latest from git, which appears to be stable". I'm sure the  
current version *IS* stabler, but there's also a comfort level in  
having a tagged version, and it's just simply a matter of the  
processes and procedures in use on my project saying that we want a  
stable released version of all libraries in the production  
environment. If you were just a few days away from a major release  
then I'd agree that waiting for the list you wrote up is a good idea,  
but if you want more hands testing the current version of the master  
branch I'd suggest packing it up as a 1.0.4 updating it both in git  
and svn.

It seems a little like a chicken and the egg scenario where you want  
users to test out the new version, but some set of users won't be able  
to check it until there's a tagged version. While I can use any  
version I want in development, it's going to be a bigger deal for me  
to try to sell using the current version in production. And truthfully  
it's a little tricky for me to even go to a new version in development  
if I have no idea how long it will be before that version would be  
suitable for production.

It's possible that some of the issue here is simply that I'm not  
familiar with git - so maybe the master branch is equivalent to a  
tagged release. But if the "master branch" is basically the same thing  
as the "trunk revision" would be in svn or cvs, then if it's the  
version you want people using in production environments then I would  
encourage you to give it a new version number. If nothing else, simply  
so people can talk about what version the problem is against.  
Otherwise we're all in a sort of limbo where the master version can  
change at any moment and it's difficult to describe what version  
anybody is actually using.

My impression is that you feel if I was pushing a version to  
production today that I should use the git master, not 1.0.3, but  
that's not really a good solution for everyone. If the git master is a  
better version than 1.0.3, then it really deserves it's own version  
tag, and that's more important to me than whether said tag is 1.0.4,  
1.1, or 2.0.

On Jul 6, 2008, at 12:17 PM, hemant wrote:

> In past, I had some bad experience with hurried releases and don't
> want to repeat the same mistake, in the meanwhile, you can happily use
> git master, it works pretty rock solid and whenever required i roll
> out new fixes.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4619 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080706/9e3944c5/attachment.bin>

From robl at mail.pigdestroyer.co.uk  Mon Jul  7 17:21:04 2008
From: robl at mail.pigdestroyer.co.uk (Rob Lacey)
Date: Mon, 07 Jul 2008 22:21:04 +0100
Subject: [Backgroundrb-devel] ActiveRecord strangeness inside workers
Message-ID: <487288C0.7010004@mail.pigdestroyer.co.uk>

Hi there,

This is my first post...erm, so if you're kind there'll be cake for desert.

I have been implementing a BackgrounDRB solution for a few remote calls 
from my rails app, sometimes the call's response is immediate and 
sometimes it takes 10mins so it seems like a likely candidate. We've 
been plugging away at this problem all day but seems that we have a 
recurring problem that we are unable to destroy a collection objects 
within our worker method with no real indication why. I am sure this 
can't be related to BackgrounDRB itself as I can't see how it could 
break ActiveRecord. However, the server logs don't show the SQL delete 
statement which is seemingly failing, is there a way I can make the log 
output more verbose to be more helpful?

Rob

From gethemant at gmail.com  Tue Jul  8 00:02:19 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 8 Jul 2008 09:32:19 +0530
Subject: [Backgroundrb-devel] ActiveRecord strangeness inside workers
In-Reply-To: <487288C0.7010004@mail.pigdestroyer.co.uk>
References: <487288C0.7010004@mail.pigdestroyer.co.uk>
Message-ID: <fb283d5e0807072102j7c7fea1et2e558abec2e27258@mail.gmail.com>

On Tue, Jul 8, 2008 at 2:51 AM, Rob Lacey <robl at mail.pigdestroyer.co.uk> wrote:
> Hi there,
>
> This is my first post...erm, so if you're kind there'll be cake for desert.
>
> I have been implementing a BackgrounDRB solution for a few remote calls from
> my rails app, sometimes the call's response is immediate and sometimes it
> takes 10mins so it seems like a likely candidate. We've been plugging away
> at this problem all day but seems that we have a recurring problem that we
> are unable to destroy a collection objects within our worker method with no
> real indication why. I am sure this can't be related to BackgrounDRB itself
> as I can't see how it could break ActiveRecord. However, the server logs
> don't show the SQL delete statement which is seemingly failing, is there a
> way I can make the log output more verbose to be more helpful?
>

Few suggestions:

1> Use C mysql bindings. You might be already using, but just make
sure. gem install mysql does the trick.
2> Reduce the number of threads in thread pool using pool_size(2) in
worker code.
3> Paste the worker code, so that everyone else can reproduce it to
fix the issue.

From robl at mail.pigdestroyer.co.uk  Wed Jul  9 07:19:53 2008
From: robl at mail.pigdestroyer.co.uk (Rob Lacey)
Date: Wed, 09 Jul 2008 12:19:53 +0100
Subject: [Backgroundrb-devel] clearing register_status for a job key on
	worker deletion
Message-ID: <48749ED9.5010902@mail.pigdestroyer.co.uk>

Hi,

I've just run into a problem with checking the register_status of a 
worker I am testing. At the moment we create a worker with a particular 
job key, do our thing and then later delete the worker. The problem 
arises when we attempt to re-use the same job key as it appears that the 
register_status of old deleted workers is stored for later retrieval. 
This is actually becoming problematic as we are going to continue to 
re-use job keys, so we need to clear the 'ask_status' results for our 
old deleted workers.

Can anyone point me in the right direction to do this?

Many thanks

Rob

From akyrho at gmail.com  Fri Jul 11 07:10:11 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Fri, 11 Jul 2008 13:10:11 +0200
Subject: [Backgroundrb-devel] cronjob, backgroundrb and crashes
Message-ID: <67b30a7b0807110410p6c729c1s12e9160f9c19244a@mail.gmail.com>

Hello everyone,

I get some trouble with backgroundrb, and I hope finding some help right
here :-)

Here's the case :

Step 1 : A cron job is calling a given URL, each minute

Step 2 : A controller, called by the previous step, is loading a new worker
after some test  :

MiddleMan.new_worker(
                  :worker => :urls_worker,
                  :job_key => "url_" + url.id.to_s,
                  :data => { :url => url.address, :from_public_form => false
})

Step 3 : The worker works... or not. Randomly, the call to the worker just
fail and the worker is not loaded, but backgroundrb is working anyway.

When the call failed once, i've got to kill and restart (most of the time,
more than once) backgroudrb to make it works again.

Thanks per advance...

-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080711/f0c3f18e/attachment.html>

From gethemant at gmail.com  Fri Jul 11 08:29:36 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 11 Jul 2008 17:59:36 +0530
Subject: [Backgroundrb-devel] cronjob, backgroundrb and crashes
In-Reply-To: <67b30a7b0807110410p6c729c1s12e9160f9c19244a@mail.gmail.com>
References: <67b30a7b0807110410p6c729c1s12e9160f9c19244a@mail.gmail.com>
Message-ID: <fb283d5e0807110529g24260261gcdc6bc6e5dee493c@mail.gmail.com>

On Fri, Jul 11, 2008 at 4:40 PM, C?dric <akyrho at gmail.com> wrote:
> Hello everyone,
>
> I get some trouble with backgroundrb, and I hope finding some help right
> here :-)
>
> Here's the case :
>
> Step 1 : A cron job is calling a given URL, each minute
>
> Step 2 : A controller, called by the previous step, is loading a new worker
> after some test  :
>
> MiddleMan.new_worker(
>                   :worker => :urls_worker,
>                   :job_key => "url_" + url.id.to_s,
>                   :data => { :url => url.address, :from_public_form => false
> })
>
> Step 3 : The worker works... or not. Randomly, the call to the worker just
> fail and the worker is not loaded, but backgroundrb is working anyway.
>
> When the call failed once, i've got to kill and restart (most of the time,
> more than once) backgroudrb to make it works again.
>

It could be because, you can't reuse job_keys while starting new
workers as reported in previous mail.

From raghu.srinivasan at gmail.com  Fri Jul 11 12:47:13 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 11 Jul 2008 09:47:13 -0700
Subject: [Backgroundrb-devel] cronjob, backgroundrb and crashes
In-Reply-To: <fb283d5e0807110529g24260261gcdc6bc6e5dee493c@mail.gmail.com>
References: <67b30a7b0807110410p6c729c1s12e9160f9c19244a@mail.gmail.com>
	<fb283d5e0807110529g24260261gcdc6bc6e5dee493c@mail.gmail.com>
Message-ID: <6482c06a0807110947i43101efal31e2538e3f49b953@mail.gmail.com>

Cedric,
I do something very similar (cron->URL wget ->Rails controller -> BDRb jobs
kicked off) and BDRb does great. Just make sure that your job key is unique
each time. Generate a random string if you don't care or you can use a
timestamp or even some DB id/key if you want to keep track of it somewhere.
Raghu

On Fri, Jul 11, 2008 at 5:29 AM, hemant <gethemant at gmail.com> wrote:

> On Fri, Jul 11, 2008 at 4:40 PM, C?dric <akyrho at gmail.com> wrote:
> > Hello everyone,
> >
> > I get some trouble with backgroundrb, and I hope finding some help right
> > here :-)
> >
> > Here's the case :
> >
> > Step 1 : A cron job is calling a given URL, each minute
> >
> > Step 2 : A controller, called by the previous step, is loading a new
> worker
> > after some test  :
> >
> > MiddleMan.new_worker(
> >                   :worker => :urls_worker,
> >                   :job_key => "url_" + url.id.to_s,
> >                   :data => { :url => url.address, :from_public_form =>
> false
> > })
> >
> > Step 3 : The worker works... or not. Randomly, the call to the worker
> just
> > fail and the worker is not loaded, but backgroundrb is working anyway.
> >
> > When the call failed once, i've got to kill and restart (most of the
> time,
> > more than once) backgroudrb to make it works again.
> >
>
> It could be because, you can't reuse job_keys while starting new
> workers as reported in previous mail.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080711/9e2b22cb/attachment.html>

From gethemant at gmail.com  Fri Jul 11 13:28:51 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 11 Jul 2008 22:58:51 +0530
Subject: [Backgroundrb-devel] Upcoming BackgrounDRb release and bug fix fest
	now.
Message-ID: <fb283d5e0807111028u1bdd5e3fxe29afdd98087f1b3@mail.gmail.com>

Folks,

We are planning to get new release of BackgrounDRb out of the door and
we need help from you.

This release will be a major improvement over previous releases and
will be 100% covered with test cases.
Here are list of things in which you can help:

1. Try running the testcase branch of BackgrounDRb, which will become
master and release this weekend.
   Instructions can be found here: http://pastie.org/pastes/231638

   Also there has been slight change in API for accomedating
clustering and multi-machine fail safe setup.
   You can read about it here: http://pastie.org/232136.txt

2. Report any bugs or issues on #backgroundrb at freenode on IRC.

3. Run "rake rcov" and "rake test". And if possible write testcases
for getting that code covered. Send me
  the link to your patch on #backgroundrb or send me a git pull request.

4. Stress test the shitty thing and report any broken behavior.



Thats all.

From raghu.srinivasan at gmail.com  Fri Jul 11 23:20:01 2008
From: raghu.srinivasan at gmail.com (Raghu Srinivasan)
Date: Fri, 11 Jul 2008 20:20:01 -0700
Subject: [Backgroundrb-devel] Question on number of processes engendered by
	BDRb
Message-ID: <6482c06a0807112020w3e913ca1yd7a3a80218bb8707@mail.gmail.com>

I'm reposting in the hope that someone answers it this time:
http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001867.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080711/f2854618/attachment.html>

From gethemant at gmail.com  Fri Jul 11 23:47:03 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 12 Jul 2008 09:17:03 +0530
Subject: [Backgroundrb-devel] Question on number of processes engendered
	by BDRb
In-Reply-To: <6482c06a0807112020w3e913ca1yd7a3a80218bb8707@mail.gmail.com>
References: <6482c06a0807112020w3e913ca1yd7a3a80218bb8707@mail.gmail.com>
Message-ID: <fb283d5e0807112047y35fc1d69t7846b2ab8f205ec0@mail.gmail.com>

On Sat, Jul 12, 2008 at 8:50 AM, Raghu Srinivasan
<raghu.srinivasan at gmail.com> wrote:
> I'm reposting in the hope that someone answers it this time:
> http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001867.html
>

Are you spawning processes at runtime?
Ususually, for bdrb, there will be:

1. Your workers
2. Log worker
3. Master process of backgroundrb

If you are seeing any other process, it might be because of you have
started it dynamically and forgot to delete it.

From gethemant at gmail.com  Sun Jul 13 11:09:17 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 13 Jul 2008 20:39:17 +0530
Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0.4 release
Message-ID: <fb283d5e0807130809s435f862freeaa56bd1196685d@mail.gmail.com>

Hi,

I am proud to release BackgrounDRb 1.0.4. This release contains, many
fixes as listed below:

Folks, as you are probably aware that I am working on getting
cluster/distributed job handling
support for BackgrounDRb.

I am trying to make a brief overview of changes that I have currently
checked into testcase
branch of github.

1. Both synchronous and asychronous task invocation API has changed
for accomadating
   clustering of BackgrounDRb servers. New API is:

# for synchronous tasks:
   # invoke method some_task in worker hello_worker
   MiddleMan.worker(:hello_worker).some_task()

   # invoke method some_task passing 'data' as an argument to method and use
   # session[:user_id] as job_key
   MiddleMan.worker(:hello_worker).some_task(:arg => data,:job_key =>
session[:user_id])

   # run method 'some_task' on all backgroundrb servers
   MiddleMan.worker(:hello_worker).some_task(:arg => data,:job_key =>
session[:user_id],:host => :all)

   # run method 'some_task' on only locally configured server
   MiddleMan.worker(:hello_worker).some_task(:arg => data,:job_key =>
session[:user_id],:host => :local)

   # if you are using a bdrb cluster and now specifying :host option,
your job will run on a
   # server selected in round-robin manner
   MiddleMan.worker(:hello_worker).some_task(:arg => data,:job_key =>
session[:user_id])

   # run the task on specified server and return the results back
   MiddleMan.worker(:hello_worker).some_task(:arg => data,:job_key =>
session[:user_id],:host => "10.0.0.2:11210")

# For asychronous tasks:
   # invoke method some_task in worker hello_worker
   MiddleMan.worker(:hello_worker).async_some_task()

   # invoke method some_task passing 'data' as an argument to method and use
   # session[:user_id] as job_key
   MiddleMan.worker(:hello_worker).async_some_task(:arg =>
data,:job_key => session[:user_id])

   # run method 'some_task' on all backgroundrb servers
   MiddleMan.worker(:hello_worker).async_some_task(:arg => data,
                                               :job_key =>
session[:user_id],:host => :all)

   # run method 'some_task' on only locally configured server
   MiddleMan.worker(:hello_worker).async_some_task(:arg => data,
                                             :job_key =>
session[:user_id],:host => :local)

   # if you are using a bdrb cluster and now specifying :host option,
your job will run on
   # server selected
   MiddleMan.worker(:hello_worker).async_some_task(:arg =>
data,:job_key => session[:user_id],:host => "10.0.0.2:11210")


2. Worker_key and job_key:
   BackgrounDRb prior to this version used job_key as worker global
data and workers
   started new were using 'job_key'. This naming convention was
somewhat confusing
   and hence has been changed.

   we use "worker_key" for associating worker with some unique key in
the same capacity
   as job_key was being used. Now, newer 'job_key' is essentially a
task associated
   data and hence will be used in that capacity. For example, for starting a new
   worker, API is:

   MiddleMan.new_worker(:worker => :hello_worker,:worker_key => <worker_key>)

   For invoking a task on the worker, you will use:

   MiddleMan.worker(:hello_worker,<worker_key>).some_task(:arg =>
"lol",:job_key => "boy")

   Also, for above method if your worker code should be like this:

   class HelloWorker
     def some_task args
       result = some_scrap_url(args)
       cache[job_key] = result
     end
   end

   register_status has been removed and all workers have access to this cache
   object which can be used to put data in. 'cache' object is thread
safe and hence
   multiple threads can write to it. Also, for each executed task, if a job_key
   was supplied while invoking the task, it will be available in worker method.

   Also, job_key is basically a thread local variable inside workers and hence
   if multiple tasks are running concurrently using thread pool or
something, job_key
   will still resolve to correct value.

3. New thread pool:
   Following syntax of adding tasks to thread pool has been deprecated:
     thread_pool.defer(args) do |arg|
       # runs in new thread
     end
   New syntax is:
     thread_pool.defer(:some_method,args)

   Where some_method will run in thread pool and argument passed to
#defer will be
   passed to methods. This was done, to fix a memory leak associated with huge
   number of blocks created at runtime.

3. Persistent job Queue:
   BackgrounDRb now have out of box support for persistent job queues
which are persisted to the
   database. API to add a task in the job_queue is pretty simple:

   MiddleMan(:hello_worker).enq_some_task(:arg =>
"hello_world",:job_key => "boy")

   So in your hello worker:

   class HelloWorker
     def some_task args
       .. do some work ..
       persistent_job.finish! #=> marks the job as finished. totally thread safe
     end
   end

   persistent_job is a thread local variable and will refer to currently
   running queued task can be used from thread pool as well. For example:

   class HelloWorker
     def some_task args
       thread_pool.defer(:fetch_url,args)
     end

     def fetch_url tags
       .. runs in thread ..
       .. fetch tasks ..
       persistent_job.finish!
     end
   end

4. Log worker no longer has rails environment.

5. Result Caching and retrieval:
   Each worker has a worker local 'cache' object, which can be used
   to cache results for later retrieval. You can use memcach as
   result storage, you just need to specify that in config file.

   Also, when you fetch objects from cache all the specified servers are
   queried and response is returned.

6. Clustering and Fail safe:
   If a node in cluster goes down, bdrb will automatically remove that
   node from participation in load balancing, but once that node comes
   up, bdrb will automatically add it back and node will start running tasks.

7. Fixes for large size data transfer.


URLS:
http://backgroundrb.rubyforge.org/

Credits:
Ezra for letting me work on BackgrounDRb. Francis for EventMachine.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Sun Jul 13 11:11:17 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 13 Jul 2008 20:41:17 +0530
Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 1.0.4 release
In-Reply-To: <fb283d5e0807130809s435f862freeaa56bd1196685d@mail.gmail.com>
References: <fb283d5e0807130809s435f862freeaa56bd1196685d@mail.gmail.com>
Message-ID: <fb283d5e0807130811q46d7ed63r6e444df225106964@mail.gmail.com>

On Sun, Jul 13, 2008 at 8:39 PM, hemant <gethemant at gmail.com> wrote:
> Hi,
> I am trying to make a brief overview of changes that I have currently
> checked into testcase
> branch of github.

Ignore this part, all these changes are available in master and svn
repository as well.


Thanks.

From akyrho at gmail.com  Mon Jul 14 10:18:54 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Mon, 14 Jul 2008 16:18:54 +0200
Subject: [Backgroundrb-devel] loading BackgroundRB with max-memory limit?
Message-ID: <67b30a7b0807140718n20e662e8ie8575ae53c14386a@mail.gmail.com>

Hi there,

I still having some trouble with BackgroundRB... sorry per advance :-)

My application has got a worker that takes, let's say, too much RAM and
while in load, it makes my mongrel running very slow. I wondered if it was
possible to load BackgroundRB with a maximal limit of memory?

Thanks.

-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080714/556ebd9d/attachment.html>

From gethemant at gmail.com  Mon Jul 14 10:49:29 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 14 Jul 2008 20:19:29 +0530
Subject: [Backgroundrb-devel] loading BackgroundRB with max-memory limit?
In-Reply-To: <67b30a7b0807140718n20e662e8ie8575ae53c14386a@mail.gmail.com>
References: <67b30a7b0807140718n20e662e8ie8575ae53c14386a@mail.gmail.com>
Message-ID: <fb283d5e0807140749p10a1e33eie3fd3ab0d8e60802@mail.gmail.com>

On Mon, Jul 14, 2008 at 7:48 PM, C?dric <akyrho at gmail.com> wrote:
> Hi there,
>
> I still having some trouble with BackgroundRB... sorry per advance :-)
>
> My application has got a worker that takes, let's say, too much RAM and
> while in load, it makes my mongrel running very slow. I wondered if it was
> possible to load BackgroundRB with a maximal limit of memory?
>

Why is it taking too much RAM? are you storing lots of results in
cache or having lots of tasks in thread pool queue?

If you are storing lots of objects in cache consider swtiching to
memcache for result storage.

From csgyuricza at gmail.com  Tue Jul 15 15:51:00 2008
From: csgyuricza at gmail.com (Csongor Gyuricza)
Date: Tue, 15 Jul 2008 12:51:00 -0700
Subject: [Backgroundrb-devel] Method hangs for the first time if called
	synchronously
Message-ID: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>

This is my first post so I apologize if I am not using the expected
conventions.
With backgroundrb version 1.0.4 and Rails version 2.1.0

The problem happens when you call a method for the first time without using
async_

Steps to reproduce:

1) Create a new worker with a simple method, that just has
logger.info'blabla'. For the example the method is called "ppp" and
worker is
"test_manager"
2) Start backgroundrb and open the rails console

Type in:

>> worker = MiddleMan.worker(:test_manager_worker)
=> #<BackgrounDRb::Rai........ >>
>> worker.async_ppp
=> nil
I see 'blabla' in the log
>> worker.ppp
=> true
I see 'blabla' in the log again

Everything works until now.

Now to reproduce the problem:

1) Exit console
2) Restart backgroundrb and reopen rails console

Type in:

>> worker = MiddleMan.worker(:test_manager_worker)
=> #<BackgrounDRb::Rai........ >>
>> worker.ppp

It hangs and never returns. I see nothing in the log. When I abort I get:

CIRB::Abort: abort then interrupt!!
from /opt/local/lib/ruby/1.8/irb.rb:81:in `irb_abort'
from /opt/local/lib/ruby/1.8/irb.rb:243:in `signal_handle'
from /opt/local/lib/ruby/1.8/irb.rb:66:in `start'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
`call'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
`select'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
`read_from_bdrb'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
`send_request'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
`synchronize'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
`send_request'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
`send'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
`invoke_on_connection'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:57:in
`run_method'
from
/Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:33:in
`method_missing'
from (irb):2

Then when I exit the console, I see "blabla" in the log.

Is this a bug? Another rails user was able to reproduce this same problem.
Is this a bug or are we missing something?
This is not only happening in the console, it happens in my application as
well. I used the console as an example as it is easier to explain.

Thank you,

Csongor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080715/4b5853d1/attachment.html>

From gethemant at gmail.com  Tue Jul 15 16:09:22 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 01:39:22 +0530
Subject: [Backgroundrb-devel] Method hangs for the first time if called
	synchronously
In-Reply-To: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
Message-ID: <fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>

can you please test the same thing on linux and tell me if it happens there too?

i am aware of something like this happening on mac!

On 7/16/08, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
> This is my first post so I apologize if I am not using the expected
> conventions.
> With backgroundrb version 1.0.4 and Rails version 2.1.0
>
> The problem happens when you call a method for the first time without using
> async_
>
> Steps to reproduce:
>
> 1) Create a new worker with a simple method, that just has
> logger.info'blabla'. For the example the method is called "ppp" and
> worker is
> "test_manager"
> 2) Start backgroundrb and open the rails console
>
> Type in:
>
>>> worker = MiddleMan.worker(:test_manager_worker)
> => #<BackgrounDRb::Rai........ >>
>>> worker.async_ppp
> => nil
> I see 'blabla' in the log
>>> worker.ppp
> => true
> I see 'blabla' in the log again
>
> Everything works until now.
>
> Now to reproduce the problem:
>
> 1) Exit console
> 2) Restart backgroundrb and reopen rails console
>
> Type in:
>
>>> worker = MiddleMan.worker(:test_manager_worker)
> => #<BackgrounDRb::Rai........ >>
>>> worker.ppp
>
> It hangs and never returns. I see nothing in the log. When I abort I get:
>
> CIRB::Abort: abort then interrupt!!
> from /opt/local/lib/ruby/1.8/irb.rb:81:in `irb_abort'
> from /opt/local/lib/ruby/1.8/irb.rb:243:in `signal_handle'
> from /opt/local/lib/ruby/1.8/irb.rb:66:in `start'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
> `call'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
> `select'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
> `read_from_bdrb'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
> `send_request'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
> `synchronize'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
> `send_request'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
> `send'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
> `invoke_on_connection'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:57:in
> `run_method'
> from
> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:33:in
> `method_missing'
> from (irb):2
>
> Then when I exit the console, I see "blabla" in the log.
>
> Is this a bug? Another rails user was able to reproduce this same problem.
> Is this a bug or are we missing something?
> This is not only happening in the console, it happens in my application as
> well. I used the console as an example as it is easier to explain.
>
> Thank you,
>
> Csongor
>

-- 
Sent from Gmail for mobile | mobile.google.com

Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From jonathan.wallace at gmail.com  Tue Jul 15 16:29:08 2008
From: jonathan.wallace at gmail.com (Jonathan Wallace)
Date: Tue, 15 Jul 2008 16:29:08 -0400
Subject: [Backgroundrb-devel] Method hangs for the first time if called
	synchronously
In-Reply-To: <fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>
Message-ID: <fd0631d50807151329r5cc1c11cmb6fbbddfe8e89f46@mail.gmail.com>

I wasn't able to reproduce it on linux.  Any suggestions on where to
look in the code for debugging the issue on the mac?
 Jonathan

On Tue, Jul 15, 2008 at 4:09 PM, hemant <gethemant at gmail.com> wrote:
> can you please test the same thing on linux and tell me if it happens there too?
>
> i am aware of something like this happening on mac!
>
> On 7/16/08, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
>> This is my first post so I apologize if I am not using the expected
>> conventions.
>> With backgroundrb version 1.0.4 and Rails version 2.1.0
>>
>> The problem happens when you call a method for the first time without using
>> async_
>>
>> Steps to reproduce:
>>
>> 1) Create a new worker with a simple method, that just has
>> logger.info'blabla'. For the example the method is called "ppp" and
>> worker is
>> "test_manager"
>> 2) Start backgroundrb and open the rails console
>>
>> Type in:
>>
>>>> worker = MiddleMan.worker(:test_manager_worker)
>> => #<BackgrounDRb::Rai........ >>
>>>> worker.async_ppp
>> => nil
>> I see 'blabla' in the log
>>>> worker.ppp
>> => true
>> I see 'blabla' in the log again
>>
>> Everything works until now.
>>
>> Now to reproduce the problem:
>>
>> 1) Exit console
>> 2) Restart backgroundrb and reopen rails console
>>
>> Type in:
>>
>>>> worker = MiddleMan.worker(:test_manager_worker)
>> => #<BackgrounDRb::Rai........ >>
>>>> worker.ppp
>>
>> It hangs and never returns. I see nothing in the log. When I abort I get:
>>
>> CIRB::Abort: abort then interrupt!!
>> from /opt/local/lib/ruby/1.8/irb.rb:81:in `irb_abort'
>> from /opt/local/lib/ruby/1.8/irb.rb:243:in `signal_handle'
>> from /opt/local/lib/ruby/1.8/irb.rb:66:in `start'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>> `call'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>> `select'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>> `read_from_bdrb'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>> `send_request'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>> `synchronize'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>> `send_request'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
>> `send'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
>> `invoke_on_connection'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:57:in
>> `run_method'
>> from
>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:33:in
>> `method_missing'
>> from (irb):2
>>
>> Then when I exit the console, I see "blabla" in the log.
>>
>> Is this a bug? Another rails user was able to reproduce this same problem.
>> Is this a bug or are we missing something?
>> This is not only happening in the console, it happens in my application as
>> well. I used the console as an example as it is easier to explain.
>>
>> Thank you,
>>
>> Csongor
>>
>
> --
> Sent from Gmail for mobile | mobile.google.com
>
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Tue Jul 15 16:41:19 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 02:11:19 +0530
Subject: [Backgroundrb-devel] Method hangs for the first time if called
	synchronously
In-Reply-To: <fd0631d50807151329r5cc1c11cmb6fbbddfe8e89f46@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>
	<fd0631d50807151329r5cc1c11cmb6fbbddfe8e89f46@mail.gmail.com>
Message-ID: <fb283d5e0807151341n514653aep604b2185a9b37191@mail.gmail.com>

its there in packet code actually. in file packet_nbio.rb, while
reading the data this happens. you will find a line thatz commented.

On 7/16/08, Jonathan Wallace <jonathan.wallace at gmail.com> wrote:
> I wasn't able to reproduce it on linux.  Any suggestions on where to
> look in the code for debugging the issue on the mac?
>  Jonathan
>
> On Tue, Jul 15, 2008 at 4:09 PM, hemant <gethemant at gmail.com> wrote:
>> can you please test the same thing on linux and tell me if it happens
>> there too?
>>
>> i am aware of something like this happening on mac!
>>
>> On 7/16/08, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
>>> This is my first post so I apologize if I am not using the expected
>>> conventions.
>>> With backgroundrb version 1.0.4 and Rails version 2.1.0
>>>
>>> The problem happens when you call a method for the first time without
>>> using
>>> async_
>>>
>>> Steps to reproduce:
>>>
>>> 1) Create a new worker with a simple method, that just has
>>> logger.info'blabla'. For the example the method is called "ppp" and
>>> worker is
>>> "test_manager"
>>> 2) Start backgroundrb and open the rails console
>>>
>>> Type in:
>>>
>>>>> worker = MiddleMan.worker(:test_manager_worker)
>>> => #<BackgrounDRb::Rai........ >>
>>>>> worker.async_ppp
>>> => nil
>>> I see 'blabla' in the log
>>>>> worker.ppp
>>> => true
>>> I see 'blabla' in the log again
>>>
>>> Everything works until now.
>>>
>>> Now to reproduce the problem:
>>>
>>> 1) Exit console
>>> 2) Restart backgroundrb and reopen rails console
>>>
>>> Type in:
>>>
>>>>> worker = MiddleMan.worker(:test_manager_worker)
>>> => #<BackgrounDRb::Rai........ >>
>>>>> worker.ppp
>>>
>>> It hangs and never returns. I see nothing in the log. When I abort I get:
>>>
>>> CIRB::Abort: abort then interrupt!!
>>> from /opt/local/lib/ruby/1.8/irb.rb:81:in `irb_abort'
>>> from /opt/local/lib/ruby/1.8/irb.rb:243:in `signal_handle'
>>> from /opt/local/lib/ruby/1.8/irb.rb:66:in `start'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>>> `call'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>>> `select'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:157:in
>>> `read_from_bdrb'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>>> `send_request'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>>> `synchronize'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:171:in
>>> `send_request'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
>>> `send'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:70:in
>>> `invoke_on_connection'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:57:in
>>> `run_method'
>>> from
>>> /Users/csgyuricza/Sites/Automation/vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:33:in
>>> `method_missing'
>>> from (irb):2
>>>
>>> Then when I exit the console, I see "blabla" in the log.
>>>
>>> Is this a bug? Another rails user was able to reproduce this same
>>> problem.
>>> Is this a bug or are we missing something?
>>> This is not only happening in the console, it happens in my application
>>> as
>>> well. I used the console as an example as it is easier to explain.
>>>
>>> Thank you,
>>>
>>> Csongor
>>>
>>
>> --
>> Sent from Gmail for mobile | mobile.google.com
>>
>> Let them talk of their oriental summer climes of everlasting
>> conservatories; give me the privilege of making my own summer with my
>> own coals.
>>
>> http://gnufied.org
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>>
>

-- 
Sent from Gmail for mobile | mobile.google.com

Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From swertyui at gmail.com  Wed Jul 16 03:54:47 2008
From: swertyui at gmail.com (Steve D)
Date: Wed, 16 Jul 2008 03:54:47 -0400
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
Message-ID: <3262a2db0807160054t2595ae9fj5701db5b13b3c600@mail.gmail.com>

hemant,

Would bdrb's queueing mechanism allow for prioritizing of tasks?

I have a few long-running low priority tasks that I'd want to run on the
condition that no high priority tasks are in the queue.

Jack's solution makes sense to my newbie brain, each time my
queue_processing_worker accesses the table, it can sort unprocessed tasks by
priority and begin from there.  But I'd rather stick to bdrb convention if
it's inbuilt already.

- Steve

On Wed, Jun 25, 2008 at 7:20 AM, hemant <gethemant at gmail.com> wrote:

> On Wed, Jun 25, 2008 at 4:30 PM, Frank Schwach <f.schwach at uea.ac.uk>
> wrote:
> > Hemant,
> > These latest additions to backgroundrb look pretty cool. Unfortunately,
> > I don't think I will be able to use it this way because in my setup I
> > can't run anything on the cluster nodes directly. I have to submit jobs
> > to a queuing system on the cluster's master node, which is why I think a
> > simple daemon running on the master node that polls the (remote) db for
> > pending jobs and then submits these to the queue would probably be
> > better for my case - but I'm far from being an expert on distributed
> > systems so any suggestions are very welcome!
>
> Hmm, so use db queuing mechanism inbuilt in bdrb. bdrb still stays
> lightweight because most of these changes aren't affecting core and
> have really went into client side of code.
>
> MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)
>
> Above snippet does exactly that. But anyways, i think you feel its too
> complicated for your setup? I can't help that feeling. Its
> complicated, if its complicated to setup and use, but its not. Those
> features totally stay out of your way, if you don't need them.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080716/18265b2a/attachment.html>

From Alan.Hayes at onevision.com  Wed Jul 16 12:45:10 2008
From: Alan.Hayes at onevision.com (Alan.Hayes at onevision.com)
Date: Wed, 16 Jul 2008 12:45:10 -0400
Subject: [Backgroundrb-devel] Questions about 1.04/ask_result and getting
	status in general
Message-ID: <OF0F77F8F1.66F7AAC7-ON85257488.0053C5DA-85257488.005BA740@onevision.de>

Hemant et al -

        I have been using BackgrounDRb 1.03 and am in the process of 
upgrading to 1.04.  I have been using the register_status/ask_status 
technique for getting info from my workers.  BTW my code worked fine in 
the recent edge version until the released 1.04 did away with 
register_status.  So I have a few questions about the new technique of 
passing cached values, e.g., status, between my workers & Rails (1.2.5).

(1)  With 1.03 I was able to determine whether a worker had been created 
by trying ask_status().  If the worker was not there, I got a nil back, 
otherwise I got the actual status as set by register_status() in the 
worker.  When I try this with ask_result(), it throws 
(bdrb_connection.rb/ask_result(), last line, bdrb_response[:data]) trying 
to evaluate nil.[].

        I could catch this exception in calling code, but I was wondering 
if it might not work to change ask_result() to return nil if 
read_from_bdrb() returns nil like send_request() does.

                Line 142:       def ask_result(p_data)
                        ...
                Line 149:         @mutex.synchronize { bdrb_response = 
read_from_bdrb() }
                Line 150:         close_connection
                Line 151:         bdrb_response? bdrb_response[:data]    : 
nil     <-- only line changed

        That would work for my purposes, but I don't know if it will break 
something else.  My quick perusal of bdrb_connection.rb says it's ok, but 
someone might be expecting ask_result() to throw and with this change it 
would return a nil instead. 

        If that is not the right way to determine whether a given worker 
is instantiated, what is?

(2)  In the more general area of the cache replacement for 1.03's 
status...  (I am using a single bdrb server, not a cluster.) 

        Is the cached shared 
                (a) ... between all instances of all workers?  OR 
                (b) ... all instances of a given worker?  OR 
                (c)  ... does a single instance of a worker have its own 
cache?

        If the cache is shared, does that mean workers can communicate 
with each other.

(3)  One of the problems I had with 1.03 was that the register_status way 
of doing things did not seem to work unless I set no_auto_load(true) and 
started the worker myself from Rails.  Without doing that (believe it or 
not) I was getting the status from a different flavor worker, but that's 
history.  Whether that was me or a bug in bdrb, I don't know. 

        So the question for 1.04 is whether I will have the same problem 
with the cache of an auto_loaded worker.  If not, great.  I have yet to 
try it.

(4)  I could use some examples of passing cached values back and forth. 
The examples in git are still using register_status().  In particular, I 
need to talk to a particular instance of a worker. 


        Thanks for bdrb and the help, alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080716/b0310285/attachment.html>

From f.schwach at uea.ac.uk  Wed Jul 16 14:10:51 2008
From: f.schwach at uea.ac.uk (Frank Schwach)
Date: Wed, 16 Jul 2008 19:10:51 +0100
Subject: [Backgroundrb-devel] best approach to managing workers
	and	getting status
In-Reply-To: <fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
Message-ID: <1216231851.15184.176.camel@fschwachbiopc.bio.uea.ac.uk>

Hi Hemant and others,

Once again thank you for your reply to my last post and the excellent
work you are doing! Having seen your announcement of all those new
features really convinced me to go ahead and use backgroundrb for my
app. However, I must admit being quite confused now with all the latest
changes to the API and would appreciate some help getting my head around
this. So, basically, I have an app with a database table of long running
tasks. There are several different types of tasks, so in my table I have
a column for the type of job to be run and another for serialized data
for the job (basically some IDs and args). I want to run the actual jobs
on a remote machine, which the new version of backgroundrb seems to have
covered nicely now. 

So this is what I thought I could do:

Have a pool of x RemoteJobWorkers on the remote machine waiting to
accept jobs:

class RemoteJobWorker
  def enque_job (job_type, job_args)
    thread_pool.defer(:job_type, job_args)
  end

  def job_type1 (args)
    # update job table with status 
    Jobs.update(job_key, :status => running)
    
    # perform the long running task
    large_dataset.each do
      # update job table with progress
      Jobs.update(job_key, :progress => x)
    end

    # job done: update status again
    Jobs.update(job_key, :status => complete)
  end

  def jobtype2 (args) ... end
  def jobtype3 (args) ... end
  .. and so on ..

end

So when a job is submitted by a user, the controller sends it to the
queue on the remote machine like this(?):

MiddleMan.worker(:remote_job_worker).async_enque_job(:arg =>
data,:job_key => new_job.id,:host => "a.b.c.d:XXX")

where new_job is the newly created instance of my Jobs model.

Does that make sense or is there a better approach that I should be
using? I also like the idea of just having a scheduled worker on the
remote machine polling the database for pending jobs and then sending
them to the queue just like I am trying to do with the above. I would
really appreciate opinions from some experienced users about this to
help me decide. 
In the above I am assuming that the jobs running in thread pool have
access to the job key and can therefore identify the row in the Jobs
table that they need to update? I hope I understood the announcement of
the latest version correctly that that is the case? 

One of my tasks would itself fork to run a third-party program during
its run time - would that be possible inside a job running in a thread
already or would the space-time continuum collapse if I tried that?

I noticed in the announcement that there is support for persistent job
queues now but I am not sure how to tie this in with my existing table.
Hemant, do you have a more detailed example of how to use this and what
the model would look like? 

Apologies for all the questions - your help is more than welcome!!
Thank you all in advance

Frank




On Wed, 2008-06-25 at 16:50 +0530, hemant wrote:
> On Wed, Jun 25, 2008 at 4:30 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> > Hemant,
> > These latest additions to backgroundrb look pretty cool. Unfortunately,
> > I don't think I will be able to use it this way because in my setup I
> > can't run anything on the cluster nodes directly. I have to submit jobs
> > to a queuing system on the cluster's master node, which is why I think a
> > simple daemon running on the master node that polls the (remote) db for
> > pending jobs and then submits these to the queue would probably be
> > better for my case - but I'm far from being an expert on distributed
> > systems so any suggestions are very welcome!
> 
> Hmm, so use db queuing mechanism inbuilt in bdrb. bdrb still stays
> lightweight because most of these changes aren't affecting core and
> have really went into client side of code.
> 
> MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)
> 
> Above snippet does exactly that. But anyways, i think you feel its too
> complicated for your setup? I can't help that feeling. Its
> complicated, if its complicated to setup and use, but its not. Those
> features totally stay out of your way, if you don't need them.
-- 
+++++++++++++++++++++++++++++++
Dr Frank Schwach
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ
Tel: 0044/(0)1603 - 592 405
www.cmp.uea.ac.uk
++++++++++++++++++++++++++++++++

From gethemant at gmail.com  Wed Jul 16 14:12:29 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 23:42:29 +0530
Subject: [Backgroundrb-devel] Questions about 1.04/ask_result and
	getting status in general
In-Reply-To: <OF0F77F8F1.66F7AAC7-ON85257488.0053C5DA-85257488.005BA740@onevision.de>
References: <OF0F77F8F1.66F7AAC7-ON85257488.0053C5DA-85257488.005BA740@onevision.de>
Message-ID: <fb283d5e0807161112g331b9134md07a82b696dbc31a@mail.gmail.com>

On Wed, Jul 16, 2008 at 10:15 PM,  <Alan.Hayes at onevision.com> wrote:
>
> Hemant et al -
>
>         I have been using BackgrounDRb 1.03 and am in the process of
> upgrading to 1.04.  I have been using the register_status/ask_status
> technique for getting info from my workers.  BTW my code worked fine in the
> recent edge version until the released 1.04 did away with register_status.
>  So I have a few questions about the new technique of passing cached values,
> e.g., status, between my workers & Rails (1.2.5).
>
> (1)  With 1.03 I was able to determine whether a worker had been created by
> trying ask_status().  If the worker was not there, I got a nil back,
> otherwise I got the actual status as set by register_status() in the worker.
>  When I try this with ask_result(), it throws
> (bdrb_connection.rb/ask_result(), last line, bdrb_response[:data]) trying to
> evaluate nil.[].
>
>         I could catch this exception in calling code, but I was wondering if
> it might not work to change ask_result() to return nil if read_from_bdrb()
> returns nil like send_request() does.
>
>                 Line 142:        def ask_result(p_data)
>                         ...
>                 Line 149:          @mutex.synchronize { bdrb_response =
> read_from_bdrb() }
>                 Line 150:          close_connection
>                 Line 151:          bdrb_response? bdrb_response[:data]
>   : nil        <-- only line changed
>
>         That would work for my purposes, but I don't know if it will break
> something else.  My quick perusal of bdrb_connection.rb says it's ok, but
> someone might be expecting ask_result() to throw and with this change it
> would return a nil instead.
>
>         If that is not the right way to determine whether a given worker is
> instantiated, what is?

That looks perfectly alright. I have added the change and put testcase
too, so should never happen again.


>
> (2)  In the more general area of the cache replacement for 1.03's status...
>  (I am using a single bdrb server, not a cluster.)
>
>         Is the cached shared
>                 (a) ... between all instances of all workers?  OR
>                 (b) ... all instances of a given worker?  OR
>                 (c)  ... does a single instance of a worker have its own
> cache?
>
>         If the cache is shared, does that mean workers can communicate with
> each other.


If you are not using memcache based cache, then each instance of
worker has its own cache and when you invoke ask_result, in a cluster
environment, its invoked on all servers and if more than one response
come with same cache key, you get an array(unlikely to happen),
otherwise you get the result from whichever server/worker combination
which had the result. On the other hand, if you use memcache(highly
recommended if planning to store cache of result objects), then each
cache key is mixed with worker key and worker name and result is
fetched straight away from memcache in client itself.

>
> (3)  One of the problems I had with 1.03 was that the register_status way of
> doing things did not seem to work unless I set no_auto_load(true) and
> started the worker myself from Rails.  Without doing that (believe it or
> not) I was getting the status from a different flavor worker, but that's
> history.  Whether that was me or a bug in bdrb, I don't know.
>
>         So the question for 1.04 is whether I will have the same problem
> with the cache of an auto_loaded worker.  If not, great.  I have yet to try

Should be pretty solid. Give it a try, and let me know about any troubles.

> it.
>
> (4)  I could use some examples of passing cached values back and forth.  The
> examples in git are still using register_status().  In particular, I need to
> talk to a particular instance of a worker.
>

Yeah, examples directory was stale(I updated it). Actually passing
cached results back and forth is pretty simple:

In your worker, after a long running task:

class LongWorker < BackgrounDRb::MetaWorker
  set_worker_name :long_worker
  def create(args = nil)
    # this method is called, when worker is loaded for the first time
  end

  def some_long_task arg
    #.. do some long task ..
    result = some_long_longer_loner_task(arg)
    cache[some_key] = result
  end
end


Then in Controller:

MiddleMan.worker(:long_worker).ask_result(some_key)

Thats all.

From gethemant at gmail.com  Wed Jul 16 14:17:09 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 23:47:09 +0530
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <3262a2db0807160054t2595ae9fj5701db5b13b3c600@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
	<3262a2db0807160054t2595ae9fj5701db5b13b3c600@mail.gmail.com>
Message-ID: <fb283d5e0807161117q10a89b51l9979b224d9b08242@mail.gmail.com>

On Wed, Jul 16, 2008 at 1:24 PM, Steve D <swertyui at gmail.com> wrote:
> hemant,
>
> Would bdrb's queueing mechanism allow for prioritizing of tasks?
>
> I have a few long-running low priority tasks that I'd want to run on the
> condition that no high priority tasks are in the queue.
>
> Jack's solution makes sense to my newbie brain, each time my
> queue_processing_worker accesses the table, it can sort unprocessed tasks by
> priority and begin from there.  But I'd rather stick to bdrb convention if
> it's inbuilt already.
>

No, it can't right now. But, it should be trivial to add such
functionality, I welcome a patch.

From gethemant at gmail.com  Wed Jul 16 14:19:44 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 23:49:44 +0530
Subject: [Backgroundrb-devel] best approach to managing workers and
	getting status
In-Reply-To: <1216231851.15184.176.camel@fschwachbiopc.bio.uea.ac.uk>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<f84841020805050135h7a4a3236l2968aaf1172ab476@mail.gmail.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
	<1216231851.15184.176.camel@fschwachbiopc.bio.uea.ac.uk>
Message-ID: <fb283d5e0807161119g223309d7ucd73d56b01dde05@mail.gmail.com>

On Wed, Jul 16, 2008 at 11:40 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> Hi Hemant and others,
>
> Once again thank you for your reply to my last post and the excellent
> work you are doing! Having seen your announcement of all those new
> features really convinced me to go ahead and use backgroundrb for my
> app. However, I must admit being quite confused now with all the latest
> changes to the API and would appreciate some help getting my head around
> this. So, basically, I have an app with a database table of long running
> tasks. There are several different types of tasks, so in my table I have
> a column for the type of job to be run and another for serialized data
> for the job (basically some IDs and args). I want to run the actual jobs
> on a remote machine, which the new version of backgroundrb seems to have
> covered nicely now.
>
> So this is what I thought I could do:
>
> Have a pool of x RemoteJobWorkers on the remote machine waiting to
> accept jobs:
>
> class RemoteJobWorker
>  def enque_job (job_type, job_args)
>    thread_pool.defer(:job_type, job_args)
>  end
>
>  def job_type1 (args)
>    # update job table with status
>    Jobs.update(job_key, :status => running)
>
>    # perform the long running task
>    large_dataset.each do
>      # update job table with progress
>      Jobs.update(job_key, :progress => x)
>    end
>
>    # job done: update status again
>    Jobs.update(job_key, :status => complete)
>  end
>
>  def jobtype2 (args) ... end
>  def jobtype3 (args) ... end
>  .. and so on ..
>
> end
>
> So when a job is submitted by a user, the controller sends it to the
> queue on the remote machine like this(?):
>
> MiddleMan.worker(:remote_job_worker).async_enque_job(:arg =>
> data,:job_key => new_job.id,:host => "a.b.c.d:XXX")
>
> where new_job is the newly created instance of my Jobs model.
>


I see you are rolling out your own table based queue here, why not
just use inbuilt persistent database queue?

For the worker, read this page:

http://backgroundrb.rubyforge.org/workers/

Persistent job queue is explained at the bottom.

For adding tasks to the queue you need to do:

MiddleMan.worker(:remote_worker).enq_some_job(:arg => some_arg)

Task will be added to the db queue and automatically picked up.
Handles hairy race conditions for you and stuff like that.

From gethemant at gmail.com  Wed Jul 16 14:23:15 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 16 Jul 2008 23:53:15 +0530
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if called
	synchronously
In-Reply-To: <fb283d5e0807161020i48d27d68ld7b6e7fc6761792a@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>
	<f8a5dfb10807151341m497499a2t4edbeb819279dfbe@mail.gmail.com>
	<fb283d5e0807161020i48d27d68ld7b6e7fc6761792a@mail.gmail.com>
Message-ID: <fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>

On Wed, Jul 16, 2008 at 2:11 AM, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
> I forgot to mention that this is on a Mac.
> Unfortunately I don't have a linux box easily available, I'd have to setup
> one. I won't be able to do it today, I can do it in a couple of days unless
> someone else does it before.

Can you try following fix in packet and report if this fixes your issue:

--- a/lib/packet/packet_nbio.rb
+++ b/lib/packet/packet_nbio.rb
@@ -13,8 +13,8 @@ module Packet
    def read_data(t_sock)
      sock_data = []
      begin
-        while(t_data = t_sock.read_nonblock((16*1024)-1))
-          #raise DisconnectError.new(t_sock,sock_data.join) if t_data.empty?
+        while(t_data = t_sock.recv_nonblock((16*1024)-1))
+          raise DisconnectError.new(t_sock,sock_data.join) if t_data.empty?
          sock_data << t_data
        end
      rescue Errno::EAGAIN

Unfortunately, I don't have Mac OSX, for testing and hence the problem.


Thanks for your patience.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From jonathan.wallace at gmail.com  Wed Jul 16 14:35:50 2008
From: jonathan.wallace at gmail.com (Jonathan Wallace)
Date: Wed, 16 Jul 2008 14:35:50 -0400
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807151309v3f324698n8a846948decd72f1@mail.gmail.com>
	<f8a5dfb10807151341m497499a2t4edbeb819279dfbe@mail.gmail.com>
	<fb283d5e0807161020i48d27d68ld7b6e7fc6761792a@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
Message-ID: <fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>

It didn't fix it for me.
 Jonathan

On Wed, Jul 16, 2008 at 2:23 PM, hemant <gethemant at gmail.com> wrote:
> On Wed, Jul 16, 2008 at 2:11 AM, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
>> I forgot to mention that this is on a Mac.
>> Unfortunately I don't have a linux box easily available, I'd have to setup
>> one. I won't be able to do it today, I can do it in a couple of days unless
>> someone else does it before.
>
> Can you try following fix in packet and report if this fixes your issue:
>
> --- a/lib/packet/packet_nbio.rb
> +++ b/lib/packet/packet_nbio.rb
> @@ -13,8 +13,8 @@ module Packet
>    def read_data(t_sock)
>      sock_data = []
>      begin
> -        while(t_data = t_sock.read_nonblock((16*1024)-1))
> -          #raise DisconnectError.new(t_sock,sock_data.join) if t_data.empty?
> +        while(t_data = t_sock.recv_nonblock((16*1024)-1))
> +          raise DisconnectError.new(t_sock,sock_data.join) if t_data.empty?
>          sock_data << t_data
>        end
>      rescue Errno::EAGAIN
>
> Unfortunately, I don't have Mac OSX, for testing and hence the problem.
>
>
> Thanks for your patience.
>
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>

From gethemant at gmail.com  Thu Jul 17 05:06:37 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 17 Jul 2008 14:36:37 +0530
Subject: [Backgroundrb-devel] Fwd: best approach to managing workers and
	getting status
In-Reply-To: <fb283d5e0807170206t266b2c4cp261131189344d34e@mail.gmail.com>
References: <96EDAA86-352C-4086-A597-702F22A0F469@sohara.com>
	<1214328415.30122.87.camel@fschwachbiopc.bio.uea.ac.uk>
	<f84841020806250032x463c3773ta036dcf1c4ed98a7@mail.gmail.com>
	<fb283d5e0806250326m1ee53ca7s71f9a2d26e62ed42@mail.gmail.com>
	<1214391616.30122.121.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0806250420t3fec337evbead1526c56ae99@mail.gmail.com>
	<1216231851.15184.176.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0807161119g223309d7ucd73d56b01dde05@mail.gmail.com>
	<1216282594.15184.202.camel@fschwachbiopc.bio.uea.ac.uk>
	<fb283d5e0807170206t266b2c4cp261131189344d34e@mail.gmail.com>
Message-ID: <fb283d5e0807170206o5103ced9oe34cea25f84aee1a@mail.gmail.com>

---------- Forwarded message ----------
From: hemant <gethemant at gmail.com>
Date: Thu, Jul 17, 2008 at 2:36 PM
Subject: Re: [Backgroundrb-devel] best approach to managing workers
and getting status
To: Frank Schwach <f.schwach at uea.ac.uk>


On Thu, Jul 17, 2008 at 1:46 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> Thanks Hemant,
>
> Yes, I saw the part about the persistent job queue but I have a couple
> of questions about this:
>
> What does the table for the job queue look like?
>

You can open mysql or whatever db you are using and run:

desc bdrb_job_queues;

to see, whats the schema of the table.

> How do I set other states of the job like "error"? Is the job status
> changed from "pending" to "running" automatically in the table so that I
> can query this table for pending/running/completed jobs?

When a task is pulled out of queue, its flagged as taken and you can
specify a timeout period while creating a task. When you invoke
"#finish!" task is marked finished. There is no, "error", because, if
you a task is "taken" and not "finished!" within specified period, its
automatically considered in error state.


>
> Can I get the jobs ID from within the worker so that I can update the
> job queue table "manually" too? If I want to record a "percentage
> completion" I guess I would need that.

Sure, from anywhere in your worker code, you can use, 'persistent_job'
attribute to get task thats dequed from job queue and is currently
running.


> Can I specify the (remote) host with enq_some_job like I can with the
> async_enque_job method?

You can't, because for persistent tasks, it doesn't matter, the worker
which fetches the task first, gets to execute it anyway.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From akyrho at gmail.com  Thu Jul 17 06:33:02 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Thu, 17 Jul 2008 12:33:02 +0200
Subject: [Backgroundrb-devel] Backgroundrb crashes since last SVN update
	(1.0.4)
Message-ID: <67b30a7b0807170333o671d1ab3h5d08ccfd9f508ba3@mail.gmail.com>

Hello,

I've got some trouble to run script/backgroundrb since the last SVN update
(v 1.0.4).

When I run ruby script/backgroundrb, everything goes, but when i call a
worker, backgroundrb crashes and returns the following message :

/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:266:in
`load_missing_constant': uninitialized constant Packet::BinParser
(NameError)
    from
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in
`const_missing'
    from
/home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_worker.rb:161:in
`post_init'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_connection.rb:21:in
`invoke_init'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:302:in
`decorate_handler'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:76:in
`accept_connection'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:202:in
`handle_external_messages'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
`handle_read_event'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
    from
/home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
`initialize'
    from script/backgroundrb:60:in `new'
    from script/backgroundrb:60
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
`read_data': Packet::DisconnectError (Packet::DisconnectError)
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
`handle_internal_messages'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
`handle_read_event'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
`start_worker'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
`fork_and_load'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:108:in
`start_worker'
    from
/home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:16:in
`initialize'
    from
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:19:in
`run'
    from
/home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
`initialize'
    from script/backgroundrb:60:in `new'
    from script/backgroundrb:60

I tried to remove/reinstall backgroundrb as mentionned in some post i found
[1] about a similar problem, but it did not make any changes.

Thanks per advance.

[1] http://railsforum.com/viewtopic.php?id=17566


-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080717/438ed494/attachment-0001.html>

From gethemant at gmail.com  Thu Jul 17 06:59:20 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 17 Jul 2008 16:29:20 +0530
Subject: [Backgroundrb-devel] Backgroundrb crashes since last SVN update
	(1.0.4)
In-Reply-To: <67b30a7b0807170333o671d1ab3h5d08ccfd9f508ba3@mail.gmail.com>
References: <67b30a7b0807170333o671d1ab3h5d08ccfd9f508ba3@mail.gmail.com>
Message-ID: <fb283d5e0807170359w2a0a03f7q2dac8808381b3e9c@mail.gmail.com>

On Thu, Jul 17, 2008 at 4:03 PM, C?dric <akyrho at gmail.com> wrote:
> Hello,
>
> I've got some trouble to run script/backgroundrb since the last SVN update
> (v 1.0.4).
>
> When I run ruby script/backgroundrb, everything goes, but when i call a
> worker, backgroundrb crashes and returns the following message :
>
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:266:in
> `load_missing_constant': uninitialized constant Packet::BinParser
> (NameError)
>     from
> /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:453:in
> `const_missing'
>     from
> /home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_worker.rb:161:in
> `post_init'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_connection.rb:21:in
> `invoke_init'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:302:in
> `decorate_handler'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:76:in
> `accept_connection'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:202:in
> `handle_external_messages'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
> `handle_read_event'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
> `run'
>     from
> /home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
> `initialize'
>     from script/backgroundrb:60:in `new'
>     from script/backgroundrb:60
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:25:in
> `read_data': Packet::DisconnectError (Packet::DisconnectError)
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:46:in
> `handle_internal_messages'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:176:in
> `handle_read_event'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `each'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
> `handle_read_event'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
> `start_reactor'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `loop'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
> `start_reactor'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:20:in
> `start_worker'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:133:in
> `fork_and_load'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:108:in
> `start_worker'
>     from
> /home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:16:in
> `initialize'
>     from
> /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:19:in
> `run'
>     from
> /home/cedric/www/project/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
> `initialize'
>     from script/backgroundrb:60:in `new'
>     from script/backgroundrb:60
>
> I tried to remove/reinstall backgroundrb as mentionned in some post i found
> [1] about a similar problem, but it did not make any changes.

I think, you are trying to run this version of BackgrounDRb with
packet gem  < 0.1.7.

You should install latest version of packet(which is 0.1.7) and try.

From maze at strahlungsfrei.de  Thu Jul 17 08:31:57 2008
From: maze at strahlungsfrei.de (Martin Honermeyer)
Date: Thu, 17 Jul 2008 14:31:57 +0200
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
	<fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
Message-ID: <200807171431.57994.maze@strahlungsfrei.de>

Hello guys,

I just installed Backgroundrb (SVN via piston, Packet gem 0.1.7) in my Rails 
2.1 project. As far as I can tell, I am experiencing exactly the same problem. 
On Linux, that is (Ubuntu 8.04, Kernel 2.6.22-14-xen)! 

The first asynchronous worker calls (in console) worked. Then, after restarting 
Backgroundrb and the console, Every synchronous call hangs at the same 
position as told by Jonathan. Asynchronous calls then have no effect either.

Hemant, I tried your patch, but it doesn't change anything for me. I will do 
anything to help ;-/


Greetz,
Martin

 
On Wednesday 16 July 2008 20:35:50 Jonathan Wallace wrote:
> It didn't fix it for me.
>  Jonathan
>
> On Wed, Jul 16, 2008 at 2:23 PM, hemant <gethemant at gmail.com> wrote:
> > On Wed, Jul 16, 2008 at 2:11 AM, Csongor Gyuricza <csgyuricza at gmail.com> 
wrote:
> >> I forgot to mention that this is on a Mac.
> >> Unfortunately I don't have a linux box easily available, I'd have to
> >> setup one. I won't be able to do it today, I can do it in a couple of
> >> days unless someone else does it before.
> >
> > Can you try following fix in packet and report if this fixes your issue:
> >
> > --- a/lib/packet/packet_nbio.rb
> > +++ b/lib/packet/packet_nbio.rb
> > @@ -13,8 +13,8 @@ module Packet
> >    def read_data(t_sock)
> >      sock_data = []
> >      begin
> > -        while(t_data = t_sock.read_nonblock((16*1024)-1))
> > -          #raise DisconnectError.new(t_sock,sock_data.join) if
> > t_data.empty? +        while(t_data = t_sock.recv_nonblock((16*1024)-1))
> > +          raise DisconnectError.new(t_sock,sock_data.join) if
> > t_data.empty? sock_data << t_data
> >        end
> >      rescue Errno::EAGAIN
> >
> > Unfortunately, I don't have Mac OSX, for testing and hence the problem.
> >
> >
> > Thanks for your patience.
> >
> >
> >
> > --
> > Let them talk of their oriental summer climes of everlasting
> > conservatories; give me the privilege of making my own summer with my
> > own coals.
> >
> > http://gnufied.org
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From gethemant at gmail.com  Thu Jul 17 09:44:33 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 17 Jul 2008 19:14:33 +0530
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <200807171431.57994.maze@strahlungsfrei.de>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
	<fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
	<200807171431.57994.maze@strahlungsfrei.de>
Message-ID: <fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>

On Thu, Jul 17, 2008 at 6:01 PM, Martin Honermeyer
<maze at strahlungsfrei.de> wrote:
> Hello guys,
>
> I just installed Backgroundrb (SVN via piston, Packet gem 0.1.7) in my Rails
> 2.1 project. As far as I can tell, I am experiencing exactly the same problem.
> On Linux, that is (Ubuntu 8.04, Kernel 2.6.22-14-xen)!
>
> The first asynchronous worker calls (in console) worked. Then, after restarting
> Backgroundrb and the console, Every synchronous call hangs at the same
> position as told by Jonathan. Asynchronous calls then have no effect either.
>
> Hemant, I tried your patch, but it doesn't change anything for me. I will do
> anything to help ;-/
>

Okay. Can you pastie your worker code and rails/controller code, using
which you are invoking this methods. Also, please paste the code as
you are using. I will see, if I can reproduce this behaviour.

Also, Check the logs. both backgroundrb_debug_11006.log and
backgroundrb_11006.log and if there is any exception there.

Lastly, what exactly you are trying to send from your worker code? You
realize, not everything in ruby can be dumped across wire.

From csgyuricza at gmail.com  Thu Jul 17 12:22:34 2008
From: csgyuricza at gmail.com (Csongor Gyuricza)
Date: Thu, 17 Jul 2008 09:22:34 -0700
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
	<fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
	<200807171431.57994.maze@strahlungsfrei.de>
	<fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
Message-ID: <f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>

Hemant,

As mentioned in my first email, this happens even when your worker has no
code (or only a logger.info line) and no arguments passed. Nothing shows up
in the log, it just hangs.

Please refer to my first email and let me know if you have any additional
questions.

Thank you for helping us with this issue.

Csongor

On Thu, Jul 17, 2008 at 6:44 AM, hemant <gethemant at gmail.com> wrote:

> On Thu, Jul 17, 2008 at 6:01 PM, Martin Honermeyer
> <maze at strahlungsfrei.de> wrote:
> > Hello guys,
> >
> > I just installed Backgroundrb (SVN via piston, Packet gem 0.1.7) in my
> Rails
> > 2.1 project. As far as I can tell, I am experiencing exactly the same
> problem.
> > On Linux, that is (Ubuntu 8.04, Kernel 2.6.22-14-xen)!
> >
> > The first asynchronous worker calls (in console) worked. Then, after
> restarting
> > Backgroundrb and the console, Every synchronous call hangs at the same
> > position as told by Jonathan. Asynchronous calls then have no effect
> either.
> >
> > Hemant, I tried your patch, but it doesn't change anything for me. I will
> do
> > anything to help ;-/
> >
>
> Okay. Can you pastie your worker code and rails/controller code, using
> which you are invoking this methods. Also, please paste the code as
> you are using. I will see, if I can reproduce this behaviour.
>
> Also, Check the logs. both backgroundrb_debug_11006.log and
> backgroundrb_11006.log and if there is any exception there.
>
> Lastly, what exactly you are trying to send from your worker code? You
> realize, not everything in ruby can be dumped across wire.
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080717/07a2218a/attachment.html>

From maze at strahlungsfrei.de  Thu Jul 17 12:47:53 2008
From: maze at strahlungsfrei.de (Martin Honermeyer)
Date: Thu, 17 Jul 2008 18:47:53 +0200
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
	<f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
Message-ID: <200807171847.54107.maze@strahlungsfrei.de>

I second that. I can remove all code from my worker so only the code from the 
generator is left.


It hangs even when calling a not-existing worker. From the Rails console:

>> MiddleMan.worker(:dfsfd)
=> #<BackgrounDRb::RailsWorkerProxy:0xb6af77e0 
@middle_man=#<BackgrounDRb::ClusterConnection:0xb76d06ac 
@backend_connections=[#<BackgrounDRb::Connection:0xb76d03b4 
@mutex=#<Mutex:0xb76d038c>, 
cluster_conn#<BackgrounDRb::ClusterConnection:0xb76d06ac ..., 
@server_port=11006, @server_ip="0.0.0.0", @connection_status=true>], 
request_count1, bdrb_servers[#<struct #<Class:0xb76d05e4 ip="0.0.0.0", 
port=11006], @round_robin=[0], @last_polled_time=Thu Jul 17 18:30:09 +0200 
2008, @disconnected_connections={}>, worker_keynil, tried_connections[], 
worker_name:dfsfd

>> MiddleMan.worker(:dfsfd).sdfsdf

(hangs)


I am calling from the console (didn't even try from within a controller). 
There is _nothing_ posted in the backgroundrb logs.

For me it seems like there is no connection to the backgroundrb server at all. 
I even rebooted the (Xen virtual Linux) machine, but the problem remains.


My background.yml:

---
:backgroundrb:
  :port: 11006
  :ip: 0.0.0.0
  :debug_log: false # whether to print debug logs to a seperate worker, 
defaults to true
  :log: foreground # will print log messages to STDOUT, defaults to seperate 
log worker


Martin


On Thursday 17 July 2008 18:22:34 you wrote:
> Hemant,
>
> As mentioned in my first email, this happens even when your worker has no
> code (or only a logger.info line) and no arguments passed. Nothing shows up
> in the log, it just hangs.
>
> Please refer to my first email and let me know if you have any additional
> questions.
>
> Thank you for helping us with this issue.
>
> Csongor
>
> On Thu, Jul 17, 2008 at 6:44 AM, hemant <gethemant at gmail.com> wrote:
> > On Thu, Jul 17, 2008 at 6:01 PM, Martin Honermeyer
> >
> > <maze at strahlungsfrei.de> wrote:
> > > Hello guys,
> > >
> > > I just installed Backgroundrb (SVN via piston, Packet gem 0.1.7) in my
> >
> > Rails
> >
> > > 2.1 project. As far as I can tell, I am experiencing exactly the same
> >
> > problem.
> >
> > > On Linux, that is (Ubuntu 8.04, Kernel 2.6.22-14-xen)!
> > >
> > > The first asynchronous worker calls (in console) worked. Then, after
> >
> > restarting
> >
> > > Backgroundrb and the console, Every synchronous call hangs at the same
> > > position as told by Jonathan. Asynchronous calls then have no effect
> >
> > either.
> >
> > > Hemant, I tried your patch, but it doesn't change anything for me. I
> > > will
> >
> > do
> >
> > > anything to help ;-/
> >
> > Okay. Can you pastie your worker code and rails/controller code, using
> > which you are invoking this methods. Also, please paste the code as
> > you are using. I will see, if I can reproduce this behaviour.
> >
> > Also, Check the logs. both backgroundrb_debug_11006.log and
> > backgroundrb_11006.log and if there is any exception there.
> >
> > Lastly, what exactly you are trying to send from your worker code? You
> > realize, not everything in ruby can be dumped across wire.
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel


From gethemant at gmail.com  Thu Jul 17 13:31:59 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 17 Jul 2008 23:01:59 +0530
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <200807171847.54107.maze@strahlungsfrei.de>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
	<f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
	<200807171847.54107.maze@strahlungsfrei.de>
Message-ID: <fb283d5e0807171031t1867725are7676db396da4dc2@mail.gmail.com>

On Thu, Jul 17, 2008 at 10:17 PM, Martin Honermeyer
<maze at strahlungsfrei.de> wrote:
> I second that. I can remove all code from my worker so only the code from the
> generator is left.


> It hangs even when calling a not-existing worker. From the Rails console:

Please read below. There is no point in doing this kinda testing.

>
>>> MiddleMan.worker(:dfsfd)
> => #<BackgrounDRb::RailsWorkerProxy:0xb6af77e0
> @middle_man=#<BackgrounDRb::ClusterConnection:0xb76d06ac
> @backend_connections=[#<BackgrounDRb::Connection:0xb76d03b4
> @mutex=#<Mutex:0xb76d038c>,
> cluster_conn#<BackgrounDRb::ClusterConnection:0xb76d06ac ...,
> @server_port=11006, @server_ip="0.0.0.0", @connection_status=true>],
> request_count1, bdrb_servers[#<struct #<Class:0xb76d05e4 ip="0.0.0.0",
> port=11006], @round_robin=[0], @last_polled_time=Thu Jul 17 18:30:09 +0200
> 2008, @disconnected_connections={}>, worker_keynil, tried_connections[],
> worker_name:dfsfd
>
>>> MiddleMan.worker(:dfsfd).sdfsdf
>
> (hangs)
>
>
> I am calling from the console (didn't even try from within a controller).
> There is _nothing_ posted in the backgroundrb logs.
>
> For me it seems like there is no connection to the backgroundrb server at all.
> I even rebooted the (Xen virtual Linux) machine, but the problem remains.
>
>
> My background.yml:
>
> ---
> :backgroundrb:
>  :port: 11006
>  :ip: 0.0.0.0
>  :debug_log: false # whether to print debug logs to a seperate worker,
> defaults to true
>  :log: foreground # will print log messages to STDOUT, defaults to seperate
> log worker

You have disabled logging, so no wonder your logs are clean. Also,
when you invoke some call on a non-existent worker in a synchronous
manner, call will obviously hang, because there is no worker that can
respond to request. What you do expect there?

However, if you have logging enabled and start backgroundrb with
"./scrip/backgroundrb start", you can see an exception about
non-existent worker logged in log files.

From gethemant at gmail.com  Thu Jul 17 13:34:43 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 17 Jul 2008 23:04:43 +0530
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
	<fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
	<200807171431.57994.maze@strahlungsfrei.de>
	<fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
	<f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
Message-ID: <fb283d5e0807171034m1dec1fd1uef6c965278760f30@mail.gmail.com>

On Thu, Jul 17, 2008 at 9:52 PM, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
> Hemant,
>
> As mentioned in my first email, this happens even when your worker has no
> code (or only a logger.info line) and no arguments passed. Nothing shows up
> in the log, it just hangs.
>
> Please refer to my first email and let me know if you have any additional
> questions.
>
> Thank you for helping us with this issue.
>

Today I wrote this worker:

class HelloWorker < BackgrounDRb::MetaWorker
  set_worker_name :hello_worker
  def create(args = nil)
    # some crap
  end

  def make_request
    logger.info "hello world"
  end
end



And invoked task on the worker from console as you said:
>> MiddleMan.worker(:hello_worker).make_request
=> "dummy_result"

And I got a response immediately. So this obviously a Mac only issue,
now trouble is, I don't have Mac OSX for testing, so if anyone can
give me ssh connection for a while, I will be thankful.

From gethemant at gmail.com  Fri Jul 18 00:45:57 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 18 Jul 2008 10:15:57 +0530
Subject: [Backgroundrb-devel] Update your packet gem for Mac OS X issue
Message-ID: <fb283d5e0807172145m5aa5c1a7r18b4d2d746a4898e@mail.gmail.com>

Folks,

I have released a new verison of Packet(0.1.8), which has the fix for
Mac OSX, please update your packet gem for getting the fix.

Thanks.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Fri Jul 18 00:52:23 2008
From: gethemant at gmail.com (hemant)
Date: Fri, 18 Jul 2008 10:22:23 +0530
Subject: [Backgroundrb-devel] Fwd: Method hangs for the first time if
	called synchronously
In-Reply-To: <fb283d5e0807171034m1dec1fd1uef6c965278760f30@mail.gmail.com>
References: <f8a5dfb10807151251u1395855ap7539209c2c6580d6@mail.gmail.com>
	<fb283d5e0807161123m1752163eod690a532f442379@mail.gmail.com>
	<fd0631d50807161135g6439c1c0q32aec7001151abd9@mail.gmail.com>
	<200807171431.57994.maze@strahlungsfrei.de>
	<fb283d5e0807170644s7b4786a8v40ee78857390e2ab@mail.gmail.com>
	<f8a5dfb10807170922k75758262ya034c8eb67ff5f1a@mail.gmail.com>
	<fb283d5e0807171034m1dec1fd1uef6c965278760f30@mail.gmail.com>
Message-ID: <fb283d5e0807172152i68fda503p2afd340cb2ff259a@mail.gmail.com>

Please, update your packet gem to (0.1.8) and issue should go away.

Thanks.

On Thu, Jul 17, 2008 at 11:04 PM, hemant <gethemant at gmail.com> wrote:
> On Thu, Jul 17, 2008 at 9:52 PM, Csongor Gyuricza <csgyuricza at gmail.com> wrote:
>> Hemant,
>>
>> As mentioned in my first email, this happens even when your worker has no
>> code (or only a logger.info line) and no arguments passed. Nothing shows up
>> in the log, it just hangs.
>>
>> Please refer to my first email and let me know if you have any additional
>> questions.
>>
>> Thank you for helping us with this issue.
>>
>
> Today I wrote this worker:
>
> class HelloWorker < BackgrounDRb::MetaWorker
>  set_worker_name :hello_worker
>  def create(args = nil)
>    # some crap
>  end
>
>  def make_request
>    logger.info "hello world"
>  end
> end
>
>
>
> And invoked task on the worker from console as you said:
>>> MiddleMan.worker(:hello_worker).make_request
> => "dummy_result"
>
> And I got a response immediately. So this obviously a Mac only issue,
> now trouble is, I don't have Mac OSX for testing, so if anyone can
> give me ssh connection for a while, I will be thankful.
>



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From derrek at ridecharge.com  Sat Jul 19 14:44:00 2008
From: derrek at ridecharge.com (Derrek Long)
Date: Sat, 19 Jul 2008 11:44:00 -0700
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
	can't figure a work around
Message-ID: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>

Hello,

I'm trying to learn how to use BackgroundRB to run a long running process while showing status to the user via a webpage.
1. I tried using Advanced Rails Recipe 43, however, it appears as though backgroundrb has been updated since that book was published (2 months ago) as methods like "register_status", "ask_stutus" no longer exist.
2. I tried reading the everything on http://backgroundrb.rubyforge.org/
3. I tried to update the recipe code with what I learned from reading the backgroundrb webpage
4. I failed spectacularly

Here's my code reworked to be simpler:

---------------
class BillingWorker < BackgrounDRb::MetaWorker
  set_worker_name :billing_worker
  set_no_auto_load(true)

  def create(args = nil)
    cache[worker_key] = 0
    index = 0
    args.times do
      sleep(4)
      index = index + 1
      cache[worker_key] =(index * 100) / args
    end
    exit
  end
end

class PaymentsController < ApplicationController
  def async_test
    session[:job_key] = Bill.find(1).start_test
    redirect_to :action => 'check_async_test_status'
  end
  def check_async_test_status
    @percent_complete = Bill.test_status(session[:job_key])
    if request.xhr?
      if @percent_complete == 100
        render :update do |page|
          flash[:notice] = "test is complete"
          session[:job_key] = nil
          page.redirect_to :action => "pending"
        end
      else
        render :update do |page|
          page[:billingStatus].setStyle :width => "#{@percent_complete *2}px"
          page[:billingStatus].replace_html "#{@percent_complete}%"
        end
      end
    end
  end
end

class Bill < ActiveRecord::Base
 def start_test
    MiddleMan.new_worker(:worker => :billing_worker, :worker_key => "foobar", :data => 5)
  end

  def self.test_status(job_key)
    MiddleMan.worker(:billing_worker, "foobar").ask_result(job_key)
    # also tried MiddleMan.worker(:billing_worker).ask_result(job_key)
  end
end
---------------

hitting /payments/async_test always results in
Packet::InvalidWorker
/usr/local/lib/ruby/gems/1.8/gems/packet-0.1.8/lib/packet/packet_connection.rb:52:in `ask_worker'
/home/derrek/repos/prod/riderway/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in `get_result_object'

I can't seem to get at the cache[worker_key] value.  Any help would be greatly appreciated.

Thanks,
-Derrek

From gethemant at gmail.com  Sat Jul 19 15:29:22 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 20 Jul 2008 00:59:22 +0530
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
	can't figure a work around
In-Reply-To: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
References: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
Message-ID: <fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>

On Sun, Jul 20, 2008 at 12:14 AM, Derrek Long <derrek at ridecharge.com> wrote:
> Hello,
>
> I'm trying to learn how to use BackgroundRB to run a long running process while showing status to the user via a webpage.
> 1. I tried using Advanced Rails Recipe 43, however, it appears as though backgroundrb has been updated since that book was published (2 months ago) as methods like "register_status", "ask_stutus" no longer exist.

Being one of the contributors for the recipe. I am terribly sorry for
this, but newer caching policies are a lot more stable. I am trying to
talk with Mike for releasing a new pdf that covers new version.


> 2. I tried reading the everything on http://backgroundrb.rubyforge.org/
> 3. I tried to update the recipe code with what I learned from reading the backgroundrb webpage
> 4. I failed spectacularly
>
> Here's my code reworked to be simpler:
>
> ---------------
> class BillingWorker < BackgrounDRb::MetaWorker
>  set_worker_name :billing_worker
>  set_no_auto_load(true)
>
>  def create(args = nil)
>    cache[worker_key] = 0
>    index = 0
>    args.times do
>      sleep(4)
>      index = index + 1
>      cache[worker_key] =(index * 100) / args
>    end
>    exit
>  end
> end
>
> class PaymentsController < ApplicationController
>  def async_test
>    session[:job_key] = Bill.find(1).start_test
>    redirect_to :action => 'check_async_test_status'
>  end
>  def check_async_test_status
>    @percent_complete = Bill.test_status(session[:job_key])
>    if request.xhr?
>      if @percent_complete == 100
>        render :update do |page|
>          flash[:notice] = "test is complete"
>          session[:job_key] = nil
>          page.redirect_to :action => "pending"
>        end
>      else
>        render :update do |page|
>          page[:billingStatus].setStyle :width => "#{@percent_complete *2}px"
>          page[:billingStatus].replace_html "#{@percent_complete}%"
>        end
>      end
>    end
>  end
> end
>
> class Bill < ActiveRecord::Base
>  def start_test
>    MiddleMan.new_worker(:worker => :billing_worker, :worker_key => "foobar", :data => 5)
>  end
>
>  def self.test_status(job_key)
>    MiddleMan.worker(:billing_worker, "foobar").ask_result(job_key)
>    # also tried MiddleMan.worker(:billing_worker).ask_result(job_key)
>  end
> end
> ---------------
>
> hitting /payments/async_test always results in
> Packet::InvalidWorker
> /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.8/lib/packet/packet_connection.rb:52:in `ask_worker'
> /home/derrek/repos/prod/riderway/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in `get_result_object'
>
> I can't seem to get at the cache[worker_key] value.  Any help would be greatly appreciated.

Two things are causing problem:

1. MiddleMan.new_worker(...) was not returning the "worker_key" with
which it was created and hence, your session[:job_key] was probably
wrong. I fixed it in latest git push and have added testcase to cover
that also, hence shouldn't happen again.

2. Now, in your worker, you are using worker_key to cache the result:
cache[worker_key] = 0

Think of cache object as hash and whatever key you used here, must be
used while getting the results back and hence your code for fetching
the results should be:

MiddleMan.worker(:billing_worker, "foobar").ask_result("foobar")

From robl at mail.pigdestroyer.co.uk  Sat Jul 19 16:02:01 2008
From: robl at mail.pigdestroyer.co.uk (Rob Lacey)
Date: Sat, 19 Jul 2008 21:02:01 +0100
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
 can't figure a work around
In-Reply-To: <fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>
References: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
	<fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>
Message-ID: <48824839.8090101@mail.pigdestroyer.co.uk>

I got the same problem when I was trying to implement something in production, having read the same Recipie. Exactly the same error in fact. I replicated it in the rails console when trying to query the worker. Dropping out of the console and starting again actually worked but of course doesn't help when you want to query straight away.

Packet::InvalidWorker

I got the impression that it was trying to query the worker just a fraction before the worker was ready. I couldn't gather much more information than that. In the end I had to abandon BackgrounDRb for my project and use a cronjob and basic queue due to time constraints, so I'd really like to know what the problem was so that I don't give up on BDRb alltogether. I have to say I found the rubyforge site documentation far too lacking in details so again I look forward to see more in depth examples.

If it is all fixed now then its a bit late for this project so maybe I'll give it another look for the next time it BDRb seems like an appropriate solution.

RobL


hemant wrote:
> On Sun, Jul 20, 2008 at 12:14 AM, Derrek Long <derrek at ridecharge.com> wrote:
>   
>> Hello,
>>
>> I'm trying to learn how to use BackgroundRB to run a long running process while showing status to the user via a webpage.
>> 1. I tried using Advanced Rails Recipe 43, however, it appears as though backgroundrb has been updated since that book was published (2 months ago) as methods like "register_status", "ask_stutus" no longer exist.
>>     
>
> Being one of the contributors for the recipe. I am terribly sorry for
> this, but newer caching policies are a lot more stable. I am trying to
> talk with Mike for releasing a new pdf that covers new version.
>
>
>   
>> 2. I tried reading the everything on http://backgroundrb.rubyforge.org/
>> 3. I tried to update the recipe code with what I learned from reading the backgroundrb webpage
>> 4. I failed spectacularly
>>
>> Here's my code reworked to be simpler:
>>
>> ---------------
>> class BillingWorker < BackgrounDRb::MetaWorker
>>  set_worker_name :billing_worker
>>  set_no_auto_load(true)
>>
>>  def create(args = nil)
>>    cache[worker_key] = 0
>>    index = 0
>>    args.times do
>>      sleep(4)
>>      index = index + 1
>>      cache[worker_key] =(index * 100) / args
>>    end
>>    exit
>>  end
>> end
>>
>> class PaymentsController < ApplicationController
>>  def async_test
>>    session[:job_key] = Bill.find(1).start_test
>>    redirect_to :action => 'check_async_test_status'
>>  end
>>  def check_async_test_status
>>    @percent_complete = Bill.test_status(session[:job_key])
>>    if request.xhr?
>>      if @percent_complete == 100
>>        render :update do |page|
>>          flash[:notice] = "test is complete"
>>          session[:job_key] = nil
>>          page.redirect_to :action => "pending"
>>        end
>>      else
>>        render :update do |page|
>>          page[:billingStatus].setStyle :width => "#{@percent_complete *2}px"
>>          page[:billingStatus].replace_html "#{@percent_complete}%"
>>        end
>>      end
>>    end
>>  end
>> end
>>
>> class Bill < ActiveRecord::Base
>>  def start_test
>>    MiddleMan.new_worker(:worker => :billing_worker, :worker_key => "foobar", :data => 5)
>>  end
>>
>>  def self.test_status(job_key)
>>    MiddleMan.worker(:billing_worker, "foobar").ask_result(job_key)
>>    # also tried MiddleMan.worker(:billing_worker).ask_result(job_key)
>>  end
>> end
>> ---------------
>>
>> hitting /payments/async_test always results in
>> Packet::InvalidWorker
>> /usr/local/lib/ruby/gems/1.8/gems/packet-0.1.8/lib/packet/packet_connection.rb:52:in `ask_worker'
>> /home/derrek/repos/prod/riderway/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:123:in `get_result_object'
>>
>> I can't seem to get at the cache[worker_key] value.  Any help would be greatly appreciated.
>>     
>
> Two things are causing problem:
>
> 1. MiddleMan.new_worker(...) was not returning the "worker_key" with
> which it was created and hence, your session[:job_key] was probably
> wrong. I fixed it in latest git push and have added testcase to cover
> that also, hence shouldn't happen again.
>
> 2. Now, in your worker, you are using worker_key to cache the result:
> cache[worker_key] = 0
>
> Think of cache object as hash and whatever key you used here, must be
> used while getting the results back and hence your code for fetching
> the results should be:
>
> MiddleMan.worker(:billing_worker, "foobar").ask_result("foobar")
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>   


From gethemant at gmail.com  Sat Jul 19 23:50:12 2008
From: gethemant at gmail.com (hemant)
Date: Sun, 20 Jul 2008 09:20:12 +0530
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
	can't figure a work around
In-Reply-To: <48824839.8090101@mail.pigdestroyer.co.uk>
References: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
	<fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>
	<48824839.8090101@mail.pigdestroyer.co.uk>
Message-ID: <fb283d5e0807192050w36089d92s14a055bd280d54fb@mail.gmail.com>

On Sun, Jul 20, 2008 at 1:32 AM, Rob Lacey <robl at mail.pigdestroyer.co.uk> wrote:
> I got the same problem when I was trying to implement something in
> production, having read the same Recipie. Exactly the same error in fact. I
> replicated it in the rails console when trying to query the worker. Dropping
> out of the console and starting again actually worked but of course doesn't
> help when you want to query straight away.
>
> Packet::InvalidWorker
>
> I got the impression that it was trying to query the worker just a fraction
> before the worker was ready. I couldn't gather much more information than
> that. In the end I had to abandon BackgrounDRb for my project and use a
> cronjob and basic queue due to time constraints, so I'd really like to know
> what the problem was so that I don't give up on BDRb alltogether. I have to
> say I found the rubyforge site documentation far too lacking in details so
> again I look forward to see more in depth examples.
>
> If it is all fixed now then its a bit late for this project so maybe I'll
> give it another look for the next time it BDRb seems like an appropriate
> solution.

I have a fix for Linux for the above mentioned problem, looking for a
fix for OS X though. Basically, it boils down to "read_nonblock()"
doesn't behave in OS X, as it should. In OSX, read_nonblock ,
*blocks*, hence we had to switch to recv_nonblock(), which doesn't
block in both Linux and OS X. But recv_nonblock is a something of a
beast and doesn't work well for newly created sockets.

From kieran776 at gmail.com  Mon Jul 21 00:55:36 2008
From: kieran776 at gmail.com (Kieran P)
Date: Mon, 21 Jul 2008 16:55:36 +1200
Subject: [Backgroundrb-devel] Accessing cached information during a
	backgroundrb process
Message-ID: <fc3b32cf0807202155k2315a87t4a476e87de810859@mail.gmail.com>

Hello,

We've just updated an application from an older version of backgroundrb to
the most recent code available on github.

And most of the code is now working as expected, except for one area which
is fairly important.

We pass the task off to an async function, which proceeds to the next page,
on which we have a regular get_status function running. It used to query the
worker, get the current information from what was set by register_status and
then update the page based on the information it received back.

I see that function has been removed in the recent code, and replaced with
ask_result. And while this works to get information back at the end of a
backgroundrb process, it returns nil if you ask_result while its still
running.

I've read over the docs, and am fairly certain I've got the syntax right.

Is there a function I'm not seeing, that would allow us to be able to get
the information we need during a backgroundrb process? And if not, is there
a suggested way to get around this?

Any help would be greatly appreciated.

-- 
Regards
Kieran
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080721/4b58ebb1/attachment.html>

From andreamorandini at gmail.com  Mon Jul 21 04:13:00 2008
From: andreamorandini at gmail.com (Andrea Morandini)
Date: Mon, 21 Jul 2008 10:13:00 +0200
Subject: [Backgroundrb-devel] Backgroundrb problem in linux
Message-ID: <f3485ab70807210113q3729e99cta33447d7b2e9e850@mail.gmail.com>

Hi all,        I am developing a worker which should perform a periodic
task.
Everything is fine when I'm testing it in development mode (mac os x, ruby
1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]) but when I move to
production (ruby 1.8.6 (2008-03-03 patchlevel 114) [i486-linux]) using
"script/bacgroundrb start -e production" the background server log reports
these errors:

/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- log_worker (LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:35:in
`load_worker'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:26:in
`initialize'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47:in
`new'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47
from /usr/bin/packet_worker_runner:19:in `load'
from /usr/bin/packet_worker_runner:19
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- backgroundrb (LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from
/var/www/reputation/releases/20080717100533/lib/workers/drb_scraper_worker.rb:2
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require'
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:31:in
`load_worker'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:26:in
`initialize'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47:in
`new'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47
from /usr/bin/packet_worker_runner:19:in `load'
from /usr/bin/packet_worker_runner:19


The project is versioned and I'm using capistrano.

Any suggestion?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080721/c09dd02e/attachment.html>

From gethemant at gmail.com  Mon Jul 21 07:28:18 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 21 Jul 2008 16:58:18 +0530
Subject: [Backgroundrb-devel] Fwd: Accessing cached information during a
	backgroundrb process
In-Reply-To: <fc3b32cf0807210247u4e8a25e2qccd2410872c17383@mail.gmail.com>
References: <fc3b32cf0807202155k2315a87t4a476e87de810859@mail.gmail.com>
	<fb283d5e0807202229l4dc43eaegcaf461a4e109d236@mail.gmail.com>
	<fc3b32cf0807210247u4e8a25e2qccd2410872c17383@mail.gmail.com>
Message-ID: <fb283d5e0807210428q70c0054crbea3f5aabdb1dcc8@mail.gmail.com>

Sorry the thread went off the mailing list, when I clicked reply,
rather than reply all.



---------- Forwarded message ----------
From: Kieran P <kieran776 at gmail.com>
Date: Mon, Jul 21, 2008 at 3:17 PM
Subject: Re: [Backgroundrb-devel] Accessing cached information during
a backgroundrb process
To: hemant <gethemant at gmail.com>


On Mon, Jul 21, 2008 at 5:29 PM, hemant <gethemant at gmail.com> wrote:
>
> On Mon, Jul 21, 2008 at 10:25 AM, Kieran P <kieran776 at gmail.com> wrote:
> > Hello,
> >
> > We've just updated an application from an older version of backgroundrb to
> > the most recent code available on github.
> >
> > And most of the code is now working as expected, except for one area which
> > is fairly important.
> >
> > We pass the task off to an async function, which proceeds to the next page,
> > on which we have a regular get_status function running. It used to query the
> > worker, get the current information from what was set by register_status and
> > then update the page based on the information it received back.
> >
> > I see that function has been removed in the recent code, and replaced with
> > ask_result. And while this works to get information back at the end of a
> > backgroundrb process, it returns nil if you ask_result while its still
> > running.
> >
> > I've read over the docs, and am fairly certain I've got the syntax right.
> >
> > Is there a function I'm not seeing, that would allow us to be able to get
> > the information we need during a backgroundrb process? And if not, is there
> > a suggested way to get around this?
> >
> > Any help would be greatly appreciated.
>
> Previously all the cached results were being stored in master process
> and hence even when worker is processing certain requests, you were
> able to fetch the results back. Now this had its own trade-off and
> register_status was never thread safe and hence cached results are by
> default now worker local data and when you invoke ask_result, it gets
> queried from worker (which may not be able to serve the data, because
> its processing some task).
>
> One very simple and effective workaround is to replace that cache with
> memcache ( actually any production app should use memcache rather than
> default cache that comes bundled). Its quite easy, just update your
> backgroundrb.yml with:
>
> :backgroundrb:
>  :ip: 0.0.0.0 #ip on which backgroundrb server is running
>  :port: 11006 #port on which backgroundrb server is running
>  :result_storage: memcache # store results in a mecache cluster
>
> :memcache: "10.0.0.1:11211" #=> location of mecache clusters seperated by comma


Thank you hemant. Your solution worked perfectly, without any other
changes (with the exception of an IP change).

backgroundrb is now upgraded and working much smoother than before. Great work.

--
Regards
Kieran


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Mon Jul 21 07:30:41 2008
From: gethemant at gmail.com (hemant)
Date: Mon, 21 Jul 2008 17:00:41 +0530
Subject: [Backgroundrb-devel] Backgroundrb problem in linux
In-Reply-To: <f3485ab70807210113q3729e99cta33447d7b2e9e850@mail.gmail.com>
References: <f3485ab70807210113q3729e99cta33447d7b2e9e850@mail.gmail.com>
Message-ID: <fb283d5e0807210430y32885a73v68ff9bd47e9e0aa3@mail.gmail.com>

On Mon, Jul 21, 2008 at 1:43 PM, Andrea Morandini
<andreamorandini at gmail.com> wrote:
> Hi all,
>         I am developing a worker which should perform a periodic task.
> Everything is fine when I'm testing it in development mode (mac os x, ruby
> 1.8.6 (2008-03-03 patchlevel 114) [universal-darwin9.0]) but when I move to
> production (ruby 1.8.6 (2008-03-03 patchlevel 114) [i486-linux]) using
> "script/bacgroundrb start -e production" the background server log reports
> these errors:
> /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> `gem_original_require': no such file to load -- log_worker (LoadError)
> from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:35:in
> `load_worker'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:26:in
> `initialize'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47:in
> `new'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47
> from /usr/bin/packet_worker_runner:19:in `load'
> from /usr/bin/packet_worker_runner:19
> /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> `gem_original_require': no such file to load -- backgroundrb (LoadError)
> from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
> from
> /var/www/reputation/releases/20080717100533/lib/workers/drb_scraper_worker.rb:2
> from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in
> `gem_original_require'
> from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:31:in
> `load_worker'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:26:in
> `initialize'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47:in
> `new'
> from /usr/lib/ruby/gems/1.8/gems/packet-0.1.7/bin/packet_worker_runner:47
> from /usr/bin/packet_worker_runner:19:in `load'
> from /usr/bin/packet_worker_runner:19
>
> The project is versioned and I'm using capistrano.

Autogenerated old scripts like "backgroundrb" and "load_worker_env.rb"
are probably still there in script directory of your rails app. remove
then and run rake backgroundrb:setup again.

From derrek at ridecharge.com  Mon Jul 21 10:49:31 2008
From: derrek at ridecharge.com (Derrek Long)
Date: Mon, 21 Jul 2008 07:49:31 -0700
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
 can't figure a work around
In-Reply-To: <fb283d5e0807192050w36089d92s14a055bd280d54fb@mail.gmail.com>
References: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
	<fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>
	<48824839.8090101@mail.pigdestroyer.co.uk>
	<fb283d5e0807192050w36089d92s14a055bd280d54fb@mail.gmail.com>
Message-ID: <8123B0AED774854D9A709CFC2C5E2D5672E903710C@EXVMBX015-1.exch015.msoutlookonline.net>

> I have a fix for Linux for the above mentioned problem, looking for a
>fix for OS X though. Basically, it boils down to "read_nonblock()"
>doesn't behave in OS X, as it should. In OSX, read_nonblock ,
>*blocks*, hence we had to switch to recv_nonblock(), which doesn't
>block in both Linux and OS X. But recv_nonblock is a something of a
>beast and doesn't work well for newly created sockets.

I pulled from git this morning and just spent another 20 minutes but couldn't get anything but the Packet::InvalidWorker error.  Is there any example of working code I can look at that accesses the backgroundrb cache?

My system is Ubuntu 7.10, rails 2.1, and ruby 1.8.6.

Thanks,
-Derrek

From eduardodmz at gmail.com  Mon Jul 21 16:22:19 2008
From: eduardodmz at gmail.com (Eduardo Dominguez)
Date: Mon, 21 Jul 2008 15:22:19 -0500
Subject: [Backgroundrb-devel] No such file: packet_worker_runner
Message-ID: <bc4bc5d00807211322x19cf4962t8c8dabdce60d89@mail.gmail.com>

I'm running the lastest BackgroundRB and I'm getting the following
error after an upgrade from a months old version of BackgroundRB. I
created the bdrb_job_queues table and upgrade the scripts:

var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
`exec': No such file or directory - packet_worker_runner
7:6:log_worker:17:/var/www/lapp/releases/20080721201748/lib/workers:/var/www/lapp/releases/20080721201748/script/load_worker_env
(Errno::ENOENT)
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
`fork_and_load'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:80:in
`start_worker'
	from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:16:in
`initialize'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:19:in
`run'
	from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
`initialize'
	from script/backgroundrb:68:in `new'
	from script/backgroundrb:68
/var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
`exec': No such file or directory - packet_worker_runner
10:9:report_worker:4:/var/www/lapp/releases/20080721201748/lib/workers:/var/www/lapp/releases/20080721201748/script/load_worker_env
(Errno::ENOENT)
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
`fork_and_load'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:68:in
`load_workers'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:63:in
`each'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:63:in
`load_workers'
	from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:20:in
`run'
	from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
`initialize'
	from script/backgroundrb:68:in `new'
	from script/backgroundrb:68


-- 
Lalo

From joost at spacebabies.nl  Mon Jul 21 16:53:03 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Mon, 21 Jul 2008 22:53:03 +0200
Subject: [Backgroundrb-devel] Scheduling one-off tasks
Message-ID: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>

I am building a Rails website which has content that expires at some  
time in the future.

Since this content is cached, by the time it expires I would like a  
method to sweep the cache at that time. Since it's based on a date, I  
cannot use the Rails app itself. I could use polling or sweep the  
cache every 5 minutes, but maybe there's a better approach?

I would like to know if I can use backgroundrb's at/cron-like features  
to schedule one-off execution of a piece of code at a date/time in the  
future. This would be a sweeper that runs at the expires_at time of my  
content.


I realise I can build a worker that goes into a sleep/wait loop until  
the desired time, but that's not really better than polling.


Could backgroundrb be helpful?

From gethemant at gmail.com  Tue Jul 22 00:59:06 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 10:29:06 +0530
Subject: [Backgroundrb-devel] Packet release 0.1.9 for Linux
Message-ID: <fb283d5e0807212159r17b91c29v5c0a6ea3cb198df5@mail.gmail.com>

Hi Folks,

As discussed previously, you might have observed that when you start a
worker dynamically, you may not be able to interact with it
immediately (Rob, I am talking to you. ;)), this release fixes that
and you should be able to interact with the worker immediately.

BUT, the fix only works in Linux and not on Mac OSX, where
Socket#read_nonblock seems to be broken and we use recv_nonblock(). If
you are using Linux in your production please update packet gem.




-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

From gethemant at gmail.com  Tue Jul 22 01:00:14 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 10:30:14 +0530
Subject: [Backgroundrb-devel] Advanced Rails Recipe 43 doesn't work,
	can't figure a work around
In-Reply-To: <8123B0AED774854D9A709CFC2C5E2D5672E903710C@EXVMBX015-1.exch015.msoutlookonline.net>
References: <8123B0AED774854D9A709CFC2C5E2D5672E92337CC@EXVMBX015-1.exch015.msoutlookonline.net>
	<fb283d5e0807191229k152f05e6i7b5d8832718fb7cf@mail.gmail.com>
	<48824839.8090101@mail.pigdestroyer.co.uk>
	<fb283d5e0807192050w36089d92s14a055bd280d54fb@mail.gmail.com>
	<8123B0AED774854D9A709CFC2C5E2D5672E903710C@EXVMBX015-1.exch015.msoutlookonline.net>
Message-ID: <fb283d5e0807212200s4908bc3es8fc8c45973ffa433@mail.gmail.com>

On Mon, Jul 21, 2008 at 8:19 PM, Derrek Long <derrek at ridecharge.com> wrote:
>> I have a fix for Linux for the above mentioned problem, looking for a
>>fix for OS X though. Basically, it boils down to "read_nonblock()"
>>doesn't behave in OS X, as it should. In OSX, read_nonblock ,
>>*blocks*, hence we had to switch to recv_nonblock(), which doesn't
>>block in both Linux and OS X. But recv_nonblock is a something of a
>>beast and doesn't work well for newly created sockets.
>
> I pulled from git this morning and just spent another 20 minutes but couldn't get anything but the Packet::InvalidWorker error.  Is there any example of working code I can look at that accesses the backgroundrb cache?
>
> My system is Ubuntu 7.10, rails 2.1, and ruby 1.8.6.

Please update to packet 0.1.9, it will work in Linux.

From gethemant at gmail.com  Tue Jul 22 01:01:40 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 10:31:40 +0530
Subject: [Backgroundrb-devel] No such file: packet_worker_runner
In-Reply-To: <bc4bc5d00807211322x19cf4962t8c8dabdce60d89@mail.gmail.com>
References: <bc4bc5d00807211322x19cf4962t8c8dabdce60d89@mail.gmail.com>
Message-ID: <fb283d5e0807212201l3010d939oef99558f7f6ea83f@mail.gmail.com>

On Tue, Jul 22, 2008 at 1:52 AM, Eduardo Dominguez <eduardodmz at gmail.com> wrote:
> I'm running the lastest BackgroundRB and I'm getting the following
> error after an upgrade from a months old version of BackgroundRB. I
> created the bdrb_job_queues table and upgrade the scripts:
>
> var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
> `exec': No such file or directory - packet_worker_runner
> 7:6:log_worker:17:/var/www/lapp/releases/20080721201748/lib/workers:/var/www/lapp/releases/20080721201748/script/load_worker_env
> (Errno::ENOENT)
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
> `fork_and_load'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:80:in
> `start_worker'
>        from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:16:in
> `initialize'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:19:in
> `run'
>        from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
> `initialize'
>        from script/backgroundrb:68:in `new'
>        from script/backgroundrb:68
> /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
> `exec': No such file or directory - packet_worker_runner
> 10:9:report_worker:4:/var/www/lapp/releases/20080721201748/lib/workers:/var/www/lapp/releases/20080721201748/script/load_worker_env
> (Errno::ENOENT)
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:109:in
> `fork_and_load'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:68:in
> `load_workers'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:63:in
> `each'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:63:in
> `load_workers'
>        from /var/lib/gems/1.8/gems/packet-0.1.8/lib/packet/packet_master.rb:20:in
> `run'
>        from /var/www/lapp/releases/20080721201748/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in
> `initialize'
>        from script/backgroundrb:68:in `new'
>        from script/backgroundrb:68
>

when packet gem, gets installed, it installs an executable called
"packet_worker_runner" for starting workers.
Can you make sure that, its in path when you start backgroundrb?

From gethemant at gmail.com  Tue Jul 22 01:04:58 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 10:34:58 +0530
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
Message-ID: <fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>

On Tue, Jul 22, 2008 at 2:23 AM, Joost Baaij <joost at spacebabies.nl> wrote:
> I am building a Rails website which has content that expires at some time in
> the future.
>
> Since this content is cached, by the time it expires I would like a method
> to sweep the cache at that time. Since it's based on a date, I cannot use
> the Rails app itself. I could use polling or sweep the cache every 5
> minutes, but maybe there's a better approach?
>
> I would like to know if I can use backgroundrb's at/cron-like features to
> schedule one-off execution of a piece of code at a date/time in the future.
> This would be a sweeper that runs at the expires_at time of my content.
>
>
> I realise I can build a worker that goes into a sleep/wait loop until the
> desired time, but that's not really better than polling.

You don't need an explicit sleep in worker, it will wait anyways on select().

Also, by "one off", you mean, a worker that starts when on schedule
and stops after that, you can use "reload_on_schedule true" in worker.

From joost at spacebabies.nl  Tue Jul 22 02:19:16 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Tue, 22 Jul 2008 08:19:16 +0200
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
	<fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>
Message-ID: <16682F42-73A4-4760-8E70-298427378CAC@spacebabies.nl>


Op 22 jul 2008, om 07:04 heeft hemant het volgende geschreven:

> On Tue, Jul 22, 2008 at 2:23 AM, Joost Baaij <joost at spacebabies.nl>  
> wrote:
>> I am building a Rails website which has content that expires at  
>> some time in
>> the future.
>>
>> I realise I can build a worker that goes into a sleep/wait loop  
>> until the
>> desired time, but that's not really better than polling.
>
> You don't need an explicit sleep in worker, it will wait anyways on  
> select().

I don't understand, which select are you referring to?

Is that a method somewhere in backgroundrb?

> Also, by "one off", you mean, a worker that starts when on schedule
> and stops after that, you can use "reload_on_schedule true" in worker.

Not entirely what I mean, I mean I only need to execute one command at  
one time in the future.

So for example I only need to call the Rails method  
expire_fragment('sidebar') at 2008-07-24 21:00:00 as a one-time event.

From gethemant at gmail.com  Tue Jul 22 02:22:51 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 11:52:51 +0530
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <16682F42-73A4-4760-8E70-298427378CAC@spacebabies.nl>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
	<fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>
	<16682F42-73A4-4760-8E70-298427378CAC@spacebabies.nl>
Message-ID: <fb283d5e0807212322m50e23328h48c9aef5828a9be5@mail.gmail.com>

On Tue, Jul 22, 2008 at 11:49 AM, Joost Baaij <joost at spacebabies.nl> wrote:
>
> I don't understand, which select are you referring to?
>
> Is that a method somewhere in backgroundrb?

I meant select system call, which does IO multiplexing.

>
>> Also, by "one off", you mean, a worker that starts when on schedule
>> and stops after that, you can use "reload_on_schedule true" in worker.
>
> Not entirely what I mean, I mean I only need to execute one command at one
> time in the future.
>
> So for example I only need to call the Rails method
> expire_fragment('sidebar') at 2008-07-24 21:00:00 as a one-time event.

use add_timer() { foobar() }

Thats your one off timer.

From joost at spacebabies.nl  Tue Jul 22 03:05:01 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Tue, 22 Jul 2008 09:05:01 +0200
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <fb283d5e0807212322m50e23328h48c9aef5828a9be5@mail.gmail.com>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
	<fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>
	<16682F42-73A4-4760-8E70-298427378CAC@spacebabies.nl>
	<fb283d5e0807212322m50e23328h48c9aef5828a9be5@mail.gmail.com>
Message-ID: <1F39BB29-63A1-4EDC-9B0A-8F261ADF62B3@spacebabies.nl>


Op 22 jul 2008, om 08:22 heeft hemant het volgende geschreven:

>> So for example I only need to call the Rails method
>> expire_fragment('sidebar') at 2008-07-24 21:00:00 as a one-time  
>> event.
>
> use add_timer() { foobar() }
>
> Thats your one off timer.

Throw me a bone here :) where is add_timer() defined? Ruby stdlib?

Or it is present in backgroundrb somewhere, it's not in the API docs  
(at least I can't find it).

From joost at spacebabies.nl  Tue Jul 22 04:13:45 2008
From: joost at spacebabies.nl (Joost Baaij)
Date: Tue, 22 Jul 2008 10:13:45 +0200
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <fb283d5e0807220011k611faa56tf32229e4d4efc928@mail.gmail.com>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
	<fb283d5e0807212204g5958e58aoc006385727fc76e9@mail.gmail.com>
	<16682F42-73A4-4760-8E70-298427378CAC@spacebabies.nl>
	<fb283d5e0807212322m50e23328h48c9aef5828a9be5@mail.gmail.com>
	<DF824251-637C-4C98-9244-83154F84D62B@spacebabies.nl>
	<fb283d5e0807220011k611faa56tf32229e4d4efc928@mail.gmail.com>
Message-ID: <270A0903-45EF-4C96-958C-0A905F330ECD@spacebabies.nl>


Op 22 jul 2008, om 09:11 heeft hemant het volgende geschreven:
>>>>
>>> use add_timer() { foobar() }
>>>
>>> Thats your one off timer.
>>
>> Throw me a bone here :) where is add_timer() defined? Ruby stdlib?
>>
>> Or it is present in backgroundrb somewhere, it's not in the API  
>> docs (at
>> least I can't find it).
>>
>
> http://backgroundrb.rubyforge.org/scheduling/
>
> Look at at beginning of the docs. ;)


I was completely blind! But now my eyes are open :-)

Thanks a lot!!

From mliu at biigroup.com  Tue Jul 22 05:49:15 2008
From: mliu at biigroup.com (=?GB2312?B?wfXD+ihMaXUgTWluZyk=?=)
Date: Tue, 22 Jul 2008 17:49:15 +0800
Subject: [Backgroundrb-devel] Background will make Mongrel too slow?
Message-ID: <dfe97ff30807220249y579a80a5td82b3fab66fdd1cc@mail.gmail.com>

Hello everyone,

I use BackgrounDRB to import some thing from excel to my railapp, I found
when the parsing is running, my mongrel process will very slow and make the
redirection from nginx timeout. Is it normal? And is there anyway to limit
the privicy of BackgrounDRB threads?

-- 
Liu Ming
Engineer
IPv6 Test Center
?BII Group Holdings Ltd(BII Group)

Address: Rm2508, Tower A, Times Fortune Building, No.6 Shuguang Xili,
Chaoyang District, Beijing, China
Postcode: 100028
Tel: 8610 58678188-170
Tax: 8610 58678466
Mobile: 86 13488685567
E-mail: mliu at biigroup.com
http://www.biigroup.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080722/bed0d913/attachment.html>

From akyrho at gmail.com  Tue Jul 22 09:37:52 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Tue, 22 Jul 2008 15:37:52 +0200
Subject: [Backgroundrb-devel] Still having trouble to load worker
Message-ID: <67b30a7b0807220637u1bb64318l16260caec08968d6@mail.gmail.com>

Hello everyone,

I still have trouble since the last SVN update. I desperate to instance a
backgroundrb worker this way :

MiddleMan.worker(:worker => :thumbnailer_worker, :worker_key =>
"thumbnailing_for_source_" + @source.id.to_s).async_generate_thumbnails(:arg
=> @source.id)

And all i got in my log file is this :

Invalid worker with name
workerthumbnailer_workerworker_keythumbnailing_for_source_13524 and key
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_connection.rb:52:in
`ask_worker'
/home/antz/
declix.com/current/vendor/plugins/backgroundrb/server/lib/master_worker.rb:103:in`async_method_invoke'
/home/antz/
declix.com/current/vendor/plugins/backgroundrb/server/lib/master_worker.rb:37:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_parser.rb:44:in
`extract'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_parser.rb:26:in
`loop'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_parser.rb:26:in
`extract'
/home/antz/
declix.com/current/vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:230:in
`read_external_socket'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:220:in
`handle_external_messages'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:194:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:190:in
`each'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:190:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:144:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:137:in
`loop'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_core.rb:137:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.9/lib/packet/packet_master.rb:21:in
`run'
/home/antz/
declix.com/current/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in`initialize'
script/backgroundrb:46:in `new'
script/backgroundrb:46
Client disconected
Packet::InvalidWorker

I think i read all about worker loading in the doc and on google.com but i
did not find anything to help me.

Thanks per advance.

-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080722/3334e6d9/attachment.html>

From gethemant at gmail.com  Tue Jul 22 10:23:11 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 19:53:11 +0530
Subject: [Backgroundrb-devel] Still having trouble to load worker
In-Reply-To: <67b30a7b0807220637u1bb64318l16260caec08968d6@mail.gmail.com>
References: <67b30a7b0807220637u1bb64318l16260caec08968d6@mail.gmail.com>
Message-ID: <fb283d5e0807220723g29e5b4e1yd209355699f28f6c@mail.gmail.com>

On 7/22/08, C?dric <akyrho at gmail.com> wrote:
> Hello everyone,
>
> I still have trouble since the last SVN update. I desperate to instance a
> backgroundrb worker this way :
>
> MiddleMan.worker(:worker => :thumbnailer_worker, :worker_key =>
> "thumbnailing_for_source_" +
> @source.id.to_s).async_generate_thumbnails(:arg =>
> @source.id)
>
> And all i got in my log file is this :

First, get *latest* version of backgroundrb from git, also, you can't
chain async_* like that, since in git version it will return
worker_key (which was intended behaviour, we had a bug in 1.0.4 where
a worker proxy was returned, rather than just worker key).

Second update to packet 0.1.9 and write above code like:


foo_key = MiddleMan.worker(:worker => :thumbnailer_worker, :worker_key
=> "thumbnailing_for_source_"+ at source.id.to_s)
MiddleMan.worker(:thumbnailer_worker,foo_key).async(arg)

From gethemant at gmail.com  Tue Jul 22 12:09:44 2008
From: gethemant at gmail.com (hemant)
Date: Tue, 22 Jul 2008 21:39:44 +0530
Subject: [Backgroundrb-devel] Still having trouble to load worker
In-Reply-To: <fb283d5e0807220723g29e5b4e1yd209355699f28f6c@mail.gmail.com>
References: <67b30a7b0807220637u1bb64318l16260caec08968d6@mail.gmail.com>
	<fb283d5e0807220723g29e5b4e1yd209355699f28f6c@mail.gmail.com>
Message-ID: <fb283d5e0807220909x4d112776n9f72d799c59d8319@mail.gmail.com>

On Tue, Jul 22, 2008 at 7:53 PM, hemant <gethemant at gmail.com> wrote:
> On 7/22/08, C?dric <akyrho at gmail.com> wrote:
>> Hello everyone,
>>
>> I still have trouble since the last SVN update. I desperate to instance a
>> backgroundrb worker this way :
>>
>> MiddleMan.worker(:worker => :thumbnailer_worker, :worker_key =>
>> "thumbnailing_for_source_" +
>> @source.id.to_s).async_generate_thumbnails(:arg =>
>> @source.id)
>>
>> And all i got in my log file is this :
>
> First, get *latest* version of backgroundrb from git, also, you can't
> chain async_* like that, since in git version it will return
> worker_key (which was intended behaviour, we had a bug in 1.0.4 where
> a worker proxy was returned, rather than just worker key).
>
> Second update to packet 0.1.9 and write above code like:
>
>
> foo_key = MiddleMan.worker(:worker => :thumbnailer_worker, :worker_key
> => "thumbnailing_for_source_"+ at source.id.to_s)
> MiddleMan.worker(:thumbnailer_worker,foo_key).async(:arg => arg)

Lastly, why not pass those args as argument to create method directly?

From peter.varga.levlista at gmail.com  Tue Jul 22 12:43:22 2008
From: peter.varga.levlista at gmail.com (Peter Varga)
Date: Tue, 22 Jul 2008 18:43:22 +0200
Subject: [Backgroundrb-devel] More data in schedule
Message-ID: <ea50ac660807220943y5c3c8574h1a5a649dd289f605@mail.gmail.com>

Hello!
How can I put more data in backgroundrb.yml, if I can?

Like that

:schedules:
  :expressz_worker:
    :sendexp:
        :trigger_args: */30 * * * * *
        :data: data1, data2

Thanks a lot!


Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080722/64995aa4/attachment.html>

From gethemant at gmail.com  Wed Jul 23 02:18:15 2008
From: gethemant at gmail.com (hemant)
Date: Wed, 23 Jul 2008 11:48:15 +0530
Subject: [Backgroundrb-devel] More data in schedule
In-Reply-To: <ea50ac660807220943y5c3c8574h1a5a649dd289f605@mail.gmail.com>
References: <ea50ac660807220943y5c3c8574h1a5a649dd289f605@mail.gmail.com>
Message-ID: <fb283d5e0807222318r15b0ea44ncfc3a3361455acba@mail.gmail.com>

On Tue, Jul 22, 2008 at 10:13 PM, Peter Varga
<peter.varga.levlista at gmail.com> wrote:
> Hello!
> How can I put more data in backgroundrb.yml, if I can?
> Like that
> :schedules:
>   :expressz_worker:
>     :sendexp:
>         :trigger_args: */30 * * * * *
>         :data: data1, data2

Not possible.

From akyrho at gmail.com  Thu Jul 24 05:46:02 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Thu, 24 Jul 2008 11:46:02 +0200
Subject: [Backgroundrb-devel] How does ask_result() work?
Message-ID: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>

Hi there,

I do not really understand how to use ask_result() method. I've got a
thumbnailing process fully managed with backgroundrb, and i would like to
notify the sender of the progress.

So, in my worker, i register the current status like this :

cache['thumbnailing_'+id.to_s] = '0,1,2'

And in my controller, i have this as a periodical call :

worker_status =
MiddleMan.worker(:thumbnailer_worker).ask_result(''thumbnailing_' +
params[:id].to_s).split(',')

I expect the return value as 0, 1 and 2 but it seems that ask_result() is
returning a nil value.

Thanks per advance.

-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080724/08d33632/attachment-0001.html>

From kieran776 at gmail.com  Thu Jul 24 06:55:04 2008
From: kieran776 at gmail.com (Kieran P)
Date: Thu, 24 Jul 2008 22:55:04 +1200
Subject: [Backgroundrb-devel] How does ask_result() work?
In-Reply-To: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
References: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
Message-ID: <fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>

Hello,

I had this same issue not more than 3 days ago. hemant responded with a
clear and effective solution.

See:

http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001911.html
and
http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001913.html

(scroll down on that last link, the thread went off the ML, but forward at
the end)

Hope this helps.

Regards
Kieran


On Thu, Jul 24, 2008 at 9:46 PM, C?dric <akyrho at gmail.com> wrote:

> Hi there,
>
> I do not really understand how to use ask_result() method. I've got a
> thumbnailing process fully managed with backgroundrb, and i would like to
> notify the sender of the progress.
>
> So, in my worker, i register the current status like this :
>
> cache['thumbnailing_'+id.to_s] = '0,1,2'
>
> And in my controller, i have this as a periodical call :
>
> worker_status =
> MiddleMan.worker(:thumbnailer_worker).ask_result(''thumbnailing_' +
> params[:id].to_s).split(',')
>
> I expect the return value as 0, 1 and 2 but it seems that ask_result() is
> returning a nil value.
>
> Thanks per advance.
>
> --
> Bousmanne C?dric
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080724/d88f2e9c/attachment.html>

From akyrho at gmail.com  Thu Jul 24 08:13:01 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Thu, 24 Jul 2008 14:13:01 +0200
Subject: [Backgroundrb-devel] How does ask_result() work?
In-Reply-To: <fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>
References: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
	<fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>
Message-ID: <67b30a7b0807240513h4d0192b9u1ab7d71478b0b1dc@mail.gmail.com>

Thank you Kieran, i tried out your solution, but it did not work for me.

I wonder if my syntax is right, could you copy/paste me a piece of your code
or explain me how to call worker responses correctly? I still retrieve a
"nil" answer :(

Thanks

On Thu, Jul 24, 2008 at 12:55 PM, Kieran P <kieran776 at gmail.com> wrote:

> Hello,
>
> I had this same issue not more than 3 days ago. hemant responded with a
> clear and effective solution.
>
> See:
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001911.html
> and
> http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001913.html
>
> (scroll down on that last link, the thread went off the ML, but forward at
> the end)
>
> Hope this helps.
>
> Regards
> Kieran
>
>
> On Thu, Jul 24, 2008 at 9:46 PM, C?dric <akyrho at gmail.com> wrote:
>
>> Hi there,
>>
>> I do not really understand how to use ask_result() method. I've got a
>> thumbnailing process fully managed with backgroundrb, and i would like to
>> notify the sender of the progress.
>>
>> So, in my worker, i register the current status like this :
>>
>> cache['thumbnailing_'+id.to_s] = '0,1,2'
>>
>> And in my controller, i have this as a periodical call :
>>
>> worker_status =
>> MiddleMan.worker(:thumbnailer_worker).ask_result(''thumbnailing_' +
>> params[:id].to_s).split(',')
>>
>> I expect the return value as 0, 1 and 2 but it seems that ask_result() is
>> returning a nil value.
>>
>> Thanks per advance.
>>
>> --
>> Bousmanne C?dric
>>
>


-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080724/d0e8c808/attachment.html>

From akyrho at gmail.com  Thu Jul 24 10:46:19 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Thu, 24 Jul 2008 16:46:19 +0200
Subject: [Backgroundrb-devel] How does ask_result() work?
In-Reply-To: <67b30a7b0807240513h4d0192b9u1ab7d71478b0b1dc@mail.gmail.com>
References: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
	<fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>
	<67b30a7b0807240513h4d0192b9u1ab7d71478b0b1dc@mail.gmail.com>
Message-ID: <67b30a7b0807240746g60106ea5p43e959e31a011560@mail.gmail.com>

As requested on IRC, here's my code.

The worker :

---
require 'uri'
require 'net/http'
require 'RMagick'
require 'open-uri'
require 'rexml/document'

class ThumbnailerWorker < BackgrounDRb::MetaWorker
  set_worker_name :thumbnailer_worker
  set_no_auto_load true

  def create(args = nil)
    logger.info("[DEBUG] starting a thumbnailer worker (Time: " +
Time.now.to_s + ")")
    generate_thumbnails(args)
  end

  def generate_thumbnails(args = nil) # args = @source.id
    logger.info("[DEBUG] generate_thumbnails(args = " + args.inspect + ")")
    @source_id = eval(args.to_s)
    cache['thumbnailing_for_source_'+ at source_id.to_s] = '0,0,0'

    thumbnails = Thumbnail.find(:all, :conditions => ['medias.source_id=?
AND thumbnails.code=?', @source_id, PROCESSING_STATUS], :order =>
"medias.created_at ASC", :include => { :media => :source })

    @failed = 0
    @success = 0
    @progress = 0
    @total = thumbnails.size unless thumbnails.nil?

    if @total.nil? or @total == 0
      cache['thumbnailing_for_source_'+ at source_id.to_s] =
'-2,'+ at progress.to_s+','+ at total.to_s
    end

    upload_directory = RAILS_ROOT + "/public/medias/"
    no_error_encountered = true

    for thumbnail in thumbnails
      break if !no_error_encountered

      if @progress == 0
        media_id = thumbnail.media_id
        begin
          Dir.mkdir(upload_directory + @source_id.to_s)
        end
      end

      begin
        url = URI.split(thumbnail.original.sub(' ', '%20'))

        resp = Net::HTTP.get_response(url[2], url[5])
        if resp.code.to_s == "200"
          http = Net::HTTP.start(url[2])
          resp = http.get(url[5])
          image_filename = upload_directory + thumbnail.id.to_s + '.tmp.jpg'
          open(image_filename, "w") { |file| file.write(resp.body) }
          pic = Magick::Image.read(image_filename).first
          GC.start
          maxwidth = 100
          maxheight = 100
          aspectratio = 1.0
          imgwidth = pic.columns
          imgheight = pic.rows
          imgratio = imgwidth.to_f / imgheight.to_f
          scaleratio = imgratio > aspectratio ? (maxwidth.to_f / imgwidth) :
(maxheight.to_f / imgheight)
          thumb = pic.thumbnail(scaleratio)
          thumb.write(thumbnail_location = upload_directory +
@source_id.to_s + '/' + thumbnail.id.to_s + '.jpg')
          File.delete(image_filename)
          no_error_encountered = File.exist?(thumbnail_location)
          @success += 1
        else
          logger.info("[WARN] thumbnail for " + thumbnail.original + " not
generated. response = " + resp.code.to_s)
          @failed += 1
        end

        if no_error_encountered
          thumbnail.update_attribute("code", DEFAULT_STATUS)
        end

        sleep 0.5
      rescue Exception => exc
        logger.info("[WARN] exception caught in thumbnailer_worker: " +
exc.inspect)
        logger.info("[WARN] backtrace: " + exc.backtrace.inspect)
        @failed += 1 if resp.code.to_s == "200"
      end
      @progress += 1
      if @progress == @total
        cache['thumbnailing_for_source_'+ at source_id.to_s] =
'9999,'+ at progress.to_s+','+ at total.to_s
      else
        cache['thumbnailing_for_source_'+ at source_id.to_s] =
@progress.to_s+','+ at progress.to_s+','+ at total.to_s
        logger.info("cache['#{'thumbnailing_for_source_'+ at source_id.to_s}']
= #{cache['thumbnailing_for_source_'+ at source_id.to_s].inspect}")
      end
    end

    if no_error_encountered
      Media.update_all("code=#{DEFAULT_STATUS}", ["source_id=?",
@source_id])
      Source.update(@source_id, { "code" => DEFAULT_STATUS })
    else
      cache['thumbnailing_for_source_'+ at source_id.to_s] =
'-1,'+ at progress.to_s+','+ at total.to_s
    end

    return
  end

  def finish_work
    cache['thumbnailing_for_source_'+ at source_id.to_s] = '-9999,0,0'
    exit
  end
end
---

And the controller :

---
def submit
(some code)
                MiddleMan.new_worker(
                  :worker => :thumbnailer_worker,
                  :worker_key => "thumbnailing_for_source_" +
@source.id.to_s,
                  :data => @source.id)
(some more code)
end

def get_loaded_medias
        begin
          worker_status =
MiddleMan.worker(:thumbnailer_worker).ask_result("thumbnailing_for_source_#{params[:id]}").split(',')
          logger.info("[DEBUG] MiddleMan.ask_result =
#{worker_status.inspect}")
          status = worker_status[0].to_i
          progress = worker_status[1].to_i
          total = worker_status[2].to_i
        rescue Exception => exc
          render :update do |page|
            logger.info("[DEBUG] worker a renvoye une exception: " +
exc.message)
            page.replace_html "thumbnailing_progress", ""
          end
          return
        end
(some more code)
end
---

Hope it will help.

On Thu, Jul 24, 2008 at 2:13 PM, C?dric <akyrho at gmail.com> wrote:

> Thank you Kieran, i tried out your solution, but it did not work for me.
>
> I wonder if my syntax is right, could you copy/paste me a piece of your
> code or explain me how to call worker responses correctly? I still retrieve
> a "nil" answer :(
>
> Thanks
>
> On Thu, Jul 24, 2008 at 12:55 PM, Kieran P <kieran776 at gmail.com> wrote:
>
>> Hello,
>>
>> I had this same issue not more than 3 days ago. hemant responded with a
>> clear and effective solution.
>>
>> See:
>>
>> http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001911.html
>> and
>> http://rubyforge.org/pipermail/backgroundrb-devel/2008-July/001913.html
>>
>> (scroll down on that last link, the thread went off the ML, but forward at
>> the end)
>>
>> Hope this helps.
>>
>> Regards
>> Kieran
>>
>>
>> On Thu, Jul 24, 2008 at 9:46 PM, C?dric <akyrho at gmail.com> wrote:
>>
>>> Hi there,
>>>
>>> I do not really understand how to use ask_result() method. I've got a
>>> thumbnailing process fully managed with backgroundrb, and i would like to
>>> notify the sender of the progress.
>>>
>>> So, in my worker, i register the current status like this :
>>>
>>> cache['thumbnailing_'+id.to_s] = '0,1,2'
>>>
>>> And in my controller, i have this as a periodical call :
>>>
>>> worker_status =
>>> MiddleMan.worker(:thumbnailer_worker).ask_result(''thumbnailing_' +
>>> params[:id].to_s).split(',')
>>>
>>> I expect the return value as 0, 1 and 2 but it seems that ask_result() is
>>> returning a nil value.
>>>
>>> Thanks per advance.
>>>
>>> --
>>> Bousmanne C?dric
>>>
>>
>
>
> --
> Bousmanne C?dric
>
> Jabber / XMPP : akyrho at gmail.com
> Mail : akyrho at gmail.com
> Blog : http://www.parenthese.be/
>



-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080724/9060420f/attachment-0001.html>

From gethemant at gmail.com  Thu Jul 24 12:58:28 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 24 Jul 2008 22:28:28 +0530
Subject: [Backgroundrb-devel] How does ask_result() work?
In-Reply-To: <67b30a7b0807240746g60106ea5p43e959e31a011560@mail.gmail.com>
References: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
	<fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>
	<67b30a7b0807240513h4d0192b9u1ab7d71478b0b1dc@mail.gmail.com>
	<67b30a7b0807240746g60106ea5p43e959e31a011560@mail.gmail.com>
Message-ID: <fb283d5e0807240958t54238708y6b513c61e937723d@mail.gmail.com>

On Thu, Jul 24, 2008 at 8:16 PM, C?dric <akyrho at gmail.com> wrote:
> As requested on IRC, here's my code.
>
> The worker :
>
> ---
> require 'uri'
> require 'net/http'
> require 'RMagick'
> require 'open-uri'
> require 'rexml/document'
>
> class ThumbnailerWorker < BackgrounDRb::MetaWorker
>   set_worker_name :thumbnailer_worker
>   set_no_auto_load true
>
>   def create(args = nil)
>     logger.info("[DEBUG] starting a thumbnailer worker (Time: " +
> Time.now.to_s + ")")
>     generate_thumbnails(args)
>   end
>
>   def generate_thumbnails(args = nil) # args = @source.id
>     logger.info("[DEBUG] generate_thumbnails(args = " + args.inspect + ")")
>     @source_id = eval(args.to_s)
>     cache['thumbnailing_for_source_'+ at source_id.to_s] = '0,0,0'
>
>     thumbnails = Thumbnail.find(:all, :conditions => ['medias.source_id=?
> AND thumbnails.code=?', @source_id, PROCESSING_STATUS], :order =>
> "medias.created_at ASC", :include => { :media => :source })
>
>     @failed = 0
>     @success = 0
>     @progress = 0
>     @total = thumbnails.size unless thumbnails.nil?
>
>     if @total.nil? or @total == 0
>       cache['thumbnailing_for_source_'+ at source_id.to_s] =
> '-2,'+ at progress.to_s+','+ at total.to_s
>     end
>
>     upload_directory = RAILS_ROOT + "/public/medias/"
>     no_error_encountered = true
>
>     for thumbnail in thumbnails
>       break if !no_error_encountered
>
>       if @progress == 0
>         media_id = thumbnail.media_id
>         begin
>           Dir.mkdir(upload_directory + @source_id.to_s)
>         end
>       end
>
>       begin
>         url = URI.split(thumbnail.original.sub(' ', '%20'))
>
>         resp = Net::HTTP.get_response(url[2], url[5])
>         if resp.code.to_s == "200"
>           http = Net::HTTP.start(url[2])
>           resp = http.get(url[5])
>           image_filename = upload_directory + thumbnail.id.to_s + '.tmp.jpg'
>           open(image_filename, "w") { |file| file.write(resp.body) }
>           pic = Magick::Image.read(image_filename).first
>           GC.start
>           maxwidth = 100
>           maxheight = 100
>           aspectratio = 1.0
>           imgwidth = pic.columns
>           imgheight = pic.rows
>           imgratio = imgwidth.to_f / imgheight.to_f
>           scaleratio = imgratio > aspectratio ? (maxwidth.to_f / imgwidth) :
> (maxheight.to_f / imgheight)
>           thumb = pic.thumbnail(scaleratio)
>           thumb.write(thumbnail_location = upload_directory +
> @source_id.to_s + '/' + thumbnail.id.to_s + '.jpg')
>           File.delete(image_filename)
>           no_error_encountered = File.exist?(thumbnail_location)
>           @success += 1
>         else
>           logger.info("[WARN] thumbnail for " + thumbnail.original + " not
> generated. response = " + resp.code.to_s)
>           @failed += 1
>         end
>
>         if no_error_encountered
>           thumbnail.update_attribute("code", DEFAULT_STATUS)
>         end
>
>         sleep 0.5
>       rescue Exception => exc
>         logger.info("[WARN] exception caught in thumbnailer_worker: " +
> exc.inspect)
>         logger.info("[WARN] backtrace: " + exc.backtrace.inspect)
>         @failed += 1 if resp.code.to_s == "200"
>       end
>       @progress += 1
>       if @progress == @total
>         cache['thumbnailing_for_source_'+ at source_id.to_s] =
> '9999,'+ at progress.to_s+','+ at total.to_s
>       else
>         cache['thumbnailing_for_source_'+ at source_id.to_s] =
> @progress.to_s+','+ at progress.to_s+','+ at total.to_s
>         logger.info("cache['#{'thumbnailing_for_source_'+ at source_id.to_s}']
> = #{cache['thumbnailing_for_source_'+ at source_id.to_s].inspect}")
>       end
>     end
>
>     if no_error_encountered
>       Media.update_all("code=#{DEFAULT_STATUS}", ["source_id=?",
> @source_id])
>       Source.update(@source_id, { "code" => DEFAULT_STATUS })
>     else
>       cache['thumbnailing_for_source_'+ at source_id.to_s] =
> '-1,'+ at progress.to_s+','+ at total.to_s
>     end
>
>     return
>   end
>
>   def finish_work
>     cache['thumbnailing_for_source_'+ at source_id.to_s] = '-9999,0,0'
>     exit
>   end
> end
> ---
>
> And the controller :
>
> ---
> def submit
> (some code)
>                 MiddleMan.new_worker(
>                   :worker => :thumbnailer_worker,
>                   :worker_key => "thumbnailing_for_source_" +
> @source.id.to_s,
>                   :data => @source.id)
> (some more code)
> end
>
> def get_loaded_medias
>         begin
>           worker_status =
> MiddleMan.worker(:thumbnailer_worker).ask_result("thumbnailing_for_source_#{params[:id]}").split(',')


We have got a trouble in above code. Since you are creating a worker
dynamically using new_worker() method and using a worker key, you must
always, use that worker key to access that worker.

MiddleMan.worker(:thumbnailer_worker,<whatever_worker_key_you_used>).ask_result("thumbnailing_for_source_#{params[:id]}").split(',')

From akyrho at gmail.com  Fri Jul 25 03:52:21 2008
From: akyrho at gmail.com (=?ISO-8859-1?Q?C=E9dric?=)
Date: Fri, 25 Jul 2008 09:52:21 +0200
Subject: [Backgroundrb-devel] How does ask_result() work?
In-Reply-To: <fb283d5e0807240958t54238708y6b513c61e937723d@mail.gmail.com>
References: <67b30a7b0807240246u60e71b5ejc0db7be2b177d55e@mail.gmail.com>
	<fc3b32cf0807240355v7565fff0vf5f9b889d66fbbf9@mail.gmail.com>
	<67b30a7b0807240513h4d0192b9u1ab7d71478b0b1dc@mail.gmail.com>
	<67b30a7b0807240746g60106ea5p43e959e31a011560@mail.gmail.com>
	<fb283d5e0807240958t54238708y6b513c61e937723d@mail.gmail.com>
Message-ID: <67b30a7b0807250052v72db063br4d01a564733dd0de@mail.gmail.com>

Thank you Hemant, this is working perfectly now !


On Thu, Jul 24, 2008 at 6:58 PM, hemant <gethemant at gmail.com> wrote:

> On Thu, Jul 24, 2008 at 8:16 PM, C?dric <akyrho at gmail.com> wrote:
> > As requested on IRC, here's my code.
> >
> > The worker :
> >
> > ---
> > require 'uri'
> > require 'net/http'
> > require 'RMagick'
> > require 'open-uri'
> > require 'rexml/document'
> >
> > class ThumbnailerWorker < BackgrounDRb::MetaWorker
> >   set_worker_name :thumbnailer_worker
> >   set_no_auto_load true
> >
> >   def create(args = nil)
> >     logger.info("[DEBUG] starting a thumbnailer worker (Time: " +
> > Time.now.to_s + ")")
> >     generate_thumbnails(args)
> >   end
> >
> >   def generate_thumbnails(args = nil) # args = @source.id
> >     logger.info("[DEBUG] generate_thumbnails(args = " + args.inspect +
> ")")
> >     @source_id = eval(args.to_s)
> >     cache['thumbnailing_for_source_'+ at source_id.to_s] = '0,0,0'
> >
> >     thumbnails = Thumbnail.find(:all, :conditions => ['medias.source_id=?
> > AND thumbnails.code=?', @source_id, PROCESSING_STATUS], :order =>
> > "medias.created_at ASC", :include => { :media => :source })
> >
> >     @failed = 0
> >     @success = 0
> >     @progress = 0
> >     @total = thumbnails.size unless thumbnails.nil?
> >
> >     if @total.nil? or @total == 0
> >       cache['thumbnailing_for_source_'+ at source_id.to_s] =
> > '-2,'+ at progress.to_s+','+ at total.to_s
> >     end
> >
> >     upload_directory = RAILS_ROOT + "/public/medias/"
> >     no_error_encountered = true
> >
> >     for thumbnail in thumbnails
> >       break if !no_error_encountered
> >
> >       if @progress == 0
> >         media_id = thumbnail.media_id
> >         begin
> >           Dir.mkdir(upload_directory + @source_id.to_s)
> >         end
> >       end
> >
> >       begin
> >         url = URI.split(thumbnail.original.sub(' ', '%20'))
> >
> >         resp = Net::HTTP.get_response(url[2], url[5])
> >         if resp.code.to_s == "200"
> >           http = Net::HTTP.start(url[2])
> >           resp = http.get(url[5])
> >           image_filename = upload_directory + thumbnail.id.to_s +
> '.tmp.jpg'
> >           open(image_filename, "w") { |file| file.write(resp.body) }
> >           pic = Magick::Image.read(image_filename).first
> >           GC.start
> >           maxwidth = 100
> >           maxheight = 100
> >           aspectratio = 1.0
> >           imgwidth = pic.columns
> >           imgheight = pic.rows
> >           imgratio = imgwidth.to_f / imgheight.to_f
> >           scaleratio = imgratio > aspectratio ? (maxwidth.to_f /
> imgwidth) :
> > (maxheight.to_f / imgheight)
> >           thumb = pic.thumbnail(scaleratio)
> >           thumb.write(thumbnail_location = upload_directory +
> > @source_id.to_s + '/' + thumbnail.id.to_s + '.jpg')
> >           File.delete(image_filename)
> >           no_error_encountered = File.exist?(thumbnail_location)
> >           @success += 1
> >         else
> >           logger.info("[WARN] thumbnail for " + thumbnail.original + "
> not
> > generated. response = " + resp.code.to_s)
> >           @failed += 1
> >         end
> >
> >         if no_error_encountered
> >           thumbnail.update_attribute("code", DEFAULT_STATUS)
> >         end
> >
> >         sleep 0.5
> >       rescue Exception => exc
> >         logger.info("[WARN] exception caught in thumbnailer_worker: " +
> > exc.inspect)
> >         logger.info("[WARN] backtrace: " + exc.backtrace.inspect)
> >         @failed += 1 if resp.code.to_s == "200"
> >       end
> >       @progress += 1
> >       if @progress == @total
> >         cache['thumbnailing_for_source_'+ at source_id.to_s] =
> > '9999,'+ at progress.to_s+','+ at total.to_s
> >       else
> >         cache['thumbnailing_for_source_'+ at source_id.to_s] =
> > @progress.to_s+','+ at progress.to_s+','+ at total.to_s
> >         logger.info("cache['#{'thumbnailing_for_source_'+ at source_id.to_s
> }']
> > = #{cache['thumbnailing_for_source_'+ at source_id.to_s].inspect}")
> >       end
> >     end
> >
> >     if no_error_encountered
> >       Media.update_all("code=#{DEFAULT_STATUS}", ["source_id=?",
> > @source_id])
> >       Source.update(@source_id, { "code" => DEFAULT_STATUS })
> >     else
> >       cache['thumbnailing_for_source_'+ at source_id.to_s] =
> > '-1,'+ at progress.to_s+','+ at total.to_s
> >     end
> >
> >     return
> >   end
> >
> >   def finish_work
> >     cache['thumbnailing_for_source_'+ at source_id.to_s] = '-9999,0,0'
> >     exit
> >   end
> > end
> > ---
> >
> > And the controller :
> >
> > ---
> > def submit
> > (some code)
> >                 MiddleMan.new_worker(
> >                   :worker => :thumbnailer_worker,
> >                   :worker_key => "thumbnailing_for_source_" +
> > @source.id.to_s,
> >                   :data => @source.id)
> > (some more code)
> > end
> >
> > def get_loaded_medias
> >         begin
> >           worker_status =
> >
> MiddleMan.worker(:thumbnailer_worker).ask_result("thumbnailing_for_source_#{params[:id]}").split(',')
>
>
> We have got a trouble in above code. Since you are creating a worker
> dynamically using new_worker() method and using a worker key, you must
> always, use that worker key to access that worker.
>
>
> MiddleMan.worker(:thumbnailer_worker,<whatever_worker_key_you_used>).ask_result("thumbnailing_for_source_#{params[:id]}").split(',')
>



-- 
Bousmanne C?dric

Jabber / XMPP : akyrho at gmail.com
Mail : akyrho at gmail.com
Blog : http://www.parenthese.be/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080725/0fe070c0/attachment.html>

From daniel at newzwag.com  Fri Jul 25 15:39:30 2008
From: daniel at newzwag.com (Daniel Lockhart)
Date: Fri, 25 Jul 2008 12:39:30 -0700
Subject: [Backgroundrb-devel] Problem with worker servers
Message-ID: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>

I am a long time user of backgroundrb, and have used it's server  
methods (start_server and connect) to communicate data between various  
servers and applications.  It has been a great tool for this, and I  
feel fairly comfortable with how the server and client communicate.

With the latest version, however, I have encountered some problems.  I  
can successfully connect with 'connect' to a server started with  
'start_server', and the post_init function is called on both the  
server and client.   However, no data can be sent.  No matter who  
sends the data (server or client), the receive_data function is never  
called.  Prior to the update, this worked fine.  Is there anything new  
I should know about with this part of backgroundrb?  I didn't see  
anything mentioned in the release announcement, but before I start  
digging deeper into the packet 1.9 code, I thought I would ask here  
first.  Thanks in advance for any help!

Daniel

From gethemant at gmail.com  Fri Jul 25 15:53:29 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 26 Jul 2008 01:23:29 +0530
Subject: [Backgroundrb-devel] Problem with worker servers
In-Reply-To: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>
References: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>
Message-ID: <fb283d5e0807251253v1e5abf4egcf6294c73200cc3e@mail.gmail.com>

On Sat, Jul 26, 2008 at 1:09 AM, Daniel Lockhart <daniel at newzwag.com> wrote:
> I am a long time user of backgroundrb, and have used it's server methods
> (start_server and connect) to communicate data between various servers and
> applications.  It has been a great tool for this, and I feel fairly
> comfortable with how the server and client communicate.
>
> With the latest version, however, I have encountered some problems.  I can
> successfully connect with 'connect' to a server started with 'start_server',
> and the post_init function is called on both the server and client.
> However, no data can be sent.  No matter who sends the data (server or
> client), the receive_data function is never called.  Prior to the update,
> this worked fine.  Is there anything new I should know about with this part
> of backgroundrb?  I didn't see anything mentioned in the release
> announcement, but before I start digging deeper into the packet 1.9 code, I
> thought I would ask here first.  Thanks in advance for any help!

Which OS?

From daniel at newzwag.com  Fri Jul 25 16:01:37 2008
From: daniel at newzwag.com (Daniel Lockhart)
Date: Fri, 25 Jul 2008 13:01:37 -0700
Subject: [Backgroundrb-devel] Problem with worker servers
In-Reply-To: <fb283d5e0807251253v1e5abf4egcf6294c73200cc3e@mail.gmail.com>
References: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>
	<fb283d5e0807251253v1e5abf4egcf6294c73200cc3e@mail.gmail.com>
Message-ID: <6B8CAF0B-727E-4F46-BCFC-4832B132227F@newzwag.com>

Both servers are RHLE, one Red Hat 5, the other 4.  The kernel on the  
4 is 2.6.9-55.0.9.ELsmp which is 32 bit.  The kernel on the 5 is  
2.6.18-92.el5, which is 64 bit.

Thanks again.


On Jul 25, 2008, at 12:53 PM, hemant wrote:

> On Sat, Jul 26, 2008 at 1:09 AM, Daniel Lockhart  
> <daniel at newzwag.com> wrote:
>> I am a long time user of backgroundrb, and have used it's server  
>> methods
>> (start_server and connect) to communicate data between various  
>> servers and
>> applications.  It has been a great tool for this, and I feel fairly
>> comfortable with how the server and client communicate.
>>
>> With the latest version, however, I have encountered some  
>> problems.  I can
>> successfully connect with 'connect' to a server started with  
>> 'start_server',
>> and the post_init function is called on both the server and client.
>> However, no data can be sent.  No matter who sends the data (server  
>> or
>> client), the receive_data function is never called.  Prior to the  
>> update,
>> this worked fine.  Is there anything new I should know about with  
>> this part
>> of backgroundrb?  I didn't see anything mentioned in the release
>> announcement, but before I start digging deeper into the packet 1.9  
>> code, I
>> thought I would ask here first.  Thanks in advance for any help!
>
> Which OS?
>


From gethemant at gmail.com  Fri Jul 25 16:28:54 2008
From: gethemant at gmail.com (hemant)
Date: Sat, 26 Jul 2008 01:58:54 +0530
Subject: [Backgroundrb-devel] Problem with worker servers
In-Reply-To: <6B8CAF0B-727E-4F46-BCFC-4832B132227F@newzwag.com>
References: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>
	<fb283d5e0807251253v1e5abf4egcf6294c73200cc3e@mail.gmail.com>
	<6B8CAF0B-727E-4F46-BCFC-4832B132227F@newzwag.com>
Message-ID: <fb283d5e0807251328v3183a738h15dd03cd9b97387e@mail.gmail.com>

On Sat, Jul 26, 2008 at 1:31 AM, Daniel Lockhart <daniel at newzwag.com> wrote:
> Both servers are RHLE, one Red Hat 5, the other 4.  The kernel on the 4 is
> 2.6.9-55.0.9.ELsmp which is 32 bit.  The kernel on the 5 is 2.6.18-92.el5,
> which is 64 bit.

Okay, I am assuming one of the workers is starting the server and
perhaps other worker uses #connect to connect to the server started on
previous worker, right?

What might be happening is, when next worker attempts to connect, it
may not find server up yet, hence you need to make reconnect attempt.
This is of course, complete guess work, but for example:

class Bar
  def receive_data p_data
    send_data(p_data)
  end

  def connection_completed
    puts "whoa man"
  end

  # will be automatically called when server refuses the connection,
lets try to reconnect after 10 seconds
  # if reconnect fails unbind will be automatically called and next
attempt will be made after 10 seconds.
  def unbind
    add_timer(10) { reconnect("localhost",2981,Bar) }
  end
end

class FooWorker < BackgrounDRb::MetaWorker
  set_worker_name :foo_worker
  def create(args = nil)
    # this method is called, when worker is loaded for the first time
    connect("localhost",2981,Bar)
  end
end

Also, there was a problem with add_timer.. when used within tcp
connections like that, hence I had to release a new version of packet
(0.1.10, its up on rubyforge.org).

Let me know, if this fixes your problem.

From daniel at newzwag.com  Fri Jul 25 16:59:33 2008
From: daniel at newzwag.com (Daniel Lockhart)
Date: Fri, 25 Jul 2008 13:59:33 -0700
Subject: [Backgroundrb-devel] Problem with worker servers
In-Reply-To: <fb283d5e0807251328v3183a738h15dd03cd9b97387e@mail.gmail.com>
References: <C48B1A75-7C1D-45E0-9215-6DC39B6D4BC9@newzwag.com>
	<fb283d5e0807251253v1e5abf4egcf6294c73200cc3e@mail.gmail.com>
	<6B8CAF0B-727E-4F46-BCFC-4832B132227F@newzwag.com>
	<fb283d5e0807251328v3183a738h15dd03cd9b97387e@mail.gmail.com>
Message-ID: <983F2C32-E9EC-45BD-A23E-B4736C913A27@newzwag.com>

Thanks again for your help.  I discovered what the problem was; I had  
receive_data defined in a module that was included in the handler  
class.  Apparently this was not getting called (I am guessing the way  
in which packet determines whether the method exists in the handler  
class changed, or the order in which modules are mixed in changed).   
Either way, moving the method directly into the handler class seems to  
have solved the problem.  Thank you again for your help, I really love  
backgroundrb!

Daniel
On Jul 25, 2008, at 1:28 PM, hemant wrote:

> On Sat, Jul 26, 2008 at 1:31 AM, Daniel Lockhart  
> <daniel at newzwag.com> wrote:
>> Both servers are RHLE, one Red Hat 5, the other 4.  The kernel on  
>> the 4 is
>> 2.6.9-55.0.9.ELsmp which is 32 bit.  The kernel on the 5 is  
>> 2.6.18-92.el5,
>> which is 64 bit.
>
> Okay, I am assuming one of the workers is starting the server and
> perhaps other worker uses #connect to connect to the server started on
> previous worker, right?
>
> What might be happening is, when next worker attempts to connect, it
> may not find server up yet, hence you need to make reconnect attempt.
> This is of course, complete guess work, but for example:
>
> class Bar
>  def receive_data p_data
>    send_data(p_data)
>  end
>
>  def connection_completed
>    puts "whoa man"
>  end
>
>  # will be automatically called when server refuses the connection,
> lets try to reconnect after 10 seconds
>  # if reconnect fails unbind will be automatically called and next
> attempt will be made after 10 seconds.
>  def unbind
>    add_timer(10) { reconnect("localhost",2981,Bar) }
>  end
> end
>
> class FooWorker < BackgrounDRb::MetaWorker
>  set_worker_name :foo_worker
>  def create(args = nil)
>    # this method is called, when worker is loaded for the first time
>    connect("localhost",2981,Bar)
>  end
> end
>
> Also, there was a problem with add_timer.. when used within tcp
> connections like that, hence I had to release a new version of packet
> (0.1.10, its up on rubyforge.org).
>
> Let me know, if this fixes your problem.
>


From caseyforbes at comcast.net  Mon Jul 28 16:04:23 2008
From: caseyforbes at comcast.net (Cassidy Forbes)
Date: Mon, 28 Jul 2008 16:04:23 -0400 (EDT)
Subject: [Backgroundrb-devel] Best way to have several single-threaded
 workers handling job queue?
Message-ID: <alpine.OSX.1.10.0807281557030.10348@rufus.local>

Hi all,

I have jobs that need to be queued up and run in a single thread 
(the labor intensive part of the job uses a library that is not thread 
safe)

Right now, I have a single worker. The thread pool_size = 1 but jobs 
are put in using thread_pool.defer so that backgroundrb takes care of 
the queue for me.  This works fine except that the queue gets larger 
than I would like.  I'd like to have multiple identical (single 
threaded, own Rais environment) workers sharing a queue and processing 
the jobs.

I could start up several workers with different worker_keys and 
distribute jobs on the client side, but this seems a little messy. Are 
there any better options?

I'm using the latest backgroundrb/packet from git and memcached for 
the result cache.

Casey

From lists at wildgooses.com  Tue Jul 29 10:21:03 2008
From: lists at wildgooses.com (Ed W)
Date: Tue, 29 Jul 2008 15:21:03 +0100
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
Message-ID: <488F274F.3080204@wildgooses.com>

Joost Baaij wrote:
> I am building a Rails website which has content that expires at some 
> time in the future.
>
> Since this content is cached, by the time it expires I would like a 
> method to sweep the cache at that time. Since it's based on a date, I 
> cannot use the Rails app itself. I could use polling or sweep the 
> cache every 5 minutes, but maybe there's a better approach?
>
> I would like to know if I can use backgroundrb's at/cron-like features 
> to schedule one-off execution of a piece of code at a date/time in the 
> future. This would be a sweeper that runs at the expires_at time of my 
> content.
>
>
> I realise I can build a worker that goes into a sleep/wait loop until 
> the desired time, but that's not really better than polling.

I think one issue is that the backgroundb will loose it's pending tasks 
if it's rebooted?  Does this cause your app a problem?

Have you considered memcached caching which has time based expiry (could 
integrate better).  Or I think it's not so hard to wrap the builtin 
partial caching to include time based expiration (see some of the 
plugins for inspiration, just need to choose a good place to store your 
metadata)

Anyway, feels like although bdrb can do this it might not be the best 
solution?

Good luck

Ed W

From lists at wildgooses.com  Tue Jul 29 10:23:53 2008
From: lists at wildgooses.com (Ed W)
Date: Tue, 29 Jul 2008 15:23:53 +0100
Subject: [Backgroundrb-devel] Scheduling one-off tasks
In-Reply-To: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
References: <7AB6B12B-652B-4804-90BC-E94B4BC720B4@spacebabies.nl>
Message-ID: <488F27F9.9000609@wildgooses.com>

Disregard some of my last post - opened mouth before reading about 
persistent queues:

http://backgroundrb.rubyforge.org/workers/

From me at retrodict.com  Thu Jul 31 01:23:26 2008
From: me at retrodict.com (P Baker)
Date: Thu, 31 Jul 2008 01:23:26 -0400
Subject: [Backgroundrb-devel] OS dependence
Message-ID: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>

Hey all, fairly new to Rails and definitely new to BackgrounDRB, thanks for
your hard work.

However, I'm curious about incompatibilities between the latest stable
release of bdrb (from the svn mirror) and mac os x. I've got (nearly)
identical setups on a Debian Etch and Mac OS X 10.5.3 system running rails
2.1.0, ruby 1.8.6 (2008-03-03 patchlevel 114), latest stable versions of
packet and chronic. The attached (mirrored at
http://www.sendspace.com/file/ulw9o2) rails project, which has a simple demo
ProgressWorker, demonstrates my concerns.

Running ./script/backgroundrb, and then in the console typing
MiddleMan.worker(:progress_worker).ask_result(:worker) produces different
results. On the debian system it produces the expected result, while mac
reports:

NoMethodError: You have a nil object when you didn't expect it!
You might have expected an instance of ActiveRecord::Base.
The error occurred while evaluating nil.[]

from.../vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:151:in
`ask_result'
    from
.../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
`ask_result'
    from
.../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
`map'
    from
.../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
`ask_result'
    from (irb):1

Has anyone else experienced this? Is this a known bug? Are there other
differences in threading between OSs? I develop on a mac and the production
server is Debian, so I'm curious as to what to expect...Any guidance would
be appreciated.

Thanks -pbaker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080731/bc13d2ad/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bdrb_test.tgz
Type: application/x-gzip
Size: 141564 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080731/bc13d2ad/attachment-0001.tgz>

From me at retrodict.com  Thu Jul 31 07:29:00 2008
From: me at retrodict.com (P Baker)
Date: Thu, 31 Jul 2008 07:29:00 -0400
Subject: [Backgroundrb-devel] OS dependence
In-Reply-To: <fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
References: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>
	<fb283d5e0807310122h21a7ebfcp4122a8c0824af1b8@mail.gmail.com>
	<fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
Message-ID: <526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>

I was hoping to avoid introducing another dependency with memcached, but
I'll consider it.

I guess my question should be more pointed. The test case seems to evaluate
basic functionality, which appears not to work. Have others experienced this
bug (on mac os)? Or is it likely that this is a configuration dependent bug?
Has this been addressed in the git version of bdrb? Is a stable version of
bdrb that works on mac os forthcoming?

-pbaker

On 7/31/08, hemant <gethemant at gmail.com> wrote:
>
> On Thu, Jul 31, 2008 at 1:52 PM, hemant <gethemant at gmail.com> wrote:
> > On Thu, Jul 31, 2008 at 10:53 AM, P Baker <me at retrodict.com> wrote:
> >> Hey all, fairly new to Rails and definitely new to BackgrounDRB, thanks
> for
> >> your hard work.
> >>
> >> However, I'm curious about incompatibilities between the latest stable
> >> release of bdrb (from the svn mirror) and mac os x. I've got (nearly)
> >> identical setups on a Debian Etch and Mac OS X 10.5.3 system running
> rails
> >> 2.1.0, ruby 1.8.6 (2008-03-03 patchlevel 114), latest stable versions of
> >> packet and chronic. The attached (mirrored at
> >> http://www.sendspace.com/file/ulw9o2) rails project, which has a simple
> demo
> >> ProgressWorker, demonstrates my concerns.
> >>
> >> Running ./script/backgroundrb, and then in the console typing
> >> MiddleMan.worker(:progress_worker).ask_result(:worker) produces
> different
> >> results. On the debian system it produces the expected result, while mac
> >> reports:
> >>
> >> NoMethodError: You have a nil object when you didn't expect it!
> >> You might have expected an instance of ActiveRecord::Base.
> >> The error occurred while evaluating nil.[]
> >>
> >>
> from.../vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_connection.rb:151:in
> >> `ask_result'
> >>     from
> >>
> .../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
> >> `ask_result'
> >>     from
> >>
> .../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
> >> `map'
> >>     from
> >>
> .../vendor/plugins/backgroundrb/lib/backgroundrb/rails_worker_proxy.rb:78:in
> >> `ask_result'
> >>     from (irb):1
> >>
> >> Has anyone else experienced this? Is this a known bug? Are there other
> >> differences in threading between OSs? I develop on a mac and the
> production
> >> server is Debian, so I'm curious as to what to expect...Any guidance
> would
> >> be appreciated.
> >>
> >
> > Well, there are some issues if you are dynamically starting workers
> > using MiddleMan.new_worker on OSX, but your code should work without
> > any problems on OSX.
>
>
> Also, use memcache for result storage, configuration is rather simple:
>
> http://backgroundrb.rubyforge.org/workers/
>
> It will give you much better stability and results.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080731/5ef946f9/attachment.html>

From gethemant at gmail.com  Thu Jul 31 08:17:42 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 31 Jul 2008 17:47:42 +0530
Subject: [Backgroundrb-devel] OS dependence
In-Reply-To: <526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
References: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>
	<fb283d5e0807310122h21a7ebfcp4122a8c0824af1b8@mail.gmail.com>
	<fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
	<526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
Message-ID: <fb283d5e0807310517y1b7b9f06i9e887b86b413e416@mail.gmail.com>

On Thu, Jul 31, 2008 at 4:59 PM, P Baker <me at retrodict.com> wrote:
> I was hoping to avoid introducing another dependency with memcached, but
> I'll consider it.
>
> I guess my question should be more pointed. The test case seems to evaluate
> basic functionality, which appears not to work. Have others experienced this
> bug (on mac os)? Or is it likely that this is a configuration dependent bug?
> Has this been addressed in the git version of bdrb? Is a stable version of
> bdrb that works on mac os forthcoming?
>

Oh, please run git version. I am bad at syncing code with svn (And yes
there has been fixes that went into git, but aren't there in svn).

From gethemant at gmail.com  Thu Jul 31 08:20:53 2008
From: gethemant at gmail.com (hemant)
Date: Thu, 31 Jul 2008 17:50:53 +0530
Subject: [Backgroundrb-devel] OS dependence
In-Reply-To: <526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
References: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>
	<fb283d5e0807310122h21a7ebfcp4122a8c0824af1b8@mail.gmail.com>
	<fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
	<526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
Message-ID: <fb283d5e0807310520q4a9fc1do4d61bf8728c8eda1@mail.gmail.com>

On Thu, Jul 31, 2008 at 4:59 PM, P Baker <me at retrodict.com> wrote:
> I was hoping to avoid introducing another dependency with memcached, but
> I'll consider it.
>

Well, think in long run, assuming your workers store hundreds of
thousands of objects in worker cache. How will Ruby take that? I bet,
half the time will be spent in trying to garbage collect the shit. So,
if you are planning to cache results at all, memcache is certainly far
better solution. Of course, you can keep using inbuilt cache, but it
will scale only so such.

From me at retrodict.com  Thu Jul 31 10:03:07 2008
From: me at retrodict.com (P Baker)
Date: Thu, 31 Jul 2008 10:03:07 -0400
Subject: [Backgroundrb-devel] OS dependence
In-Reply-To: <fb283d5e0807310616o1ee3740diddfd241ccd2bed1@mail.gmail.com>
References: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>
	<fb283d5e0807310122h21a7ebfcp4122a8c0824af1b8@mail.gmail.com>
	<fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
	<526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
	<fb283d5e0807310520q4a9fc1do4d61bf8728c8eda1@mail.gmail.com>
	<526944450807310556x30f0579dtdd13d283b72aff3d@mail.gmail.com>
	<fb283d5e0807310616o1ee3740diddfd241ccd2bed1@mail.gmail.com>
Message-ID: <526944450807310703h73d74feeu67856e6b36003360@mail.gmail.com>

Ok. So I upgraded to the git version of backgroundrb and now experience the
same results on mac 10.5.3 and debian. The problem is that ask_result
doesn't seem to work at all. Hemant, is the move towards using memcached for
passing results/status queries a sign that native ruby handling of that will
be (is?) deprecated?

-pbaker

On 7/31/08, hemant <gethemant at gmail.com> wrote:
>
> On Thu, Jul 31, 2008 at 6:26 PM, P Baker <me at retrodict.com> wrote:
> > Granted there is inherent scalability problems in Ruby. I'm not really
> > planning on storing more than status markers in the cache. I'm so
> reluctant
> > to implement memcached on our stack because I don't see the need for it
> at
> > this point. I can see the argument for large scale websites and the need
> to
> > pass such large numbers of objects, but I don't think I fit into either
> of
> > those categories.
> >
> > I've read your hints on debugging backgroundrb, which suggested running
> the
> > git version, although I'm also looking for stability. What is the
> stability
> > like of the master git branch? Since that requires the git version of
> > packet, what's the stability of 'edge' packet?
>
>
> For running git version of backgroundrb you don't need git version of
> packet. Latest gem will do.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080731/b8317e31/attachment.html>

From me at retrodict.com  Thu Jul 31 10:47:44 2008
From: me at retrodict.com (P Baker)
Date: Thu, 31 Jul 2008 10:47:44 -0400
Subject: [Backgroundrb-devel] OS dependence
In-Reply-To: <526944450807310703h73d74feeu67856e6b36003360@mail.gmail.com>
References: <526944450807302223o6dac15a8g2d22eabe291feeab@mail.gmail.com>
	<fb283d5e0807310122h21a7ebfcp4122a8c0824af1b8@mail.gmail.com>
	<fb283d5e0807310123j5c2c54danb3b68ad864b5c2fa@mail.gmail.com>
	<526944450807310429y39c32e3end6c0d6c4b1b1461d@mail.gmail.com>
	<fb283d5e0807310520q4a9fc1do4d61bf8728c8eda1@mail.gmail.com>
	<526944450807310556x30f0579dtdd13d283b72aff3d@mail.gmail.com>
	<fb283d5e0807310616o1ee3740diddfd241ccd2bed1@mail.gmail.com>
	<526944450807310703h73d74feeu67856e6b36003360@mail.gmail.com>
Message-ID: <526944450807310747u44ae9e89lec0140c74bfc65db@mail.gmail.com>

Or...I need to spell things correctly. My mistake. ask_result definitely
works. Thanks for your comments hemant.

-pbaker

On 7/31/08, P Baker <me at retrodict.com> wrote:
>
> Ok. So I upgraded to the git version of backgroundrb and now experience the
> same results on mac 10.5.3 and debian. The problem is that ask_result
> doesn't seem to work at all. Hemant, is the move towards using memcached for
> passing results/status queries a sign that native ruby handling of that will
> be (is?) deprecated?
>
> -pbaker
>
> On 7/31/08, hemant <gethemant at gmail.com> wrote:
>>
>> On Thu, Jul 31, 2008 at 6:26 PM, P Baker <me at retrodict.com> wrote:
>> > Granted there is inherent scalability problems in Ruby. I'm not really
>> > planning on storing more than status markers in the cache. I'm so
>> reluctant
>> > to implement memcached on our stack because I don't see the need for it
>> at
>> > this point. I can see the argument for large scale websites and the need
>> to
>> > pass such large numbers of objects, but I don't think I fit into either
>> of
>> > those categories.
>> >
>> > I've read your hints on debugging backgroundrb, which suggested running
>> the
>> > git version, although I'm also looking for stability. What is the
>> stability
>> > like of the master git branch? Since that requires the git version of
>> > packet, what's the stability of 'edge' packet?
>>
>>
>> For running git version of backgroundrb you don't need git version of
>> packet. Latest gem will do.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080731/e2c6bd9f/attachment.html>