[Backgroundrb-devel] Recommendations for eternally-running backgroundrb workers?
jnutting at gmail.com
Wed May 23 03:37:45 EDT 2007
Hmmm, I meant to reply all on this...
---------- Forwarded message ----------
From: Jack Nutting <jnutting at gmail.com>
Date: May 22, 2007 1:39 PM
Subject: Re: [Backgroundrb-devel] Recommendations for
eternally-running backgroundrb workers?
To: Eden Li <eden at mojiti.com>
On 5/22/07, Eden Li <eden at mojiti.com> wrote:
> Can't you remove the periodic checker code that wraps the actual
> worker and let backgroundrb scheduler do the periodic thing for you?
> In other words, why do you have do_work run "in an infinite loop, ...
> periodically to look for work" when essentially that's what
> backgroundrb does for you?
Well, that's the first thing I tried. What I found is that if one
worker was tied up doing something that took a long time, the other
workers wouldn't be triggered to run at all. So, I've got 3 workers
(let's call them A, B, and C), each set to pause about a minute
between looking for work. At some point, worker A finds it has some
tasks waiting for it, and starts doing them. If this ends up taking
15 minutes, workers B and C are completely idle for 15 minutes. I've
never figured out if this is happening on accident or by design, but
it seems that the scheduler runs in a single thread and waits for each
worker call to finish before proceeding.
I suppose I could get around that by breaking up the work, so that
each call to A.real_work only handles a chunk of its work before
returning and allowing the others to run. But even then, the effect
is that I'd still be stuck serializing (though in smaller chunks)
something that I want to run in parallel. In particular, both A and B
spend a large amount of their working time waiting for results from
http requests (which are independent of each other, coming from
different hosts), so these are good candidates for running in parallel
instead of saying, "OK, now A can make some requests.... OK, now it's
B's turn...." etc.
More information about the Backgroundrb-devel