[Backgroundrb-devel] periodic scheduling

hemant gethemant at gmail.com
Thu Jan 17 12:41:37 EST 2008

Hi Jack,

On Jan 17, 2008 10:30 PM, Jack Nutting <jnutting at gmail.com> wrote:
> I've been using backgroundrb since back in March 2007 or so.  It's
> been working mostly OK for me, but since the old version doesn't seem
> to work out of the box with rails 2.0 for me, I decided to test out
> the latest version.
> So, I'm looking at the scheduling, and trying to figure out the best
> option for what I want to do.  What I have is a set of workers that
> each poll the database occasionally.  I'd like to set it up so that
> each of them can poll for work, do the work (however long that may
> take), then pause for a "sleep time" and start over.  With the old
> version of backgroundrb, I was never able to get any of the scheduling
> options to quite do what I wanted to do, so I just ran without
> scheduling. I manually started all my workers by calling
> MiddleMan.new_worker in a script, passing in the "sleep time", and
> then implemented each worker something like this:
>   def do_work(args)
>     @args = args
>     @sleep_time = @args[:sleep_time]
>     logger.info "CampaignStarter #{jobkey} started"
>     while(true)
>       # do the actual work in another method
>       main_work
>       sleep @sleep_time
>     end
>   end
> That seemed to work pretty well for the most part.

Yuck. Never use sleep() in new backgroundrb workers. :)

> So now, looking at the current version.  I'm guessing that it's
> probably not OK for me to use create() the way I was using do_work()
> in the past;  Presumably create() is supposed to return at some point.
>  Besides which, one of my workers needs to have multiple instances, so
> I have to pass in a job_key somewhere.  I'm guessing that I should
> call MiddleMan.new_worker(), telling it to run my main_work() method
> with some scheduling option, but I'm not sure which scheduling option
> to use.  I don't want to schedule it to run every 10 seconds, since
> each call to main_work could take anything up to several minutes, and
> I don't want the invocations piling up. The docs also seem to suggest
> that I could do things my old way by calling MiddleMan.ask_work() and
> letting it run forever, but it's not clear whether I can pass in a
> job_key there.  I'd love any suggestions here...

I am still unclear about whats your purpose of using bdrb? If you just
want to use it to schedule tasks, then you can specify your schedule
in config file, using either Cron on UNIX syntax and it will work. If
you are scheduling at in interval of 10 seconds and your actual tasks
takes longer than that, then also there is nothing to worry and bdrb
will schedule tasks as they are ready to run and worker is free. So
your tasks won't be piling up. create() does return at some point, but
at each iteration of reactor loop, tasks ready to run are executed.

Currently if you need more than one instance of same worker running at
the same time, its not supported via backgroundrb.yml file, but you
can create as many instances of workers as you want using different
job_keys using MiddleMan.new_worker() method from rails. Hell, I think
you can invoke MiddleMan.new_worker() from one of your workers to
start workers on demand and supply job_keys.

ask_work() is for doing one shot task execution. It can be only used
in a already runner worker. If you started a worker with a job_key
using new_worker() method, then you will have to use that job_key
along with worker name in ask_work() or ask_status() invocation.

> Next, is there any way from inside a worker to see what its job_key
> is?  With the old version I've been using the jobkey as a way to split
> up some of the work requests, so each instance of my worker class
> needs to know its own key in order to find what work is assigned to
> it.

Sad truth of life is that, you can't. Not from inside of a worker. But
since inside any worker you have full rails environment and hence
MiddleMan proxy object, you can query job_key using:

MiddleMan.worker_info(:worker => worker_name.to_sym)

Haven't tried, but it should work.

> Finally, it's not at all clear to me how the current version handles
> multiple instances and threads.  For example, if I call
> MiddleMan.new_worker(), does it actually create a new instance of my
> worker class (as the old version did), and if so, is that instance in
> a new thread of its own?  How about MiddleMan.ask_work() and
> MiddleMan.send_request()?

new_worker() creates a full forked process. ask_work(), send_request()
can only operate on workers which are already running.

And lastly, sorry for documentation, if you found it a bit out of
touch. *vows to improve it*

Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.


More information about the Backgroundrb-devel mailing list