[Backgroundrb-devel] Scheduling async jobs to workers, and checking if workers are currently running a job

Meyers, Dan d.meyers at lancaster.ac.uk
Thu May 14 11:20:08 EDT 2009

I'm trying to schedule jobs from our own database scheduling system that
we hook BackgrounDRb into. I assign a job to a worker asynchronously

MiddleMan.worker(worker).async_wrapper(:job_key => jobkey, :arg =>

This gets called whenever I have a job scheduled to start according to
my db. The behaviour in an older version of BackgrounDRb was that if
that worker was already running a job then the new job didn't get
started or queued, it just disappeared. 

This was exactly what I wanted, as I would periodically check my db and
attempt to start all workers scheduled to do so using this method. Any
that were still processing the previous run of the job just didn't run.
For example I have a job scheduled to run every 5 seconds on a specific
worker written just for it, for an indefinite period of time. There
should only ever be one copy of it running. Normally it takes 3 or 4
seconds to run, so this is fine. Sometimes it gets a sudden backlog of
data to deal with, and takes 30 seconds to process it all. Using the old
version of BackgrounDRb after the 30 second job was up it would start a
new job on the next 5 second 'tick' of our scheduler, but all the jobs
it had tried to start while the worker was already handling a job would
have disappeared. The behaviour using the new version seems to be to
queue up the next run of the job on a call to async_*. The result now
becomes that any time I have a backlog I get the one 30 second job run
that actually processes the data, then immediately it finishes I get 6
or so other runs of the job which have no data to process, so execute in
almost no time but fill my logging table with 'started...stopped'
messages. These jobs were queued up to run, 1 per 5 second tick, while
the 30 second job was still executing.

I hoped I could use


To check whether the job was running or not. This *seemed* to work on
the jobs that I started manually (rather than starting on BackgrounDRb
load) by putting 'set_no_auto_load true' at the top of the worker file
and then calling

MiddleMan.new_worker(:worker => worker, :worker_key => key)

to create the worker before giving it work to do. However this seems to
sometimes get out of sync. It will always return status :stopped if a
worker or job doesn't exist, but I had a worker running and outputting
data, and worker_info was still claiming it was stopped. I have also had
worker_info continue to claim that a worker is running after it has
finished processing data and disappeared from the process list. Is there
any way to be sure the data returned from worker_info is current and

Secondly, worker_info seems to *always* return :running if the worker is
started on BackgrounDRb load, instead of being spawned using new_worker
by my own code. This is one of my problems in the example of the
indefinite task above. I cannot use worker_info to see whether the
worker is currently running a job (even if worker_info reliably returned
correct information), and not call async_wrapper if it is, because
worker_info *always* says the task is running. Is there any way that I
can find out if a worker started on BackgrounDRb load is currently
executing a job or not?

Dan Meyers
Network Specialist, Lancaster University
E-Mail: d.meyers at lancaster.ac.uk

More information about the Backgroundrb-devel mailing list