[Backgroundrb-devel] best approach to managing workers and getting status

Frank Schwach f.schwach at uea.ac.uk
Wed Jul 16 14:10:51 EDT 2008

Hi Hemant and others,

Once again thank you for your reply to my last post and the excellent
work you are doing! Having seen your announcement of all those new
features really convinced me to go ahead and use backgroundrb for my
app. However, I must admit being quite confused now with all the latest
changes to the API and would appreciate some help getting my head around
this. So, basically, I have an app with a database table of long running
tasks. There are several different types of tasks, so in my table I have
a column for the type of job to be run and another for serialized data
for the job (basically some IDs and args). I want to run the actual jobs
on a remote machine, which the new version of backgroundrb seems to have
covered nicely now. 

So this is what I thought I could do:

Have a pool of x RemoteJobWorkers on the remote machine waiting to
accept jobs:

class RemoteJobWorker
  def enque_job (job_type, job_args)
    thread_pool.defer(:job_type, job_args)

  def job_type1 (args)
    # update job table with status 
    Jobs.update(job_key, :status => running)
    # perform the long running task
    large_dataset.each do
      # update job table with progress
      Jobs.update(job_key, :progress => x)

    # job done: update status again
    Jobs.update(job_key, :status => complete)

  def jobtype2 (args) ... end
  def jobtype3 (args) ... end
  .. and so on ..


So when a job is submitted by a user, the controller sends it to the
queue on the remote machine like this(?):

MiddleMan.worker(:remote_job_worker).async_enque_job(:arg =>
data,:job_key => new_job.id,:host => "a.b.c.d:XXX")

where new_job is the newly created instance of my Jobs model.

Does that make sense or is there a better approach that I should be
using? I also like the idea of just having a scheduled worker on the
remote machine polling the database for pending jobs and then sending
them to the queue just like I am trying to do with the above. I would
really appreciate opinions from some experienced users about this to
help me decide. 
In the above I am assuming that the jobs running in thread pool have
access to the job key and can therefore identify the row in the Jobs
table that they need to update? I hope I understood the announcement of
the latest version correctly that that is the case? 

One of my tasks would itself fork to run a third-party program during
its run time - would that be possible inside a job running in a thread
already or would the space-time continuum collapse if I tried that?

I noticed in the announcement that there is support for persistent job
queues now but I am not sure how to tie this in with my existing table.
Hemant, do you have a more detailed example of how to use this and what
the model would look like? 

Apologies for all the questions - your help is more than welcome!!
Thank you all in advance


On Wed, 2008-06-25 at 16:50 +0530, hemant wrote:
> On Wed, Jun 25, 2008 at 4:30 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> > Hemant,
> > These latest additions to backgroundrb look pretty cool. Unfortunately,
> > I don't think I will be able to use it this way because in my setup I
> > can't run anything on the cluster nodes directly. I have to submit jobs
> > to a queuing system on the cluster's master node, which is why I think a
> > simple daemon running on the master node that polls the (remote) db for
> > pending jobs and then submits these to the queue would probably be
> > better for my case - but I'm far from being an expert on distributed
> > systems so any suggestions are very welcome!
> Hmm, so use db queuing mechanism inbuilt in bdrb. bdrb still stays
> lightweight because most of these changes aren't affecting core and
> have really went into client side of code.
> MiddleMan.worker(:foo_worker).enq_some_task(:job_key,args)
> Above snippet does exactly that. But anyways, i think you feel its too
> complicated for your setup? I can't help that feeling. Its
> complicated, if its complicated to setup and use, but its not. Those
> features totally stay out of your way, if you don't need them.
Dr Frank Schwach
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ
Tel: 0044/(0)1603 - 592 405

More information about the Backgroundrb-devel mailing list