[Backgroundrb-devel] Implementation details for worker Queue

hemant gethemant at gmail.com
Sat Dec 22 08:30:41 EST 2007

Hi All,

One can easily code a backgroundrb worker so as it can act as a worker
queue, having reached a stable status, perhaps its time we should add
this feature.

Because if you are already running your application using
BackgrounDRb, perhaps you don't want to manage other solutions when
capability to do that is right there with you.

This where i need feedback. You can already dynamically start a worker
and submit a job to it ( using MiddleMan.new_worker) and it would be
done in a new process.
But its not a queue, it starts executing your task immediately.
Sometimes I doubt if its seriously required, because you can easily
queue your tasks within a worker
using inbuilt thread pool ( if you don't want to spawn new worker ).

What then?
Perhaps what we CAN do is, have a specialized worker that can act on a
priority queue. This priority queue is nothing but a database table
and depending upon the number
of tasks in the queue, master process automatically starts new workers
or deletes them as need be.Our specialized worker, when picks a task
from db, it marks that task as taken
and starts processing them. The control to start new workers depending
upon size of queue can be left either to master or the "specialized
worker" itself.

I was thinking on lines of having a ratio as in lets say per 10 queued
tasks you want one worker, then you can specify that in configuration
file. If the number of tasks goes higher either "specialized worker"
or master starts a new instance of "specialized worker".

There could be a upper limit on number of parallel workers running and
if on one machine this limit is reached backgroundrb can request
master to start workers on another machine. How does this sound?

Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.


More information about the Backgroundrb-devel mailing list