[Backgroundrb-devel] Implementation details for worker Queue

Jim Scott jim.scott at yahoo.com
Sat Dec 22 11:10:44 EST 2007

Hemant -

Strangely enough, you have exactly described how we use backgroundrb.  We
queue jobs to a table, and BackgroundRb processes them.  We have specialized
workers for each row type.  When we have higher volumes of work, we use Xen
virtualization to stand up new utility servers that process queued jobs.

We can scale this solution horizontally, much in the same we that we scale
application servers.

I for one vote YES.

Jim Scott
Plan Source

-----Original Message-----
From: backgroundrb-devel-bounces at rubyforge.org
[mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of hemant
Sent: Saturday, December 22, 2007 6:31 AM
To: backgroundrb-devel
Subject: [Backgroundrb-devel] Implementation details for worker Queue

Hi All,

One can easily code a backgroundrb worker so as it can act as a worker
queue, having reached a stable status, perhaps its time we should add
this feature.

Because if you are already running your application using
BackgrounDRb, perhaps you don't want to manage other solutions when
capability to do that is right there with you.

This where i need feedback. You can already dynamically start a worker
and submit a job to it ( using MiddleMan.new_worker) and it would be
done in a new process.
But its not a queue, it starts executing your task immediately.
Sometimes I doubt if its seriously required, because you can easily
queue your tasks within a worker
using inbuilt thread pool ( if you don't want to spawn new worker ).

What then?
Perhaps what we CAN do is, have a specialized worker that can act on a
priority queue. This priority queue is nothing but a database table
and depending upon the number
of tasks in the queue, master process automatically starts new workers
or deletes them as need be.Our specialized worker, when picks a task
from db, it marks that task as taken
and starts processing them. The control to start new workers depending
upon size of queue can be left either to master or the "specialized
worker" itself.

I was thinking on lines of having a ratio as in lets say per 10 queued
tasks you want one worker, then you can specify that in configuration
file. If the number of tasks goes higher either "specialized worker"
or master starts a new instance of "specialized worker".

There could be a upper limit on number of parallel workers running and
if on one machine this limit is reached backgroundrb can request
master to start workers on another machine. How does this sound?

Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

Backgroundrb-devel mailing list
Backgroundrb-devel at rubyforge.org

No virus found in this incoming message.
Checked by AVG Free Edition. 
Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007
1:17 PM

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.516 / Virus Database: 269.17.6/1192 - Release Date: 12/21/2007
1:17 PM

More information about the Backgroundrb-devel mailing list