[Backgroundrb-devel] Job Queue

Ryan Leavengood leavengood at gmail.com
Fri Feb 1 14:19:23 EST 2008


On Feb 1, 2008 1:58 PM, Norman Elton <normelton at gmail.com> wrote:
>
> I can see how to build an array containing jobs. And a mutex could be
> used to ensure thread-safe access to the array. But how can the
> workers "go to sleep" until a job arrives?
>
> Any thoughts?

I had a similar requirement and ended up just using a single worker
and thread_pool.defer (the documentation talks about how to use this.)
I've looked at the BackgrounDRb code and thread_pool.defer makes use
of a queue which a configurable number of threads read from. You can
set the size of the pool by calling pool_size in your worker. The
default pool size is 20 threads.

If you need to know the status of each thread you need to use a mutex
to synchronize access to a hashtable member variable which can contain
the status for each thread (you would need to decide what the key
would be for each thread, maybe the database ID of the job the thread
is processing.) You would then pass this hashtable to the
register_status method. There is a post from hemant on this mailing
list from December which shows how to do this pretty well:

http://rubyforge.org/pipermail/backgroundrb-devel/2007-December/001170.html

The only caveat here is the green Ruby threads used in the thread_pool
may not play well with the job processing you are doing. But from the
testing I've done it seems pretty good, especially for something
involving the network which probably has a lot of latency.

The general architecture I would suggest would be to have a jobs table
in the database, and when jobs are added the Rails model
(after_create) can call MiddleMan.ask_work and pass the ID of the job
just created. The worker will pass that job_id into thread_pool.defer
which will then process the job. For my own work I tend to put all the
heavy processing into separate classes which I simply call from the
worker. So for you maybe something like JobProcessor.run(job_id) or
whatever.

Regards,
Ryan


More information about the Backgroundrb-devel mailing list