[Backgroundrb-devel] Multiple worker threads per type per server?

hemant gethemant at gmail.com
Thu Aug 21 22:56:12 EDT 2008


On Fri, Aug 22, 2008 at 5:06 AM, benjamin smith
<smith.benjamin at gmail.com> wrote:
> Hello,
>
> I've recently integrated backgroundrb into my rails application and I
> am very happy with how it is performing. We strictly use the 'enq' job
> queuing functionality and it works well for us. We have 2 clustered
> web/app servers, both of which backgroundrb is running on. When
> starting up backgroundrb, it looks for all the workers classes in the
> default location (lib/workers), and starts up a packet worker for
> each.
>
> However, I am not clear on how to spin up multiple instances of a
> worker on the same machine. One type of worker is very taxed and I'd
> like to create several of him to facilitate processing queue contents
> quicker. The documents seem to suggest using MiddleMan.new_worker, but
> this has not worked well for me. I'm able to get the middleman to
> create a new worker instance, but it doesn't do any work to pull tasks
> out of the queue.
>
> How do you recommend I spin up multiples of one type of worker on one
> machine for parallel processing of a job queue?

Thats correct, since each enqueued task belongs to a worker with a
name and worker_key, it won't be pulled out by workers, other than for
what it was created. Now, since each newly started worker, will have
its own new key, the arrangement won't work.

You can perhaps, patch up bdrb_job_queue.rb and remove the worker_key
match restriction, make it configurable and submit a patch. ;)


More information about the Backgroundrb-devel mailing list