[Backgroundrb-devel] Threadpool and queuing of tasks

Dave Dupre gobigdave at gmail.com
Tue Jan 22 16:43:37 EST 2008

I recently switched over to v1.0, and things are rolling along pretty well.
However, one thing that has always been a little confusing to me is knowing
when to use thread_pool.  Since most of my bgrb workers are called from my
web app to process rather than being scheduled, I'm using the thread_pool
for every call.  Unfortunately, that means that I have to split up workers
by how many threads I can have.  It would be great if one worker could
partition a single thread pool among the methods.  I want to avoid too many
workers to keep the process count down.

I'm now working on a new scheme that pushes this example.  Basically, I have
some long running, saved searches that are triggered by various events
throughout the site.  All I want my site to do is update a status that the
job is queued and have it picked up from there.  Here is where I run into
trouble, possibly because I've built too many systems like this that use
real queuing packages.  Here is what I want:

Dispatch method (usually one thread is necessary):
1. Find the oldest 'queued' record (make sure to find with :lock => true)
1a. If none, goto step 5
2. Update status to 'processing'
3. Send to search method
4. Repeat 1
5. Done

Search method (many threads):
1. Perform the search
2. Update status to 'complete'
3. Done

The easy answer is to split these into two workers.  Set the pool_size of
Dispatch to 1, and Search to 5 or 10.  However, eating two processes (master
and worker) for something so simple as Dispatch seems like serious overkill
to me.  Since I currently run on one server, the extra processes cut into
the memory the main site wants.

A related question is how to implement Dispatch without polling.  Call me
anal, but I feel dirty whenever I using polling, especially something that I
want to be picked up immediately.  Is there a way I can trigger it to run if
it isn't already?  The old bgrb had a singleton that let me do something
like that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080122/6f2e1e03/attachment.html 

More information about the Backgroundrb-devel mailing list