[Backgroundrb-devel] Threadpool and queuing of tasks
gobigdave at gmail.com
Tue Jan 22 16:43:37 EST 2008
I recently switched over to v1.0, and things are rolling along pretty well.
However, one thing that has always been a little confusing to me is knowing
when to use thread_pool. Since most of my bgrb workers are called from my
web app to process rather than being scheduled, I'm using the thread_pool
for every call. Unfortunately, that means that I have to split up workers
by how many threads I can have. It would be great if one worker could
partition a single thread pool among the methods. I want to avoid too many
workers to keep the process count down.
I'm now working on a new scheme that pushes this example. Basically, I have
some long running, saved searches that are triggered by various events
throughout the site. All I want my site to do is update a status that the
job is queued and have it picked up from there. Here is where I run into
trouble, possibly because I've built too many systems like this that use
real queuing packages. Here is what I want:
Dispatch method (usually one thread is necessary):
1. Find the oldest 'queued' record (make sure to find with :lock => true)
1a. If none, goto step 5
2. Update status to 'processing'
3. Send to search method
4. Repeat 1
Search method (many threads):
1. Perform the search
2. Update status to 'complete'
The easy answer is to split these into two workers. Set the pool_size of
Dispatch to 1, and Search to 5 or 10. However, eating two processes (master
and worker) for something so simple as Dispatch seems like serious overkill
to me. Since I currently run on one server, the extra processes cut into
the memory the main site wants.
A related question is how to implement Dispatch without polling. Call me
anal, but I feel dirty whenever I using polling, especially something that I
want to be picked up immediately. Is there a way I can trigger it to run if
it isn't already? The old bgrb had a singleton that let me do something
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Backgroundrb-devel