I recently switched over to v1.0, and things are rolling along pretty well. However, one thing that has always been a little confusing to me is knowing when to use thread_pool. Since most of my bgrb workers are called from my web app to process rather than being scheduled, I'm using the thread_pool for every call. Unfortunately, that means that I have to split up workers by how many threads I can have. It would be great if one worker could partition a single thread pool among the methods. I want to avoid too many workers to keep the process count down.
<br><br>I'm now working on a new scheme that pushes this example. Basically, I have some long running, saved searches that are triggered by various events throughout the site. All I want my site to do is update a status that the job is queued and have it picked up from there. Here is where I run into trouble, possibly because I've built too many systems like this that use real queuing packages. Here is what I want:
<br><br>Dispatch method (usually one thread is necessary):<br>1. Find the oldest 'queued' record (make sure to find with :lock => true)<br>1a. If none, goto step 5<br>2. Update status to 'processing'<br>
3. Send to search method<br>4. Repeat 1<br>5. Done<br><br>Search method (many threads):<br>1. Perform the search<br>2. Update status to 'complete'<br>3. Done<br><br>The easy answer is to split these into two workers. Set the pool_size of Dispatch to 1, and Search to 5 or 10. However, eating two processes (master and worker) for something so simple as Dispatch seems like serious overkill to me. Since I currently run on one server, the extra processes cut into the memory the main site wants.
<br><br>A related question is how to implement Dispatch without polling. Call me anal, but I feel dirty whenever I using polling, especially something that I want to be picked up immediately. Is there a way I can trigger it to run if it isn't already? The old bgrb had a singleton that let me do something like that.
<br>