I wasn't specific. I am using the trunk version.<br><br>Yes, I want to
limit methods to the number of threads they run. For example, I have a
worker with many methods that I use for several longer running tasks.
All methods are called from the web application so I use the
thread_pool to queue up requests. However, during some processes, I
may queue up a few hundred calls to a couple of methods. These methods
use a 3rd party web service to gather information, and at a pool_size of
20 I overload the 3rd party server. Through trial and error, 2-3 is
about as many as I can run. The problem now is that I have to reduce
the threads for every method in the worker or create another process.<br><br>Regarding the queue, are you saying just do something like:<br><br>def create(args = nil)<br> # Fire everyone back up if the process is restarted
<br> jobs = Job.find(:all, :conditions => "state = 'queued' or state = 'processing'", :order => 'created_at ')<br> jobs.each { |j| my_worker(<a href="http://j.id" target="_blank">
j.id</a>)<br>end
<br><br>def my_worker(args = nil)<br> thread_pool.defer(args) do |job_id|<br> job = Job.find(job_id, :lock => true) # Lock the row till we update the state<br> job.process! # assuming acts_as_state_machine (this will change the state and process the job)
<br> end<br>end<br><br>This I get, and in fact I started with an implementation just like this. The reason I added the dispatcher is because of the way my app works. My app makes use of saved searches to connect users based on several properties. The searches are very complex and are automatically kicked off based on several triggers. The problem with the above is that multiple quick triggers will result in multiple calls to my_worker, and an expensive operation runs multiple times. However, if all the triggers do is update the databased to change the state to 'queued' and wake up the dispatcher, I will only have one call to my_worker (acts_as_state_machine will only change the state to 'queued' if the current state is 'ready'). Obviously, I can add a lot of logic to handle this, but using the persistent, database queue seems so much simpler and safer.
<br><br>Also, another reason for the dispatcher to allow for future scale. I got in the habit of adding a dispatcher in cases like this to make it very simple to have the dispatcher send tasks to multiple processing queues, often on separate servers. One dispatcher process can easily handle dozens of processing servers (I've had as many as 75 processing servers without a problem).
<br><br>Dave<br><br><div class="gmail_quote">On Jan 23, 2008 12:30 AM, hemant <<a href="mailto:gethemant@gmail.com" target="_blank">gethemant@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Dave,<br><div><br>On Jan 23, 2008 3:13 AM, Dave Dupre <<a href="mailto:gobigdave@gmail.com" target="_blank">gobigdave@gmail.com</a>> wrote:<br>> I recently switched over to v1.0, and things are rolling along pretty well.
<br><br></div>Switch to trunk or 1.0.1. Preferably trunk, we fixed lots of issues in past.<br><div><br>> However, one thing that has always been a little confusing to me is knowing<br>> when to use thread_pool. Since most of my bgrb workers are called from my
<br>> web app to process rather than being scheduled, I'm using the thread_pool<br>> for every call. Unfortunately, that means that I have to split up workers<br>> by how many threads I can have. It would be great if one worker could
<br>> partition a single thread pool among the methods. I want to avoid too many<br>> workers to keep the process count down.<br><br></div>I don't think I follow you here. Since a worker comes with thread pool
<br>
of size 20, you should be good to go. But obviously Ruby green threads<br>don't offer you any parallel execution.<br>Thread pools has been designed to run concurrent tasks (not parallel),<br>it wouldn't be useful to have the ability to partition thread pool
<br>among methods.<br>In fact, I don't follow that notion at all, you mean you want to<br>assign number of threads for each method that you are invoking from<br>rails?<br><br>Can you just clarify things a bit?<br><div>
<br><br>><br>> I'm now working on a new scheme that pushes this example. Basically, I have<br>> some long running, saved searches that are triggered by various events<br>> throughout the site. All I want my site to do is update a status that the
<br>> job is queued and have it picked up from there. Here is where I run into<br>> trouble, possibly because I've built too many systems like this that use<br>> real queuing packages. Here is what I want:
<br>
><br>> Dispatch method (usually one thread is necessary):<br>> 1. Find the oldest 'queued' record (make sure to find with :lock => true)<br>> 1a. If none, goto step 5<br>> 2. Update status to 'processing'
<br>> 3. Send to search method<br>> 4. Repeat 1<br>> 5. Done<br>><br>> Search method (many threads):<br>> 1. Perform the search<br>> 2. Update status to 'complete'<br>> 3. Done<br>><br>
> The easy answer is to split these into two workers. Set the pool_size of<br>> Dispatch to 1, and Search to 5 or 10. However, eating two processes (master<br>> and worker) for something so simple as Dispatch seems like serious overkill
<br>> to me. Since I currently run on one server, the extra processes cut into<br>> the memory the main site wants.<br><br></div>Again, I didn't quite follow you there, so let me just rephrase the<br>thing that you want and if I understood it correctly.
<br>Basically you want to implement a queue, so when a task is submitted<br>from rails the task gets queued.<br>Now dispatch worker, finds the latest(or oldest?) task, and updates<br>the status of the task as taken and hands over the tasks to search
<br>worker.<br>Is that right?<br><br>You can use Queue class for that purpose. And you don't need to poll<br>on a queue, because if Queue is empty all the threads that reading<br>from the queue are automatically blocked.
<br>bdrb thread pool implementation makes use of that. So, whats wrong<br>with a thread pool of size 10 or 20 and add each task to that queue,<br>when a task is removed from the queue, mark the status as processing<br>and just go on with processing of the task. You don't need two
<br>processes for this.<br><div><div></div><div><br>><br>> A related question is how to implement Dispatch without polling. Call me<br>> anal, but I feel dirty whenever I using polling, especially something that I
<br>> want to be picked up immediately. Is there a way I can trigger it to run if<br>> it isn't already? The old bgrb had a singleton that let me do something<br>> like that.<br>><br><br><br><br></div></div>
<font color="#888888">--<br>Let them talk of their oriental summer climes of everlasting<br>conservatories; give me the privilege of making my own summer with my<br>own coals.<br><br><a href="http://gnufied.org" target="_blank">
http://gnufied.org</a><br></font></blockquote></div><br>