[Backgroundrb-devel] Thread Pool Size?

Ezra Zygmuntowicz ezmobius at gmail.com
Thu Dec 7 23:35:05 EST 2006

Hey Brian-

On Dec 7, 2006, at 6:59 PM, Brian Mulloy wrote:

> Hi All,
> It might be lack of sleep, but I am struggling to accurately limit  
> our pool size.   It seems like I can specify it on the server with  
> the -s command line option and also on the client via the YAML  
> pool_size.  Is that right?  Which one wins?
> Our problem is that we are getting about 40 threads on each  
> backgroundrb box and it's flooring our db and each bgrb box.
> We want around 8.
> Is there a way to put a hard ceiling on the server side thread pool?
> Here is our setup:
> 5 app boxes, app01 - app05, with 8 mongrel instances each (this is  
> where the 40 comes from, I think)
> Each app box points to a load balancer in front of two backgroundrb  
> boxes, crawler01 and crawler02
> this is the backgroundrb.yml on each app box
> :host: backgroundrb
> :port: 2000
> :rails_env: production
> :pool_size: 1

	You are using the latest release of backgroundrb? 0.2.1? The pool  
size works like this.. It is supposed to only allow the number you  
specify of workers to be running at one time. Any more requests for  
new workers when the pool is full get queued up and then they are run  
when another worker leaves the pool and dies. What exactly is  
happening to you? Are you sending a bunch of requests and they all  
run at once even though you have set the limit?

	It may be that we need to add some tighter control to how many  
active jobs there are. Maybe a way for new_worker to return an error  
that you can rescue and retry on a different drb server? The current  
thread pool should limit the actively running workers but writing  
properly threaded code is hard so I wouldn't be too surprised if I  
made a mistake.

  	I can imagine a way to be more strict about the pool limit but it  
will take some doing I imagine. Can you explain how your bdrb stuff  
works? Are you starting a new worker for each time you do something?  
Or do you have a certain number of workers that always run and keep  
running in a loop accepting jobs? The reason I ask is that the way  
that the new backgroundrb works its better to have a number of named  
workers that always run in a loop accepting jobs from a queue. So  
instead of calling new_worker all the time you set your workers to  
autostart at server start time and then just call methods on the  
daemon style workers. This way you can start exactly how many workers  
you want and give each one a named job key. Then you can either have  
the workers running in a loop where they wake up every second or  
whatever interval you set and do whatever jobs they need to do,

	Since you say that you are already keeping the state of the workers  
in the database then maybe an approach like this would be better  
suited. The basic idea is something like this:

An example worker that pulls events off the db and executes them,  
then saves them out and loops again

class RSSWorker < BackgrounDRb::Worker::RailsBase
   require 'rss'
   require 'net/http'

   def do_work(args)
     @args = args

   def mainloop
     loop {
       sleep @args[:sleep]
       RssUrls.find_all_by_pending(true).each do |rss|
         Net::HTTP.start(rss.host) do |http|
           response = http.get(rss.path)
           raise response.code unless response.code == "200"
           rss_parser = RSS::Parser.new(response.body)
           rss.output = rss_parser.parse


	Something like that may work better although I am not certain  
exactly what your workers need to do. You coudl also do the same  
thing with long lived workers but create a method other then do_work  
which you can call from rails via the work_thread method.

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Backgroundrb-devel mailing list