[Backgroundrb-devel] how many processes?

Ezra Zygmuntowicz ezmobius at gmail.com
Sat Nov 11 16:15:32 EST 2006

On Nov 11, 2006, at 12:48 PM, David Lee wrote:

> Hello,
> I'm currently using backgroundrb to fetch feeds for an RSS  
> aggregator. So as you can imagine, backgroundrb is a very critical  
> piece of the app. I was wondering what a recommended pool size  
> would be for this kind of application. Ideally, I'd like to have as  
> many processes in the pool as possible, but what would be too many?
> Thank you,
> David

Hey David-

	That really depends on the hardware and memory you have to use as  
well as the kind of traffic and throughput you need to handle. Are  
you using ActiveRecord models in the workers? If so each worker  
requires quite a bit of memory. Without using AR its only 1 or 2 MB  
per process. If you drag rails into the mix each rails base workers  
is 20Mb or so.

	The new version also has a slower startup time for spawning workers.  
So the best way to approach development with the new code is to run a  
number of named immortal workers that boot when you start  
backgroundrb. Then these workers can always be working or can be  
repeatedly called to do the same job over and over. So if you can do  
without ActiveRecord then you can run many processes. If each worker  
has to laod rails then it adds up quick and you should use fewer  

	I know AR is great convenience but loading all of rails to accces  
the database is a lot of overhead. It may be best to use the mysql  
ruby bindings directly to query or insert to the db. Then you still  
get db access but you can run a lot more workers since you don't have  
to load rails.

	But all in all I think somewhere around 10-20 workers that don't use  
AR can do a *ton* of work.


-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Backgroundrb-devel mailing list