[Backgroundrb-devel] how many processes?
ezmobius at gmail.com
Sat Nov 11 16:15:32 EST 2006
On Nov 11, 2006, at 12:48 PM, David Lee wrote:
> I'm currently using backgroundrb to fetch feeds for an RSS
> aggregator. So as you can imagine, backgroundrb is a very critical
> piece of the app. I was wondering what a recommended pool size
> would be for this kind of application. Ideally, I'd like to have as
> many processes in the pool as possible, but what would be too many?
> Thank you,
That really depends on the hardware and memory you have to use as
well as the kind of traffic and throughput you need to handle. Are
you using ActiveRecord models in the workers? If so each worker
requires quite a bit of memory. Without using AR its only 1 or 2 MB
per process. If you drag rails into the mix each rails base workers
is 20Mb or so.
The new version also has a slower startup time for spawning workers.
So the best way to approach development with the new code is to run a
number of named immortal workers that boot when you start
backgroundrb. Then these workers can always be working or can be
repeatedly called to do the same job over and over. So if you can do
without ActiveRecord then you can run many processes. If each worker
has to laod rails then it adds up quick and you should use fewer
I know AR is great convenience but loading all of rails to accces
the database is a lot of overhead. It may be best to use the mysql
ruby bindings directly to query or insert to the db. Then you still
get db access but you can run a lot more workers since you don't have
to load rails.
But all in all I think somewhere around 10-20 workers that don't use
AR can do a *ton* of work.
-- Ezra Zygmuntowicz
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)
More information about the Backgroundrb-devel