[Backgroundrb-devel] Behaviour of pool_size setting

Mason Hale masonhale at gmail.com
Wed May 16 09:27:43 EDT 2007

I resolved a similar situation by pre-allocating a fixed-size pool of
These workers are long-running and they individually poll the database
for new work.

My backgroundrb_schedules.yml includes this:

<% pub_worker_pool_size = 5 %>
<% pub_worker_pool_size.times do |i| %>
pubworker<%= i %>:
    :class: :pub_worker
    :job_key: :pubworker<%= i %>
    :worker_method: :do_work
        :ignored: 1
        :start: <%= Time.now + (10 + (10 * i)).seconds %>
<% end %>

My PubWorker#do_work method is:

  def do_work(args)
    logger.debug("#{self.jobkey}: do_work called")
    loop do # loop forever
      unless do_publish
        sleep 60 # pause 1 minute, if no publications were found on last try
        sleep 1 # pause 1 second otherwise

My do_publish method checks the queue for work, and if there is a
publication due, it publishes it and returns true, if there was no work it
returns false. In the do_work you'll see that if the queue is empty it gets
polled once per minute per worker. Otherwise workers try to empty the queue
as quickly as possible. I stagger the start up of the workers at 10 second
intervals so as to distribute the polls to the database. I avoid
multi-threading issues by making each worker responsible for its own loop.
There is no external scheduler that calls the do_work method periodically.

My previous implementation had a single long-running worker that would spawn
additional workers as needed to handle the work in the queue. But my
experience was that creating a worker from a worker leads to socket not
found and connection reset by peer errors, and poor reliability in general.
So I scrapped that in favor of the fixed-size pool above and it has been
much more reliable.

I do think there is potential to create a fixed-size pool of workers, and
another long-running 'queue' manager worker, and implement it such that the
workers request work from the queue manager worker rather than the database.
This way the queue manager could get  a block of work and parcel it out with
fewer queries to the database. In addition the workers could check in with
the manager when they are done, making it possible for the manager to
compile statistics on the jobs. For me, using the database to manage the
queue -- and the inherent multi-threading issues -- was the more expedient


On 5/16/07, skaar <skaar at waste.org> wrote:
> * Christian Schlaefcke (cschlaefcke at wms-network.de) [070515 23:49]:
> > Hi,
> >
> > I have backgroundrb running to decouple the execution of massive
> > business logic from an ActionWebservice request. The service is designed
> > to take some configuration parameters and fire a lot of background
> > workers to do the requested work. Due to performance reasons I want to
> > limit the number of workers to a maximum number of 30. But when I start
> > a configuration that requires for example 300 worker executions I can
> > see that the limit of 30 workers is not kept and a number of about 180
> > worker processes are filling up my process list.
> ok, I might look at adding a 'max_worker' configuration parameter as
> something separate from the thread pool (which really just say something
> about the MiddleMan, not the exisiting workers) - changing the pool_size
> semantics could impact current installations.
> /skaar
> >
> > I start my workers like this:
> >
> > key = MiddleMan.new_worker(:class => :execution_worker, :args =>
> > {...some_args...})
> >
> >
> > Why do I see so much more than my declared number of 30 workers? Am I
> > wrong somehow? How do I have to understand the behaviour of pool_size?
> > What happens when I have 30 workers working and the 31st, 32nd, ...,
> > 300th request to start a worker comes in?
> >
> > Thanks & Regards!
> >
> > Christian
> >
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> --
> ----------------------------------------------------------------------
> |\|\             where in the       |          s_u_b_s_t_r_u_c_t_i_o_n
> | | >===========  W.A.S.T.E.        |                  genarratologies
> |/|/    (_)     is the wisdom       |                  skaar at waste.org
> ----------------------------------------------------------------------
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070516/f73bbf90/attachment-0001.html 

More information about the Backgroundrb-devel mailing list