[Backgroundrb-devel] Behaviour of pool_size setting
jonathan.wallace at gmail.com
Thu Jul 26 13:21:31 EDT 2007
On 5/16/07, Mason Hale <masonhale at gmail.com> wrote:
> I resolved a similar situation by pre-allocating a fixed-size pool of
> These workers are long-running and they individually poll the database
> for new work.
I'd prefer to not have the workers polling the database, especially if the
amount of workers is large. Since responsiveness is one of the goals of my
app, I'd prefer to let users queries slow down the db. :)
My previous implementation had a single long-running worker that would spawn
> additional workers as needed to handle the work in the queue. But my
> experience was that creating a worker from a worker leads to socket not
> found and connection reset by peer errors, and poor reliability in general.
> So I scrapped that in favor of the fixed-size pool above and it has been
> much more reliable.
Has anyone successfully implemented this technique in a high reliability
I do think there is potential to create a fixed-size pool of workers, and
> another long-running 'queue' manager worker, and implement it such that the
> workers request work from the queue manager worker rather than the database.
> This way the queue manager could get a block of work and parcel it out with
> fewer queries to the database. In addition the workers could check in with
> the manager when they are done, making it possible for the manager to
> compile statistics on the jobs. For me, using the database to manage the
> queue -- and the inherent multi-threading issues -- was the more expedient
How about using the result hash as way to push information from the "queue
managing" worker to other workers? Has anyone tried something like this?
Is it reliable? Or are there race conditions in with the queue-managing
worker writing to the hash and the other workers reading from their key?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Backgroundrb-devel