[Backgroundrb-devel] best approach to managing workers and getting status

Jack Nutting jnutting at gmail.com
Mon May 5 04:35:51 EDT 2008

The way that I (and perhaps many others here?) handle this is to use
 the database to hold a "work queue", and have a long-running worker
 that polls the database periodically and handles any pending requests.
  In your case for example, you could have an AudioFileWork model,
 containing fields for "release_id" and "pending";  Your app would
 create a new instance of that with pending=true, and your background
 worker would poll for rows where pending=true, then mark them as false
 when they are complete.  For your post-creation interactions (to show
 the status with ajax) you'd use the id of the created AudioFileWork
 instead of a job key.

 There are many advantages to this approach:

 - you always have a known number of workers (the number you configure
 and start), so you won't have uncontrollable memory usage explosions
 if your site gets busy, just slower response times
 - you can check the status of a request by querying the database
 - there is very little in-memory data that is lost in unhappy events
 (a crash or unhandled exception)
 - you have a historical record in the databse of all work that is
 completed by a background worker
 - you get a very loose coupling to backgroundrb.  In my case, I simply
 specify in a model class that it needs processing, and it just
 happens.  My app doesn't "talk" directly with backgroundrb at all,
 except for an admin page where I can make sure it's up and running.

// jack
// http://www.nuthole.com

More information about the Backgroundrb-devel mailing list