[Backgroundrb-devel] best approach to managing workers and getting status

Frank Schwach f.schwach at uea.ac.uk
Tue Jun 24 13:26:55 EDT 2008


Jack,
I just found your interesting post in the archive and I would like to
come back to this. I need to implement something like this:

I have some very long running tasks (several hours) that should run on a
remote machine and talk to the database on the Rails server. I need to
keep track of jobs including those that have been run in the past, so a
table for background jobs with their status as you describe would be the
best solution for me. 

I am just wondering whether backgroundrb wouldn't be a bit of an
overkill in the scenario you describe? In the new "Advanced Rails
Recipes" from the Pragmatic Programmers Bookshelf there is a recipe
using a simple daemonized ruby process that polls the database for
pending jobs and uses acts_as_state_machine to set the state of the jobs
(there is also a nice BackgrounDRb recipe in the book by the way). 
I am just wondering if the daemonized process isn't easier to handle in
this case since you don't integrate your app with backgroundrb very
tightly anyway?

I would be grateful for any suggestions because there seem to be lots of
possible solutions for this problem and some more or less well
documented plugins and I haven't used any of them before. I need a
simple and robust method that doesn't have too many dependencies and
doesn't require too much maintenance because I want to make the finished
app available for others to install on their local systems. 

Thanks in advance,

Frank

On Mon, 2008-05-05 at 10:35 +0200, Jack Nutting wrote:
> The way that I (and perhaps many others here?) handle this is to use
>  the database to hold a "work queue", and have a long-running worker
>  that polls the database periodically and handles any pending requests.
>   In your case for example, you could have an AudioFileWork model,
>  containing fields for "release_id" and "pending";  Your app would
>  create a new instance of that with pending=true, and your background
>  worker would poll for rows where pending=true, then mark them as false
>  when they are complete.  For your post-creation interactions (to show
>  the status with ajax) you'd use the id of the created AudioFileWork
>  instead of a job key.
> 
>  There are many advantages to this approach:
> 
>  - you always have a known number of workers (the number you configure
>  and start), so you won't have uncontrollable memory usage explosions
>  if your site gets busy, just slower response times
>  - you can check the status of a request by querying the database
>  - there is very little in-memory data that is lost in unhappy events
>  (a crash or unhandled exception)
>  - you have a historical record in the databse of all work that is
>  completed by a background worker
>  - you get a very loose coupling to backgroundrb.  In my case, I simply
>  specify in a model class that it needs processing, and it just
>  happens.  My app doesn't "talk" directly with backgroundrb at all,
>  except for an admin page where I can make sure it's up and running.
> 
> 
> 



More information about the Backgroundrb-devel mailing list