[Backgroundrb-devel] best approach to managing workers and getting status
gethemant at gmail.com
Wed Jul 16 14:19:44 EDT 2008
On Wed, Jul 16, 2008 at 11:40 PM, Frank Schwach <f.schwach at uea.ac.uk> wrote:
> Hi Hemant and others,
> Once again thank you for your reply to my last post and the excellent
> work you are doing! Having seen your announcement of all those new
> features really convinced me to go ahead and use backgroundrb for my
> app. However, I must admit being quite confused now with all the latest
> changes to the API and would appreciate some help getting my head around
> this. So, basically, I have an app with a database table of long running
> tasks. There are several different types of tasks, so in my table I have
> a column for the type of job to be run and another for serialized data
> for the job (basically some IDs and args). I want to run the actual jobs
> on a remote machine, which the new version of backgroundrb seems to have
> covered nicely now.
> So this is what I thought I could do:
> Have a pool of x RemoteJobWorkers on the remote machine waiting to
> accept jobs:
> class RemoteJobWorker
> def enque_job (job_type, job_args)
> thread_pool.defer(:job_type, job_args)
> def job_type1 (args)
> # update job table with status
> Jobs.update(job_key, :status => running)
> # perform the long running task
> large_dataset.each do
> # update job table with progress
> Jobs.update(job_key, :progress => x)
> # job done: update status again
> Jobs.update(job_key, :status => complete)
> def jobtype2 (args) ... end
> def jobtype3 (args) ... end
> .. and so on ..
> So when a job is submitted by a user, the controller sends it to the
> queue on the remote machine like this(?):
> MiddleMan.worker(:remote_job_worker).async_enque_job(:arg =>
> data,:job_key => new_job.id,:host => "a.b.c.d:XXX")
> where new_job is the newly created instance of my Jobs model.
I see you are rolling out your own table based queue here, why not
just use inbuilt persistent database queue?
For the worker, read this page:
Persistent job queue is explained at the bottom.
For adding tasks to the queue you need to do:
MiddleMan.worker(:remote_worker).enq_some_job(:arg => some_arg)
Task will be added to the db queue and automatically picked up.
Handles hairy race conditions for you and stuff like that.
More information about the Backgroundrb-devel