[Mongrel] Offloading Background Tasks

[Steve Midgley]

At 06:17 PM 3/1/2008, mongrel-users-request at rubyforge.org wrote:
>Date: Sun, 2 Mar 2008 07:47:43 +0530
>From: hemant <gethemant at gmail.com>
>Subject: Re: [Mongrel] Offloading Background Tasks
>To: mongrel-users at rubyforge.org
>Message-ID: 
><fb283d5e0803011817t764d1b12w7fdafda64ea7ebc5 at mail.gmail.com>
>Content-Type: text/plain; charset=ISO-8859-1
>
>On Sat, Mar 1, 2008 at 7:10 AM, Nathan Esquenazi 
><lists at ruby-forum.com> wrote:
> > Hello everyone,
> >
> >  I am very interested in off-loading tasks from the 
> request/response
> >  cycle in order to speed up my application. My application is 
> heavily
> >  connected to using web services both external and from other 
> servers we
> >  host.
>[snip]
> >
> >  I feel like I am missing something because I don't see how the 
> worker
> >  can "tell" the main rails application that it now has the data and 
> then
> >  somehow show it when its "ready". Do I need to constantly poll the
> >  server or how is this usually done.
> >
> >  Also, I need to allow the user to make certain changes to data 
> fields
> >  and then I need to push those changes to twitter and facebook via 
> a
> >  background process.
> >
> >  What solution is best suited based on what I described and could 
> someone
> >  please help me understand the pattern for how to do this. I will 
> be
> >  eternally grateful.
> >  --
>
>For what you are describing, I think BackgrounDRb will be best fit.
>Since, its the only solution ( that I know of ) provides interfacing
>with rails so as you can actually fetch results retrieved by a worker
>in rails.
>
>Bj is a worker queue to offload tasks. AFAIK, you can't really
>interact seamlessly from rails. I don't know much about startling. But
>this is coming from guy who maintains bdrb to take it with a grain of
>salt.

Hi,

Firstly - apologies to Hemant for misinformation from me a while ago 
about the current status of BackgrounDrb. It's clear now of course to 
me that this an active and healthy project.

I have been messing around with Ara Howards BackgroundJob (bj) and it's 
got a way to spin out a bunch of jobs and then wait until they all 
return, which is similar to the way threads work:

>       jobs = Bj.submit list_of_jobs, :tag => 'important'
>       ...
>
>       jobs.each do |job|
>         if job.finished?
>           p job.exit_status
>           p job.stdout
>           p job.stderr
>         end
>       end

You can see that if by the time you get down to the "if jobs.finished?" 
section, you could choose to ignore whatever jobs haven't returned yet 
(or possibly even TERM them if that's appropriate).

I wonder also if you've considered adding some architecture to your 
system so that you can collect whatever you can from your external 
services from within Rails but then whatever is left over you could 
pull down via Ajax after the page had mostly loaded (so some parts of 
the page might be left with placeholders and then get filled in with 
the correct info later as the Ajax calls retrieve it from Rails..

That'd be my thought on this: build something (with BackgrounDRb, Bj or 
whatever) that lets you spawn a bunch of a async queries. Give those 
queries some kind of unique handle. When you're done building the whole 
page, have your view fill in data from whichever queries have returned. 
Then send the incomplete page to the browser and have ajax queries pull 
down the remaining query results (using a webservice interface passing 
in those unique handles you set up above to ID which job you want).

Hope this helps!

Steve