[Backgroundrb-devel] How to obtain task progress for long-running methods

hemant gethemant at gmail.com
Fri Nov 7 13:04:10 EST 2008


On Fri, Nov 7, 2008 at 9:34 PM, Jim Nicholson <jnicholson at skarven.net> wrote:
> Hmm... I'm using the git version of backgroundrb, and currently up-to-
> date with the repository; I don't have an "ask_worker" method; I have
> "ask_result." Maybe I should be asking: where can I obtain the version
> of Backgroundrb that's covered in "The Rails Way," because it seems the
> API has changed significantly.
>
> Anyway, I'm creating new workers via something like
>
> MiddleMan.new_worker(:worker => :foobar_worker, :worker_key => 'some key
> ', :data => {:arguments => values})
>
> The worker is coded to such that the process logic is called inside the
> create method, and it posts progress via cache[worker_key]
>
>  def create(args = nil)
>        run_my_process(args)
>  end
>
>  def run_my_process(args)
>    cache[worker_key] = {}
>    ... do stuff
>    cache[worker_key][:progress] = 25
>    ... do more stuff
>    cache[worker_key][:progress] = 50
>    ... etc
>  end
>
> While this runs, I can see the worker chugging away (because it logs as
> it runs,) but if I call
>
> MiddleMan.worker(:foobar_worker).ask_result('some key')
>
> while it is running, I get nil.
>
> Prior to this, I had a prior version of the worker that stayed in
> menory. To use that one, I invoked the process via
>
> MiddleMan.worker(:foobar_worker).async_run_my_process('some key')
>
> The only difference in the coding of the worker for that version was
> that the cache key was job_key rather than worker_key. Calling
> ask_results('some key') produced nil until the task was completed, at
> which point it contained my end-state results.

We sort of grew very conservative about memory usage in 1.1 release
and decided to handover this task to memcache. Earlier, bdrb master
process used to cache all these results and hence they were readily
available. But unless I maintain an LRU based access, I will have hard
time keeping memory usage of master process under check.

Hence this responsibility was shelved off to worker. But hey, while a
worker is processing a task, it can't server cache result, and thus
use Memcache, which is out of process. Now even if worker is
processing task, you still get your cache result requests served
alright.

More details can be found here:

http://backgroundrb.rubyforge.org/workers/#result_caching

Lastly, compared to any book, that page is usually more up to date about stuff.


More information about the Backgroundrb-devel mailing list