[Backgroundrb-devel] Please comment on upcoming changes of backgroundrb

[hemant]

On Fri, Jun 20, 2008 at 8:52 PM, Stevie Clifton <stevie at slowbicycle.com> wrote:
> Hey Hemant,
>
> This is great.  Thanks for putting the effort to getting the git
> version tagged so people feel comfortable using it in production!  I'm
> assuming when you say that you'll tag the git release you'll also push
> out new bdrd and packet gems, correct?
>
> I had a quick question about fetch_parallely/run_concurrently.  Is it
> basically doing the same thing as defer(), but just with a thread-safe
> callback?  If so, it might be more intuitive to make the name of the
> method reflect that, such as "defer_concurrently" so as not to make
> people think that they do completely different things.
>
> Also, if the above is true (run_concurrently is a thread-safe version
> of defer), I think it would be less cumbersome if you just extended
> defer() to allow the same functionality by passing the name of a
> callback method in an options hash instead of creating an entirely new
> method.  This way people could just use the block passed to defer() as
> the request_proc, and another method at the worker level for the
> response_proc instead of creating a bunch of procs to pass to the
> method.  For example, instead of:
>
> run_concurrently(args, request_proc, response_proc)
>
> You could do:
>
> def example_worker(args)
>  defer(args, :callback => :my_callback_method) |args|
>    # same as the body of request_proc
>  end
> end
>
> def my_callback_method(result)
>  # same as body of response_proc (call register_status or whatever)
> end
>
> I looked through the code in git for fetch_parallely, and I'm not sure
> if this would give you the same flexibility, but it would be a more
> intuitive solution to a user IMHO.
>

Well, thanks for looking up. Actually I made quite bit of changes and
pushed an update to git on testcase branch.

http://github.com/gnufied/backgroundrb/commits/testcase

Well, I take your advice for run_concurrent method. But I have made
result storage completely thread safe and hence user can call it from
threads without any worries.

Actually, I made quite a bit of API changes today:

Here is a sample API, if you try new branch:

http://pastie.org/218967

Or

# Run a task like:

# this is our dear remote process
class HelloWorker
  set_worker_name :hello_worker
  def barbar t_user
    # runs method some_task inside thread pool
    thread_pool.defer(t_user,method(:some_task))
  end

  def some_task user
    user_feeds = user.feeds
    loop do
      # user can retrieve/add/edit objects from result/cache without
worrying about thread safety
      # when you call job_key in your threads it automatically
resolves to job_key for the task being executed
      # there may be another task being executed in another thread,
but since job_key is Thread local variable
      # it will always resolve to the correct one.
      old_counter = cache[job_key]
      cache[job_key] += 10
    end
  end
end

# invoke tasks/methods in worker async
MiddleMan.worker(:hello_worker).async_barbar(<some_job_key>, at user)

# invoke method in worker sync
MiddleMan.worker(:hello_worker).barbar(<some_job_key>, at user)

# ask the result object stored with job_key "wow"
MiddleMan.worker(:hello_worker).ask_result(<job_key_or_key_with_which_you_stored_result>)
# If you are doing this in production, i will strongly advise to use
mecache for result storage


# there is an issue if user invokes multiple tasks in thread pool
directly from one of the worker
# under current settings they are going to end up with same job key

# Also, new_worker can't have same method invocation conventions
because it accepts more parameters.


Again, MiddleMan wraps basically a cluster of bdrb servers now and when you say:

MiddleMan.worker(:hello_worker).async_barbar(<some_job_key>, at user)

It will be invoked in a round robin manner in all the bdrb servers
specified in configuration file. Complete configuration file looks
like:

# A Sample YAML configuration file
---
:backgroundrb:
  :ip: 0.0.0.0 #ip on which backgroundrb server is running
  :port: 11006 #port on which backgroundrb server is running
  :environment: production # rails environment loaded, defaults to development
  :debug_log: true # whether to print debug logs to a seperate worker,
defaults to true
  :log: foreground # will print log messages to STDOUT, defaults to
seperate log worker
  :result_storage: memcache # store results in a mecache cluster, you
also need to specify location of your memcache clusters in next
section

:memcache: "10.0.0.1:11211,10.0.0.2:11211" #=> location of mecache
clusters seperated by comma

# following section is totally optional, and only useful if you are
trying to cluster of backgroundrb server
# if you do not specify this section backgroundrb will assume that,
from rails you are connecting to the
# backgroundrb server which has been specified in previous section
:client:
  :drb_servers: "10.0.0.1:11006,10.0.0.2:11007"

# You specify your worker schedules here
:schedules:
  :foo_worker: # worker name
    :barbar: #worker method
      :trigger_args: */5 * * * * * * #worker schedule



Please comment on this.