[Backgroundrb-devel] [ANN] BackgrounDRb 1.0 pre-release available now

Emil Marceta emarceta at gmail.com
Tue Dec 11 08:52:22 EST 2007

On Dec 11, 2007 11:50 AM, hemant kumar <gethemant at gmail.com> wrote:
> Hi
> On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote:
> > Does this means that slave/daemons are not the dependency anymore?
> Yes, its gone. bdrb no longer depends on slave and daemons.
> > By 'not encouraged' do you mean that 1.0 is not supporting multiple
> > threads in the worker or just as a general guidance?
> >
> > Could you please comment, how would you approach the following
> > scenario with 1.0. Currently, we have a worker that creates threads
> > that process financial payment transactions. An http request sends
> > several 10s or 100s payment transaction records. They are handled by
> > the single worker instance. Within the worker there is a pool of
> > threads created that is calculated based on the number of
> > transactions. For example for 200 transactions there will be 20
> > threads where each thread handles 10 requests in a squence. Each
> > transaction takes about 3-5 seconds, so our throughput is
> > significantly improved by internal worker parallelization with a
> > thread pool. The worker periodically updates custom backgroundjob
> > databse record, so that following ajax request from the client can
> > read the status of the worker process. The job is identified with the
> > worker key that is stored in the session.
> Its not encouraged, thats all. You can still have threads in your
> workers.

Thanks for clarifying.
snip ...

> Also ideally, when using EventDriven network programming, you want all
> your sockets within select loop for efficiency. So, you wouldn't need
> any damn threads, if you can use a HTTP handler that works in Evented
> manner. What i mean to say is, you don't do this:
> a = Net::HTTP.get("http://www.google.com")
> but you do,
> Backgroundrb::HTTP.open("http://www.google.com") do |data|
> process_data(data)
> end
> What I am trying to illustrate is, when you ask to open, google.com
> page, evented model allows you to attach callback ( the block in this
> case ), which will be called when data arrives from google.com, rather
> than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns
> immediately. And you are concurrent as hell.
> But this is not possible, because if you are charging cards, then you
> are probably using ActiveMerchant which is using Net::HTTP and which
> blocks when you make request. But trust me, writing a simple http client
> is not that difficult, there is already connect() available in all
> workers.

We are actually on of the ActiveMerchant providers (E-Xact), so
strictly we are talking what is actually behind ActiveMerchant. There
are many protocols involved in financial networks, depending where the
transaction is routed. We are very familiar with Reactor engines and
patterns you are advocating, and they work great, especially in
uniform scenarios without throttling, sequencing etc. In our case, I
don't see a clear gain I'm afraid. While a thread pool was done in
no-time and is dead simple maintain, test etc.

> > How this works with fastcgi or multiple mongrel based engines where it
> > is not guaranteed to hit the same process with the next request? We
> > are using custom database tables and code for sharing the status
> > information now but I was wandering whether the plumbing includes
> > something to address this.
> Thats no problem at all, BackgrounDRb is a TCP server, so if you have
> followed the README file, no matter from which machine, you are making
> the request if you are specifying worker X, then its guaranteed to hit
> the same worker(with optional job_key if you are starting your worker
> dynamically)

Our Rails cluster runs bdrb on each Rails server and uses domain
sockets. This to avoid a single point of failure and have uniform
architecture. Would that work too? That is, does bdrb now works sort
of like memcache where each server knows of every other instance? But
even with that in place, in fastcgi for example, fastcgi processor may
recycle the Rails process where callback has been registered.


More information about the Backgroundrb-devel mailing list