[Backgroundrb-devel] Load balancing

Robert Bjarnason robert.bjarnason at gmail.com
Tue Jan 16 12:59:19 EST 2007


FYI: Here is how I ended up implemented the load-balancing for our sites:
1. Create an array of available backgroundrb servers that includes how 
many workers can be active simultaneously:
 BACKGROUNDRB_SERVERS = [ ["druby://video1:2223", 3], 
["druby://video2:2223", 1] , ... ]
2. When a middleman is needed I iterate over the array, connect to each 
server until I find a backgroundrb server that has a free slot.
3. Use BackgrounDRb::MiddleManDRbObject.init(:uri => "#{server[0]}") to 
open the connection to the selected server
4. If no server is free then the client request is placed in a queue and 
the user is informed.
5. Use self.delete in workers running on the servers, taking that 
responsibility away from client initiated actions.
6. Use the database to store the progress to get around that results bug.

Note: I tried the setup with ruby-1.8.5-p12 and it has a silent crash 
that was not detected by begin/rescue when running more than one 
process, everything still works fine with ruby-1.8.4.

The only thing I haven't figured out yet is how I close the connection 
from the client, self.delete doesn't close the middleman connection from 
the server side.  The symptom is an ever growing list of established 
connections between clients and servers.  I couldn't find an API in the 
middleman to close the connection explicitly - I think it must be the 
opposite to "DRb.start_service" but can't find anything in the 
backgroundrb code that would do this.  Any ideas?

Robert Bjarnason

skaar wrote:
> * Robert Bjarnason (robert.bjarnason at gmail.com) [070111 16:56]:
>> Hi Erik,
>> Thanks for the pointer, I have used JavaSpaces in the past and really 
>> like this pattern.
>> However, my workers are user initiated and trigger x264 and mp4box video 
>> encoding/editing processes and I'm planning to have a limit of 1-2 
>> simulations backgroundrb processes running on each physical server.
>> I have an entry in the database for each user initiated job, maybe I can 
>> add the IP of each backgroundrb server to that entry and do the 
>> load-balancing in Rails using the database?  The question then is how do 
>> I dynamically change the backgroundrb server IP for each instantiated 
>> MiddleMan object?
> you can use BackgrounDRb::MiddleManDRbObject and instead of calling the
> MiddleMan constant, use a regular object - actually it's a class method
> which takes a DRb uri as it's argument - behaves just like the
> MiddleMan.
> /skaar
>> Warm regards,
>> Robert Bjarnason
>> ps. here are links to the sites that are currently using 
>> backgroundrb-2.1 more or less successfully in production mode:
>> http://store.longwayround.com/
>> http://store.missingface.com/
>> http://store.racetodakar.com/
>> http://store.murderormutiny.com/
>> Erik Morton wrote:
>>> We chose to bypass MiddleMan and place our requests for workers into a 
>>> queue. This way we can create an arbitrary number of workers that loop 
>>> and ask the queue for work. The Mongrels have no need to know the 
>>> location or number of workers. You will of course need another 
>>> strategy to get progress/results from the workers.
>>> On Jan 11, 2007, at 4:41 PM, Robert Bjarnason wrote:
>>>> Hi,
>>>> Can somebody point me in the right direction about what approach would
>>>> be best to load-balance between two or more physical servers running
>>>> backgroundrb while using  BackgrounDRb::Worker::RailsBase with mongrel?
>>>> Thanks,
>>>> Robert Bjarnason
>>>> _______________________________________________
>>>> Backgroundrb-devel mailing list
>>>> Backgroundrb-devel at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>> _______________________________________________
>> Backgroundrb-devel mailing list
>> Backgroundrb-devel at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

More information about the Backgroundrb-devel mailing list