[Backgroundrb-devel] right usage of bdrb

[Matthias Heigl]

Hi,

i'm going to implement a syndication-service, which will get lists in 
xml with some meta-data an enclosed video files, which will get encoded 
   at the end. The syndication run will be startet every five minutes of 
a full hour.

So i thought to build 4 Worker. One for checking which feeds to 
syndicate (syndication_worker) at a specific time, one for processing 
the list (import_worker), one for fetching the clips (download_worker) 
and one for encoding (encoding) worker.

In my tests all went fine, all my jobs are invoked properly.

Q1)
Is this procedure right in the intention of backgroundrb?
I meen, there will be at worst up to about hundret of download_worker 
started in one syndication run. Is bdrb able to handle such a queue, 
without to lose one job?

Q2)
Some downloads are really slow, so i would rather start about 5 
downloads in parallel and queue the encoding in one queue again.

                    syndication
                         |
                       import
        /        /       |        \        \
    download download download download download
        \        \       |        /        /
                      encoder

Is this possible?

I know i can give another job_key, maybe the name of the content-source, 
but this can go up to 20 or more download-queues in one syndication run 
and this is too much.

Thanks in advance!

Cheers,

Matze


------------------------------------------------

backgroundrb.yml
- - - - - - -
:backgroundrb:
   :ip: 0.0.0.0
   :port: 11006
   :environment: production
:schedules:
   :syndication_worker:
     :checksyndicate:
       :trigger_args: 0 5 * * * * *
- - - - - - -

syndication_worker.rb
- - - - - - -
class SyndicationWorker < BackgrounDRb::MetaWorker
   set_worker_name :syndication_worker
   def create(args = nil)
     # this method is called, when worker is loaded for the first time
   end
   def checksyndicate
     syndications = # all Syndication in this hour
     syndications.each do |syndication|
       MiddleMan.ask_work(:worker => import_worker, :worker_method => 
import, :data => syndication.id)
     end
   end
- - - - - - -

import_worker.rb
- - - - - - -
class ImportWorker < BackgrounDRb::MetaWorker
   set_worker_name :import_worker
   def create(args = nil)
     # this method is called, when worker is loaded for the first time
   end
   def import(feedid)
     feed = Feed.find_by_id(feedid)
     # fetch feed
     feed.items.each do |item|
       # create item
       MiddleMan.ask_work(:worker => download_worker, :worker_method => 
download, :data => item.id)
     end
   end
- - - - - - -

download_worker.rb
- - - - - - -
class DownloadWorker < BackgrounDRb::MetaWorker
   set_worker_name :download_worker
   def create(args = nil)
     # this method is called, when worker is loaded for the first time
   end
   def download(itemid)
     item = Item.find_by_id(:itemid)
     # fetch enclosure
     MiddleMan.ask_work(:worker => encoder_worker, :worker_method => 
encode, :data => item.id)
   end
- - - - - - -

encoder_worker.rb
- - - - - - -
class EncoderWorker < BackgrounDRb::MetaWorker
   set_worker_name :encoder_worker
   def create(args = nil)
     # this method is called, when worker is loaded for the first time
   end
   def download(itemid)
     item = Item.find_by_id(:itemid)
     # encode item
   end
   - - - - - -