[Backgroundrb-devel] right usage of bdrb

hemant kumar gethemant at gmail.com
Sun Jan 13 22:56:46 EST 2008


Hi Matthias,

On Mon, 2008-01-14 at 08:52 +0530, hemant kumar wrote:
> Hi Matthias,
> Yup, but let me suggest an alternative architecture and see for yourself
> if this will work better or not. Here is the deal:
> 
> class SyndicationWorker < BackgrounDRb::MetaWorker
>   def create(args = nil)
>     add_periodic_timer(300) { search_for_syndication }
>   end
>   def search_for_syndication
>     # when you are saving syndications which are to be imported 
>     # downloaded and encoded. Rather than calling another worker
>     # from here, you should have a flag for each Feed, and each
>     # item indicating that this feed needs to be imported, 
>     # downloaded and encoded
>     syndications
>   end
> end
> 
> class ImportWorker < BackgrounDRb::MetaWorker
>   def create(args = nil)
>     add_periodic_timer(300) { import_syndication }
>   end
>   def import_syndication
>      new_feeds = Feed.find(:all,:conditions => {:to_be_imported => \ 
>             true })
>      new_feeds.each do |feed|
>        # fetch the feed and have two flags in associated item
>        # indicating item is to be downloaded and encoded
>        fetch_feed(feed)
>        # update the 'to_be_imported' flag for given feed
>        # generally, we should prefer all this logic to go into
>        # model itself ( remember thick model, thin controllers)
>        # but I am writing here for demonstration
>        feed.to_be_imported = false
>        feed.save
>      end
>   end
> end
> 
> class DownloadWorker < BackgrounDRb::MetaWorker
>   # assuming you want 5 downloads to go concurrently
>   # I am setting thread pool size of 5
>   # default pool size is of 20 threads
>   pool_size 5
>   def create(args = nil)
>     add_periodic_timer(300) { download_syndication }
>   end
>   def download_syndication
>     new_items = Item.find(:conditions => {:to_be_downloaded => \
>        true})
>     new_items.each do |item_id|
>       thread_pool.defer(item_id) do |item_id|
>         # using item_id, rather than item itself for
>         # thread local objects, with underlying db 
>         # connection of their own.
>         item = Item.find_by_id(item_id)
>         download_video(item)
>         # update the to_be_downloaded flag for item.
>       end
>     end
>   end
> end
> 
> class EncodeWorker < BackgrounDRb::MetaWorker
>   def create(args = nil)
>     add_periodic_timer(300) { encode_video }
>   end
>   def encode_video
>     new_encodes = Item.find(:conditions => {:to_be_encoded => \
>       true })
>     new_encodes.each do |item|
>       encode_item(item)
>       # update the to_be_encoded flag and save the model
>     end
>   end
> end
> 
> 
> I guess, above architecture will work quite well, and I am avoiding
> inter dependence in workers and rather using table itself as a queue. It
> has multiple advantages that way. Tables are persistent, so you will
> always know, which feeds are to be fetched, downloaded and encoded and
> which are already done. 
> 

Okay, I digress. Above architecture will not work well, when you will
have multiple copies of same workers running parallely.


So, you can use your existing code, with thing I suggested for download
worker. Also, if you want to queue jobs invoked with ask_work, I will
suggest you to use defer. What I am saying is:

class ImportWorker < BackgrounDRb::MetaWorker
  pool_size 1
  def import(feed_id)
    thread_pool.defer { |b_feed_id| start_import(t_feed_it) }
  end
end

And invoke above job as usual:

MiddleMan.ask_work(:worker => :import_worker, :worker_method
=> :import, :data => feed_id)

#defer will make sure that, your task is queued and having thread pool
size of 1, makes sure that only one job is running concurrently.






More information about the Backgroundrb-devel mailing list