[Backgroundrb-devel] questions about persistent jobs

Mitchell Curtis Hatter curtis.hatter at insightbb.com
Tue Oct 7 03:33:52 EDT 2008

I've been having problems with losing async'ed tasks, so I've moved to  
trying to use enqueued tasks.

I'm trying to offload user queries to the background, using a thread  
pool size of 40 and two workers (on different machines). It seems that  
if I exceed the thread pool size those queries get lost, or for some  
other reason I can't determine yet I'm losing my tasks when there are  
lots of them at once.

For that reason I've moved to trying enqued tasks. This method seems  
to be fine for not losing jobs but I can't seem to run more then 2 at  
a time? (one for each worker). Is there a way to up the number of  
enqueued tasks that will be ran at a time? This way I could make use  
of my thread pool, and the enqueueing tasks.

The behavior I expected from this method is that it checks every  
second (like I have configured) for enqueued tasks. Then running as  
many enqueued tasks as the thread pool limit for that worker. Is there  
a way to get this behavior from the system? Right now it seems to pull  
out one at a time and run them till they finish and then pull out the  

If my expected behavior isn't how it's supposed to work, would there  
be a way to do that?

I'm running version 1.0.4 with packet 0.1.14

My drb config:
   :port: 11006
   :persistent_delay: 1
   :debug_log: true
   :environment: development
   :result_storage: memcache

:memcache: "xxx.xxx.xxx.xxx:11211,xxx.xxx.xxx.xxx:11211"
:client: "xxx.xxx.xxx.xxx:11006,xxx.xxx.xxx.xxx:11006"

# :schedules: # optional task scheduling

My worker:

require 'hpricot'
require 'open-uri'

class TexisWorker < BackgrounDRb::MetaWorker
   set_worker_name :texis_worker
   pool_size 40

   def create(args = nil)
     logger.info "Creating Texis Worker Instance"

   def search(arg)
     thread_pool.defer(:perform_search, arg)

   def perform_search(arg)
     # calls out to a server runs a search, gets parsed by hpricot and  
then caches the results


More information about the Backgroundrb-devel mailing list