[Backgroundrb-devel] Polling is REALLY slow

Dave Dupre gobigdave at gmail.com
Tue Mar 18 00:08:13 EDT 2008


I'm having some trouble with using brb for polling for work. Basically, what
I want to do is update the state of a record and have brb notice and start a
process. I do not want to call ask_work for each process because there may
be several places where the object's state is changed, and I only want to
process once.  Plus, I can schedule things better this way.  My first
thought was to create a worker that polled for records with the correct
state, but it doesn't work as expected.  Here is what I have with error
checking, etc. removed:

class RequestQueuePollerWorker < BackgrounDRb::MetaWorker
  set_worker_name :request_queue_poller_worker

  QUEUE_SLEEP_TIME = 30   # seconds

  def create(args = nil)
    @running = true
    self.poll_queue
  end

  def build_all_matches(args = nil)
    thread_pool.defer(args) do |args|
       requests = Request.find_active(:all)
       requests.each { |request| request.queue! }  # using
acts_as_state_machine
    end
  end

  protected

  # Was hoping to get multiple threads processing
  def build_matches(args = nil)
    thread_pool.defer(args) do |args|
       request = Request.find(args.to_i)
       request.build_matches
    end
  end

  def poll_queue
    Thread.new do
      while @running do
        sleep_time = QUEUE_SLEEP_TIME
        request = nil
        Request.transaction do
          request = Request.find_in_state(:first, :queued, :order => '
requests.updated_at ASC', :lock => true)
          request.search! unless request.nil?  # make sure I don't hit again
        end
        if request
          self.build_matches(request.id)
          sleep_time = 0
          request = nil
        end
        sleep(sleep_time)
      end
    end
  end

  def stop_polling
    @running = false
  end

end

In my test case, there are 20 Request items. Normally, it would take about 5
seconds to call Request.build_matches on 20 Request items.  However, asking
for work like so:

MiddleMan.ask_work(:worker => :request_queue_poller_worker, :worker_method
=> :build_all_matches)

It takes 20 minutes(!) to process the same 20 Request items.  Processing
basically c r a w l s along.  I expected to immediately see all the states
change to :queued, and then very quickly changed to :searching. However, it
doesn't happen that way. Things go to queued pretty quickly, by searching
takes FOREVER. It's like poll_queue and build_matches don't get any CPU
cycles.

Is there a recommended way to do something like this?

Please note, I do not want to use a periodic timer because I don't know how
long handling a message will take. Plus, if there are more items in the
queue, I don't want to wait around to get them. My thought is to generalize
this as a db queue eventually.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080318/67d1a32f/attachment.html 


More information about the Backgroundrb-devel mailing list