[Backgroundrb-devel] Backgroundrb fixes for transfering large amounts of data

Mike Evans mike at metaswitch.com
Tue Jun 10 10:42:36 EDT 2008


Hemant

We've continued testing our application with backgroundrb and found a
couple of other problems when transfering large amounts of data.  Both
of these problems are still present in the github version of code.

The first problem is an exception in the Marshal.load call in the
receive_data method in the Packet::MetaPimp class.  The root cause is in
the BinParser module, in the arm of code handling the parser state 1
(reading in the data).  The issue is that, at the marked line of code
the pack_data array will be at most @numeric_length entries because of
the format string passed to the unpack call.  This results in the code
dropping a chunk of data and then hitting the exception in a subsequent
Marshal.load call.

      elsif @parser_state == 1
        pack_data,remaining = new_data.unpack("a#{@numeric_length}a*")
        if pack_data.length < @numeric_length
          @data << pack_data
          @numeric_length = @numeric_length - pack_data.length
        elsif pack_data.length == @numeric_length     <======== this
should be "elsif remaining.length == 0"
          @data << pack_data
          extracter_block.call(@data.join)
          @data = []
          @parser_state = 0
          @length_string = ""
          @numeric_length = 0
        else
          @data << pack_data
          extracter_block.call(@data.join)
          @data = []
          @parser_state = 0
          @length_string = ""
          @numeric_length = 0
          extract(remaining,&extracter_block)
        end
      end

The second problem we hit was ask_status repeatedly returning nil.  The
root cause of this problem is in the read_object method of the
BackgrounDRb::WorkerProxy class when a data record is large enough to
cause connection.read_nonblock to throw the Errno::EAGAIN exception
multiple times.  We changed the code to make sure read_nonblock is
called repeatedly until the tokenizer finds a complete record, and this
fixed the problem.

  def read_object
    begin
      while (true)
        sock_data = ""
        begin
          while(sock_data << @connection.read_nonblock(1023)); end
        rescue Errno::EAGAIN
          @tokenizer.extract(sock_data) { |b_data| return b_data }
        end
      end
    rescue
      raise BackgrounDRb::BdrbConnError.new("Not able to connect")
    end
  end

Regards, Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080610/d02a94bb/attachment.html>


More information about the Backgroundrb-devel mailing list