[Backgroundrb-devel] Upper limit to number of jobs queued by BDRb?

Raghu Srinivasan raghu.srinivasan at gmail.com
Fri May 30 00:05:10 EDT 2008


I use BDRb to process RSS feeds for users on my site (http://feedflix.com).

I have a batch job that queries the DB for records that haven't been updated
in the last so many hours and kicks off a background job for each of them.
If N records are returned by the DB, N background jobs gets queued and get
done serially. As long as N is 255 or under, everything works like a charm.
I've noticed that whenever N is >= 256 (2 power 8), then at the 257th job
BDRb stops processing any more users. I can get around it by limiting the DB
query to return no more than 255 records and then all is fine. No problems
at all. But over that, I see this issue. Repeatedly.

Here's what I found in my log/backgroundrb_11007_debug.log file.
=========
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
./script/backgroundrb:42:in `new'
./script/backgroundrb:42
{:type=>:do_work, :worker=>:batch_worker, :data=>{:body=>"All Done!!!",
:subject=>"Completed processing for : 275 users"},
:worker_method=>:email_admin}
You have a nil object when you didn't expect it!
The error occurred while evaluating nil.instance
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in
`schedule_write'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in
`write_and_schedule'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in
`each_with_index'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`each_with_index'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in
`write_and_schedule'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in
`dump_object'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_pimp.rb:23:in
`send_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_meta_pimp.rb:76:in
`send_request'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:44:in
`ask_worker'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:104:in`process_work'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:35:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`call'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:29:in
`extract'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_parser.rb:42:in
`extract'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:31:in`receive_data'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:212:in
`read_external_socket'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:204:in
`handle_external_messages'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:178:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`each'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:174:in
`handle_read_event'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:130:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`loop'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in
`start_reactor'
/usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:21:in
`run'
/home/raghus/public_html/
feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize'
./script/backgroundrb:42:in `new'
./script/backgroundrb:42
=========
My log/backgroundrb_server_11007.log is clean - there are no error messages
at all there.
Is anyone else seeing this? Am I missing something obvious? Any solutions?
Thanks,
Raghu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080529/dced6a98/attachment.html>


More information about the Backgroundrb-devel mailing list