[Backgroundrb-devel] Greetings

Fri Apr 24 12:06:58 EDT 2009

My name's Nick, and I work for a company called Bytemark - we use backgroundrb 
in a range of internal projects (all our internal apps are Ruby on Rails, so 
it makes sense). 

Basically, and as I mentioned on IRC, I've been tasked with making 
backgroundrb "better" over the next month or so, and I'd like to push as much 
upstream as I can while I'm at it (although some stuff I can guarantee you 
won't want - I'll my making my own local branch use a login plugin with a 
different syntax, for instance).

First on the agenda is this backtrace: 
 /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:147:in 
`log': Mysql::Error: MySQL server has gone away: SELECT * FROM 
`bdrb_job_queues`     WHERE ( worker_name = 'dhshell_export_worker' AND taken 
= 0 AND scheduled_at <= '2009-04-24 10:41:44' )  LIMIT 1 FOR UPDATE 
(ActiveRecord::StatementInvalid)
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:302:in 
`execute'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:537:in 
`select'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in 
`select_all_without_query_cache'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in 
`select_all'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:586:in 
`find_by_sql'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1345:in 
`find_every'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1307:in 
`find_initial'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:538:in 
`find'
from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:11:in 
`find_next'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in 
`transaction'
from /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/transactions.rb:79:in 
`transaction'
from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:8:in 
`find_next'
from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:271:in 
`check_for_enqueued_tasks'
from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:133:in 
`worker_init'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in 
`call'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in 
`run'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in 
`check_for_timer_events'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in 
`each'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in 
`check_for_timer_events'
from /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:296:in 
`check_for_timer_events'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:140:in 
`start_reactor'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in 
`loop'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in 
`start_reactor'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_worker.rb:20:in 
`start_worker'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:33:in 
`load_worker'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:26:in 
`initialize'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47:in 
`new'
from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47
from /usr/bin/packet_worker_runner:19:in `load'
from /usr/bin/packet_worker_runner:19

(not latest git, but definitely git - we're just updating to packet 0.1.15, 
but I doubt that'll affect this particular error) - this kills the server 
from time to time for us (we have our MySQL server set to go away after half 
a day) - it's not the full story, since for that to happen, worker requests 
must have stopped too (or something else), but fixing this is number 1 on my 
list. I've not delved much into the source yet (5pm on a Friday is *not* the 
time to start with that!); I was wondering if you guys have a preferred / 
global strategy for dealing with errors. My approach to the above would be to 
catch the error, and respond by attempting to re-establish the connection. If 
that succeeds, I'd re-do the scheduled item (so the database going away would 
be transparent to the client); if not... hmm. Do we have a method of 
reporting a failed task back to the worker? 

Another problem I was seeing that led me to stop using the backgroundrb cache 
object was that requests into a worker via the MiddleMan API would just 
freeze up from time to time, leaving the Rails controller method hanging on 
for it. That's another one for me to investigate (we weren't using memcached, 
mind, but it annoys me when the Hash - a far simpler system - is less 
reliable).

I'm also tasked with redeploying the current backgroundrb setup - right now, 
we have one server per application using it (which is about 4,5 backgroundrb 
servers in total), with the backgroundrb table stored in the application's 
database. I'm kind of moving towards a scheme where we have a pair of 
backgroundrb servers (transparently load-balanced) used by all applications. 
Each backgroundrb server would have its own separate *SQLite* database. 
Requests would go to one or another of the backgroundrb servers; if one of 
the servers died, all the requests would go to the other server, and 
vice-versa. If possible (I know this is a -devel list), I'd like to get 
comments on whether that's a good setup or not, and suggestions for 
improvement. If not, well, just consider it to be a bit of context ;)

Regards,
-- 
Nick Thomas
Bytemark Hosting Limited