[Backgroundrb-devel] mysterious crash of a particular worker

Meng Kuan mengkuan at gmail.com
Wed Jan 16 03:27:55 EST 2008 

Hi,

I am using the latest checkout from backgroundrb release 1.0.1. I  
have a worker called status_checker that periodically (every minute)  
checks the status of certain hosts over the network. It works fine at  
the beginning but after a while the worker will mysteriously  
disappear and stop working. I have other workers running but they do  
not disappear like this worker does.

These are the logs generated from logger.info calls for this  
particular worker.

2008-01-16 15:55 StatusChecker: check method started
2008-01-16 15:55 StatusChecker: processing host head0
2008-01-16 15:55 StatusChecker: processing domain webconsole
2008-01-16 15:55 StatusChecker: processing host host1
2008-01-16 15:55 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 15:55 StatusChecker: check method completed
2008-01-16 15:56 StatusChecker: check method started
2008-01-16 15:56 StatusChecker: processing host head0
2008-01-16 15:56 StatusChecker: processing domain webconsole
2008-01-16 15:56 StatusChecker: processing host host1
2008-01-16 15:56 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 15:56 StatusChecker: check method completed
2008-01-16 15:57 StatusChecker: check method started
2008-01-16 15:57 StatusChecker: processing host head0
2008-01-16 15:57 StatusChecker: processing domain webconsole
2008-01-16 15:57 StatusChecker: processing host host1
2008-01-16 15:57 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 15:57 StatusChecker: check method completed
2008-01-16 15:58 StatusChecker: check method started
2008-01-16 15:58 StatusChecker: processing host head0
2008-01-16 15:58 StatusChecker: processing domain webconsole
2008-01-16 15:58 StatusChecker: processing host host1
2008-01-16 15:58 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 15:58 StatusChecker: check method completed
2008-01-16 15:59 StatusChecker: check method started
2008-01-16 15:59 StatusChecker: processing host head0
2008-01-16 15:59 StatusChecker: processing domain webconsole
2008-01-16 15:59 StatusChecker: processing host host1
2008-01-16 15:59 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 15:59 StatusChecker: check method completed
2008-01-16 16:00 StatusChecker: check method started
2008-01-16 16:00 StatusChecker: processing host head0
2008-01-16 16:00 StatusChecker: processing domain webconsole
2008-01-16 16:00 StatusChecker: processing host host1
2008-01-16 16:00 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:00 StatusChecker: check method completed
2008-01-16 16:01 StatusChecker: check method started
2008-01-16 16:01 StatusChecker: processing host head0
2008-01-16 16:01 StatusChecker: processing domain webconsole
2008-01-16 16:01 StatusChecker: processing host host1
2008-01-16 16:01 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:01 StatusChecker: check method completed
2008-01-16 16:02 StatusChecker: check method started
2008-01-16 16:02 StatusChecker: processing host head0
2008-01-16 16:02 StatusChecker: processing domain webconsole
2008-01-16 16:02 StatusChecker: processing host host1
2008-01-16 16:02 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:02 StatusChecker: check method completed
2008-01-16 16:03 StatusChecker: check method started
2008-01-16 16:03 StatusChecker: processing host head0
2008-01-16 16:03 StatusChecker: processing domain webconsole
2008-01-16 16:03 StatusChecker: processing host host1
2008-01-16 16:03 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:03 StatusChecker: check method completed
2008-01-16 16:04 StatusChecker: check method started
2008-01-16 16:04 StatusChecker: processing host head0
2008-01-16 16:04 StatusChecker: processing domain webconsole
2008-01-16 16:04 StatusChecker: processing host host1
2008-01-16 16:04 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:04 StatusChecker: check method completed
2008-01-16 16:05 StatusChecker: check method started
2008-01-16 16:05 StatusChecker: processing host head0
2008-01-16 16:05 StatusChecker: processing domain webconsole
2008-01-16 16:05 StatusChecker: processing host host1
2008-01-16 16:05 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:05 StatusChecker: check method completed
2008-01-16 16:06 StatusChecker: check method started
2008-01-16 16:06 StatusChecker: processing host head0
2008-01-16 16:06 StatusChecker: processing domain webconsole
2008-01-16 16:06 StatusChecker: processing host host1
2008-01-16 16:06 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:06 StatusChecker: check method completed
2008-01-16 16:07 StatusChecker: check method started
2008-01-16 16:07 StatusChecker: processing host head0
2008-01-16 16:07 StatusChecker: processing domain webconsole
2008-01-16 16:07 StatusChecker: processing host host1
2008-01-16 16:07 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:07 StatusChecker: check method completed
2008-01-16 16:08 StatusChecker: check method started
2008-01-16 16:08 StatusChecker: processing host head0
2008-01-16 16:08 StatusChecker: processing domain webconsole
2008-01-16 16:08 StatusChecker: processing host host1
2008-01-16 16:08 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:08 StatusChecker: check method completed
2008-01-16 16:09 StatusChecker: check method started
2008-01-16 16:09 StatusChecker: processing host head0
2008-01-16 16:09 StatusChecker: processing domain webconsole
2008-01-16 16:09 StatusChecker: processing host host1
2008-01-16 16:09 StatusChecker: error encountered for host host1:  
RuntimeError: Unable to ssh to 192.168.1.2 as root
2008-01-16 16:09 StatusChecker: check method completed
2008-01-16 16:10 StatusChecker: check method started
2008-01-16 16:10 StatusChecker: processing host head0
2008-01-16 16:10 StatusChecker: processing domain webconsole
2008-01-16 16:10 StatusChecker: processing host host1

After 16:10 the worker seems to have disappeared. I found the  
following message in backgroundrb_debug.log:

/home/webcon/apps/webconsole/releases/20080115103838/vendor/plugins/ 
backgroundrb/server/meta_worker.rb:329:in `check_for
_timer_events': undefined method `info' for nil:NilClass (NoMethodError)
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/core.rb:148:in `
start_reactor'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/core.rb:147:in `
loop'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/core.rb:147:in `
start_reactor'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/worker.rb:21:in
`start_worker'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/packet_master.rb
:134:in `fork_and_load'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/packet_master.rb
:98:in `load_workers'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/packet_master.rb
:93:in `each'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/packet_master.rb
:93:in `load_workers'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/framework/packet_master.rb
:19:in `run'
         from /home/webcon/apps/webconsole/releases/20080115103838/ 
vendor/plugins/backgroundrb/server/master_worker.rb:16
3:in `initialize'
         from script/backgroundrb:41:in `new'
         from script/backgroundrb:41


The code for this worker can be found here:

   https://svn.sxven.com/webconsole/trunk/lib/workers/status_checker.rb


Any pointers on how to debug this is much appreciated.

cheers,
mengkuan

More information about the Backgroundrb-devel mailing list