[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster

Ezra Zygmuntowicz ezmobius at gmail.com
Mon Mar 5 16:08:14 EST 2007


Mathew-

	This may be the same worker class name length overflow error. If  
your classname for your worker is too long then slave can fail to  
create the proper unix domain socket. Can you try to shorten the name  
of your worker class and try again?

-Ezra


On Mar 5, 2007, at 12:40 PM, Matthew wrote:

> Thanks for the suggestion, Mason. I did that and noticed that each  
> time I attempted to create a new worker, I got a new entry. This is  
> what I ended up with in netstat after trying several attempts:
>
> -bash-2.05b$ netstat -an | grep 2001
> c4cb5af0 stream      0      0        0 c8cb5af0        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c8d58a64 stream      0      0        0        0        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c4dcc94c stream      0      0        0 c97ac94c        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c8d20690 stream      0      0        0 c8cb5000        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c8cca348 stream      0      0        0 c4dcc690        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c8cf7b7c stream      0      0        0 c8d209d8        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
> c4cb62bc stream      0      0 c6b63d68        0        0         
> 0 ///tmp/backgroundrbunix_localhost_2001
>
> My process list looked like this:
>
> -bash-2.05b$ ps aux | grep ruby
> cwarner 30993 31.9  1.3 46932 39748  ??  R     3:27PM   2: 22.28  
> ruby: pdf_generator_worker_dde6f63034382c4727
> cwarner 31131 31.6  1.3 47344 40160  ??  R     3:30PM   0:26.14  
> ruby: pdf_generator_worker_031025c9be743ca12e
> cwarner 31079 31.5  1.3 47072 39888  ??  R     3:29PM   0: 43.74  
> ruby: pdf_generator_worker_c617aa5cc57c7b59b7
> cwarner 23921  0.1  3.1 103420 97144  ??  S    10:35AM   6:08.78  
> ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23910  0.0  4.5 148668 140424  ??  S    10:35AM  11: 28.06  
> ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23914  0.0  3.1 107792 98560  ??  S    10:35AM   7:13.36  
> ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23918  0.0  3.1 106548 97144  ??  S    10:35AM   6: 39.64  
> ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 30836  0.0  1.3 47480 40300  ??  S     3:24PM   0:02.42  
> ruby: backgroundrb (ruby)
> cwarner 30837  0.0  0.7 21512 20552  ??  S     3:24PM   0:00.06  
> ruby: backgroundrb_logger (ruby)
> cwarner 30838  0.0  0.7 21468 20560  ??  S     3:24PM   0:00.06  
> ruby: backgroundrb_results (ruby)
> cwarner 31162  0.0  0.0  1484   832  p3  S+    3:31PM   0:00.00  
> grep ruby
>
> Looks like a new pdf_generator_worker process gets created and  
> immediately freezes each time I try to create a worker. I can't  
> communicate with any of these workers and have to manually kill  
> them to get them to go away.
>
> Anyone else have an idea of what could be going on?
>
> Matthew
>
>
> On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew -
>
> Trying running netstat to see what ports your mongrels and  
> backgroundrb processes are using. I've run into some problems with  
> those two trying to use the same ports in past.
>
> Mason
>
> On 3/3/07, Matthew <pelargir at gmail.com> wrote:
> I've been troubleshooting this problem for several hours now and  
> haven't been able to figure out what's wrong. I have two boxes, dev  
> and prod, both are identical except that prod uses Ruby 1.8.5 and  
> dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and  
> Backroundrb 0.2.1.
>
> The behavior I'm seeing is this. When I first start Backgroundrb,  
> this is what the process list looks like:
>
> cwarner  80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 / 
> usr/local/bin/ruby18 /usr/local/bin/mon
> cwarner  80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 / 
> usr/local/bin/ruby18 /usr/local/bin/mon
> cwarner  80614  0.0  1.6 57944 50628  ??  S    10:28PM   0:03.48 / 
> usr/local/bin/ruby18 /usr/local/bin/mon
> cwarner  80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 / 
> usr/local/bin/ruby18 /usr/local/bin/mon
> cwarner  80804  0.0  1.3 46872 39688  ??  S    10:37PM   0:02.43  
> ruby: backgroundrb (ruby)
> cwarner  80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07  
> ruby: backgroundrb_logger (ruby)
> cwarner  80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06  
> ruby: backgroundrb_results (ruby)
>
> My four Mongrels in the cluster sitting out there, and the three  
> backgroundrb processes. I then proceed to generate a worker using  
> MiddleMan via IRB, get the auto generated job key, look up the  
> worker, and then delete him. All goes well, and the process list  
> still looks like what I posted above. I can do this as many times  
> as I want to via IRB and nothing goes wrong. However, when I try to  
> call the same code through a Rails action, Backgroundrb doesn't  
> behave. The browser just sits there spinning. Examining the process  
> list yields:
>
> cwarner 80934 92.8  1.3 46936 39752  ??  R    10:45PM   0:02.07  
> ruby: backgroundrb (ruby)
> cwarner 80804  0.1  1.3 47072 39888  ??  S    10:37PM   0:02.60  
> ruby: backgroundrb (ruby)
> cwarner 80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 / 
> usr/local/bin/ruby18 /usr/local/bin/mong
> cwarner 80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 / 
> usr/local/bin/ruby18 /usr/local/bin/mong
> cwarner 80614  0.0  1.6 57944 50628  ??  R    10:28PM   0:03.51 / 
> usr/local/bin/ruby18 /usr/local/bin/mong
> cwarner 80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 / 
> usr/local/bin/ruby18 /usr/local/bin/mong
> cwarner 80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07  
> ruby: backgroundrb_logger (ruby)
> cwarner 80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06  
> ruby: backgroundrb_results (ruby)
> cwarner 80936  0.0  0.0  1484   828  p3  S+   10:45PM   0:00.00  
> grep ruby
>
> Aha! An extra Backgroundrb process that's spiking the CPU. I've let  
> the browser sit there until it times out, and the Backgroundrb  
> process will still be there, spiking the CPU. If I stop  
> Backgroundrb at this point, the extra process is still there. I  
> have to kill it manually to get rid of it.
>
> Something else I've noticed is that, once this process is out  
> there, even if I try creating new workers via IRB, the call to  
> MiddleMan.new_worker just sits there. So this definitely appears to  
> be a problem with the way Backgroundrb and my Mongrel clusters are  
> interacting, but I'm not sure where to even begin. It's frustrating  
> that this isn't happening on the dev box. I've tried to figure out  
> what could be different between the boxes, but other than the Ruby  
> versions, I'm not seeing anything.
>
> I'm out of options. Does anyone have any ideas?
>
> Thanks,
> Matthew
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel

-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)




More information about the Backgroundrb-devel mailing list