[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster

Matthew pelargir at gmail.com
Mon Mar 5 15:49:25 EST 2007


Something I'm also noticing that may be related is this message. I get it
every so often when I try to stop backgroundrb:

-bash-2.05b$ script/backgroundrb stop
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`initialize': No such file or directory -
/usr/home/cwarner/rails/log/backgroundrb.pid (Errno::ENOENT)
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`open'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`pid'
        from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:350:in
`stop'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:133:in
`stop_all'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`each'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`stop_all'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:74:in
`run'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in
`run_proc'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`call'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`catch_exceptions'
        from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in
`run_proc'
        from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in
`run'
        from script/backgroundrb:29

I'm also seeing a file called "backgroundrb.ppid" (yes, with two Ps) in my
log dir. "backgroundrb.pid" shows up there when backgroundrb is running, but
it gets deleted when I stop. On the other hand, "backgroundrb.ppid" stays
there unless I delete it manually.

Matthew


On 3/5/07, Matthew <pelargir at gmail.com> wrote:
>
> Thanks for the suggestion, Mason. I did that and noticed that each time I
> attempted to create a new worker, I got a new entry. This is what I ended up
> with in netstat after trying several attempts:
>
> -bash-2.05b$ netstat -an | grep 2001
> c4cb5af0 stream      0      0        0 c8cb5af0        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c8d58a64 stream      0      0        0        0        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c4dcc94c stream      0      0        0 c97ac94c        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c8d20690 stream      0      0        0 c8cb5000        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c8cca348 stream      0      0        0 c4dcc690        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c8cf7b7c stream      0      0        0 c8d209d8        0        0
> ///tmp/backgroundrbunix_localhost_2001
> c4cb62bc stream      0      0 c6b63d68        0        0        0
> ///tmp/backgroundrbunix_localhost_2001
>
> My process list looked like this:
>
> -bash-2.05b$ ps aux | grep ruby
> cwarner 30993 31.9  1.3 46932 39748  ??  R     3:27PM   2: 22.28 ruby:
> pdf_generator_worker_dde6f63034382c4727
> cwarner 31131 31.6  1.3 47344 40160  ??  R     3:30PM   0:26.14 ruby:
> pdf_generator_worker_031025c9be743ca12e
> cwarner 31079 31.5  1.3 47072 39888  ??  R     3:29PM   0: 43.74 ruby:
> pdf_generator_worker_c617aa5cc57c7b59b7
> cwarner 23921  0.1  3.1 103420 97144  ??  S    10:35AM   6:08.78 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23910  0.0  4.5 148668 140424  ??  S    10:35AM  11: 28.06 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23914  0.0  3.1 107792 98560  ??  S    10:35AM   7:13.36 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23918  0.0  3.1 106548 97144  ??  S    10:35AM   6: 39.64 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 30836  0.0  1.3 47480 40300  ??  S     3:24PM   0:02.42 ruby:
> backgroundrb (ruby)
> cwarner 30837  0.0  0.7 21512 20552  ??  S     3:24PM   0:00.06 ruby:
> backgroundrb_logger (ruby)
> cwarner 30838  0.0  0.7 21468 20560  ??  S     3:24PM   0:00.06 ruby:
> backgroundrb_results (ruby)
> cwarner 31162  0.0  0.0  1484   832  p3  S+    3:31PM   0:00.00 grep ruby
>
> Looks like a new pdf_generator_worker process gets created and immediately
> freezes each time I try to create a worker. I can't communicate with any of
> these workers and have to manually kill them to get them to go away.
>
> Anyone else have an idea of what could be going on?
>
> Matthew
>
>
> On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:
> >
> > Matthew -
> >
> > Trying running netstat to see what ports your mongrels and backgroundrb
> > processes are using. I've run into some problems with those two trying to
> > use the same ports in past.
> >
> > Mason
> >
> > On 3/3/07, Matthew <pelargir at gmail.com> wrote:
> >
> > >  I've been troubleshooting this problem for several hours now and
> > > haven't been able to figure out what's wrong. I have two boxes, dev and
> > > prod, both are identical except that prod uses Ruby 1.8.5 and dev uses
> > > 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb
> > > 0.2.1.
> > >
> > > The behavior I'm seeing is this. When I first start Backgroundrb, this
> > > is what the process list looks like:
> > >
> > > cwarner  80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80614  0.0  1.6 57944 50628  ??  S    10:28PM   0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80804  0.0  1.3 46872 39688  ??  S    10:37PM   0:02.43 ruby:
> > > backgroundrb (ruby)
> > > cwarner  80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07 ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner  80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06 ruby:
> > > backgroundrb_results (ruby)
> > >
> > > My four Mongrels in the cluster sitting out there, and the three
> > > backgroundrb processes. I then proceed to generate a worker using MiddleMan
> > > via IRB, get the auto generated job key, look up the worker, and then delete
> > > him. All goes well, and the process list still looks like what I posted
> > > above. I can do this as many times as I want to via IRB and nothing goes
> > > wrong. However, when I try to call the same code through a Rails action,
> > > Backgroundrb doesn't behave. The browser just sits there spinning. Examining
> > > the process list yields:
> > >
> > > cwarner 80934 92.8  1.3 46936 39752  ??  R    10:45PM   0:02.07 ruby:
> > > backgroundrb (ruby)
> > > cwarner 80804  0.1  1.3 47072 39888  ??  S    10:37PM   0:02.60 ruby:
> > > backgroundrb (ruby)
> > > cwarner 80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80614  0.0  1.6 57944 50628  ??  R    10:28PM   0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07 ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner 80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06 ruby:
> > > backgroundrb_results (ruby)
> > > cwarner 80936  0.0  0.0  1484   828  p3  S+   10:45PM   0:00.00 grep
> > > ruby
> > >
> > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let
> > > the browser sit there until it times out, and the Backgroundrb process will
> > > still be there, spiking the CPU. If I stop Backgroundrb at this point, the
> > > extra process is still there. I have to kill it manually to get rid of it.
> > >
> > > Something else I've noticed is that, once this process is out there,
> > > even if I try creating new workers via IRB, the call to
> > > MiddleMan.new_worker just sits there. So this definitely appears to be
> > > a problem with the way Backgroundrb and my Mongrel clusters are interacting,
> > > but I'm not sure where to even begin. It's frustrating that this isn't
> > > happening on the dev box. I've tried to figure out what could be different
> > > between the boxes, but other than the Ruby versions, I'm not seeing
> > > anything.
> > >
> > > I'm out of options. Does anyone have any ideas?
> > >
> > > Thanks,
> > > Matthew
> > >
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/e1134c2d/attachment.html 


More information about the Backgroundrb-devel mailing list