[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster
Matthew
pelargir at gmail.com
Mon Mar 5 15:49:25 EST 2007
Something I'm also noticing that may be related is this message. I get it
every so often when I try to stop backgroundrb:
-bash-2.05b$ script/backgroundrb stop
/usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`initialize': No such file or directory -
/usr/home/cwarner/rails/log/backgroundrb.pid (Errno::ENOENT)
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`open'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in
`pid'
from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:350:in
`stop'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:133:in
`stop_all'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`each'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in
`stop_all'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:74:in
`run'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in
`run_proc'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`call'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in
`catch_exceptions'
from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in
`run_proc'
from
/usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in
`run'
from script/backgroundrb:29
I'm also seeing a file called "backgroundrb.ppid" (yes, with two Ps) in my
log dir. "backgroundrb.pid" shows up there when backgroundrb is running, but
it gets deleted when I stop. On the other hand, "backgroundrb.ppid" stays
there unless I delete it manually.
Matthew
On 3/5/07, Matthew <pelargir at gmail.com> wrote:
>
> Thanks for the suggestion, Mason. I did that and noticed that each time I
> attempted to create a new worker, I got a new entry. This is what I ended up
> with in netstat after trying several attempts:
>
> -bash-2.05b$ netstat -an | grep 2001
> c4cb5af0 stream 0 0 0 c8cb5af0 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8d58a64 stream 0 0 0 0 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c4dcc94c stream 0 0 0 c97ac94c 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8d20690 stream 0 0 0 c8cb5000 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8cca348 stream 0 0 0 c4dcc690 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c8cf7b7c stream 0 0 0 c8d209d8 0 0
> ///tmp/backgroundrbunix_localhost_2001
> c4cb62bc stream 0 0 c6b63d68 0 0 0
> ///tmp/backgroundrbunix_localhost_2001
>
> My process list looked like this:
>
> -bash-2.05b$ ps aux | grep ruby
> cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 ruby:
> pdf_generator_worker_dde6f63034382c4727
> cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 ruby:
> pdf_generator_worker_031025c9be743ca12e
> cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 ruby:
> pdf_generator_worker_c617aa5cc57c7b59b7
> cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 ruby
> /usr/local/lib/ruby/gems/1.8/gems/mongre
> cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 ruby:
> backgroundrb (ruby)
> cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 ruby:
> backgroundrb_logger (ruby)
> cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 ruby:
> backgroundrb_results (ruby)
> cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 grep ruby
>
> Looks like a new pdf_generator_worker process gets created and immediately
> freezes each time I try to create a worker. I can't communicate with any of
> these workers and have to manually kill them to get them to go away.
>
> Anyone else have an idea of what could be going on?
>
> Matthew
>
>
> On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:
> >
> > Matthew -
> >
> > Trying running netstat to see what ports your mongrels and backgroundrb
> > processes are using. I've run into some problems with those two trying to
> > use the same ports in past.
> >
> > Mason
> >
> > On 3/3/07, Matthew <pelargir at gmail.com> wrote:
> >
> > > I've been troubleshooting this problem for several hours now and
> > > haven't been able to figure out what's wrong. I have two boxes, dev and
> > > prod, both are identical except that prod uses Ruby 1.8.5 and dev uses
> > > 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb
> > > 0.2.1.
> > >
> > > The behavior I'm seeing is this. When I first start Backgroundrb, this
> > > is what the process list looks like:
> > >
> > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby:
> > > backgroundrb (ruby)
> > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby:
> > > backgroundrb_results (ruby)
> > >
> > > My four Mongrels in the cluster sitting out there, and the three
> > > backgroundrb processes. I then proceed to generate a worker using MiddleMan
> > > via IRB, get the auto generated job key, look up the worker, and then delete
> > > him. All goes well, and the process list still looks like what I posted
> > > above. I can do this as many times as I want to via IRB and nothing goes
> > > wrong. However, when I try to call the same code through a Rails action,
> > > Backgroundrb doesn't behave. The browser just sits there spinning. Examining
> > > the process list yields:
> > >
> > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby:
> > > backgroundrb (ruby)
> > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby:
> > > backgroundrb (ruby)
> > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby:
> > > backgroundrb_logger (ruby)
> > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby:
> > > backgroundrb_results (ruby)
> > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep
> > > ruby
> > >
> > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let
> > > the browser sit there until it times out, and the Backgroundrb process will
> > > still be there, spiking the CPU. If I stop Backgroundrb at this point, the
> > > extra process is still there. I have to kill it manually to get rid of it.
> > >
> > > Something else I've noticed is that, once this process is out there,
> > > even if I try creating new workers via IRB, the call to
> > > MiddleMan.new_worker just sits there. So this definitely appears to be
> > > a problem with the way Backgroundrb and my Mongrel clusters are interacting,
> > > but I'm not sure where to even begin. It's frustrating that this isn't
> > > happening on the dev box. I've tried to figure out what could be different
> > > between the boxes, but other than the Ruby versions, I'm not seeing
> > > anything.
> > >
> > > I'm out of options. Does anyone have any ideas?
> > >
> > > Thanks,
> > > Matthew
> > >
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/e1134c2d/attachment.html
More information about the Backgroundrb-devel
mailing list