[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster

Matthew pelargir at gmail.com
Mon Mar 5 16:25:01 EST 2007


Thanks for the reply, Ezra. Just tried that and I'm still experiencing the
same problem.

As before, running from either script/console or the browser works fine on
the first go round. I can generate a worker, query it, and delete it. Trying
to generate a worker the second time is a different story, though. In the
browser the generate appears to work, but when I try to get a worker for the
returned key I get a nil object. In script/console, the generate just sits
there spinning until I kill the backgroundrb process.

Something interesting I noticed as well is the dir that gets created in /tmp
with the PID on the end does not match the actual PID of my backgroundrb
process. Is it supposed to?

-bash-2.05b$ ls /tmp/ba*
/tmp/backgroundrbunix_localhost_2020
/tmp/backgroundrb.32589

-bash-2.05b$ ps aux | grep backgroundrb
cwarner 32591  0.0  0.7 21528 20564  ??  S     4:13PM   0:00.17 ruby:
backgroundrb (ruby)
cwarner 32592  0.0  0.7 21512 20552  ??  S     4:13PM   0:00.06 ruby:
backgroundrb_logger (ruby)
cwarner 32593  0.0  0.7 21468 20560  ??  S     4:13PM   0:00.05 ruby:
backgroundrb_results (ruby)


On an aside, thanks for building a really great tool here. It's simple to
use and perfect for what I'm trying to do. I'm sure the issue is on my end
with configuration or something.

Matthew


On 3/5/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
>
> Mathew-
>
>         This may be the same worker class name length overflow error. If
> your classname for your worker is too long then slave can fail to
> create the proper unix domain socket. Can you try to shorten the name
> of your worker class and try again?
>
> -Ezra
>
>
> On Mar 5, 2007, at 12:40 PM, Matthew wrote:
>
> > Thanks for the suggestion, Mason. I did that and noticed that each
> > time I attempted to create a new worker, I got a new entry. This is
> > what I ended up with in netstat after trying several attempts:
> >
> > -bash-2.05b$ netstat -an | grep 2001
> > c4cb5af0 stream      0      0        0 c8cb5af0        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c8d58a64 stream      0      0        0        0        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c4dcc94c stream      0      0        0 c97ac94c        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c8d20690 stream      0      0        0 c8cb5000        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c8cca348 stream      0      0        0 c4dcc690        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c8cf7b7c stream      0      0        0 c8d209d8        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> > c4cb62bc stream      0      0 c6b63d68        0        0
> > 0 ///tmp/backgroundrbunix_localhost_2001
> >
> > My process list looked like this:
> >
> > -bash-2.05b$ ps aux | grep ruby
> > cwarner 30993 31.9  1.3 46932 39748  ??  R     3:27PM   2: 22.28
> > ruby: pdf_generator_worker_dde6f63034382c4727
> > cwarner 31131 31.6  1.3 47344 40160  ??  R     3:30PM   0:26.14
> > ruby: pdf_generator_worker_031025c9be743ca12e
> > cwarner 31079 31.5  1.3 47072 39888  ??  R     3:29PM   0: 43.74
> > ruby: pdf_generator_worker_c617aa5cc57c7b59b7
> > cwarner 23921  0.1  3.1 103420 97144  ??  S    10:35AM   6:08.78
> > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > cwarner 23910  0.0  4.5 148668 140424  ??  S    10:35AM  11: 28.06
> > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > cwarner 23914  0.0  3.1 107792 98560  ??  S    10:35AM   7:13.36
> > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > cwarner 23918  0.0  3.1 106548 97144  ??  S    10:35AM   6: 39.64
> > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > cwarner 30836  0.0  1.3 47480 40300  ??  S     3:24PM   0:02.42
> > ruby: backgroundrb (ruby)
> > cwarner 30837  0.0  0.7 21512 20552  ??  S     3:24PM   0:00.06
> > ruby: backgroundrb_logger (ruby)
> > cwarner 30838  0.0  0.7 21468 20560  ??  S     3:24PM   0:00.06
> > ruby: backgroundrb_results (ruby)
> > cwarner 31162  0.0  0.0  1484   832  p3  S+    3:31PM   0:00.00
> > grep ruby
> >
> > Looks like a new pdf_generator_worker process gets created and
> > immediately freezes each time I try to create a worker. I can't
> > communicate with any of these workers and have to manually kill
> > them to get them to go away.
> >
> > Anyone else have an idea of what could be going on?
> >
> > Matthew
> >
> >
> > On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew -
> >
> > Trying running netstat to see what ports your mongrels and
> > backgroundrb processes are using. I've run into some problems with
> > those two trying to use the same ports in past.
> >
> > Mason
> >
> > On 3/3/07, Matthew <pelargir at gmail.com> wrote:
> > I've been troubleshooting this problem for several hours now and
> > haven't been able to figure out what's wrong. I have two boxes, dev
> > and prod, both are identical except that prod uses Ruby 1.8.5 and
> > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and
> > Backroundrb 0.2.1.
> >
> > The behavior I'm seeing is this. When I first start Backgroundrb,
> > this is what the process list looks like:
> >
> > cwarner  80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 /
> > usr/local/bin/ruby18 /usr/local/bin/mon
> > cwarner  80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 /
> > usr/local/bin/ruby18 /usr/local/bin/mon
> > cwarner  80614  0.0  1.6 57944 50628  ??  S    10:28PM   0:03.48 /
> > usr/local/bin/ruby18 /usr/local/bin/mon
> > cwarner  80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 /
> > usr/local/bin/ruby18 /usr/local/bin/mon
> > cwarner  80804  0.0  1.3 46872 39688  ??  S    10:37PM   0:02.43
> > ruby: backgroundrb (ruby)
> > cwarner  80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07
> > ruby: backgroundrb_logger (ruby)
> > cwarner  80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06
> > ruby: backgroundrb_results (ruby)
> >
> > My four Mongrels in the cluster sitting out there, and the three
> > backgroundrb processes. I then proceed to generate a worker using
> > MiddleMan via IRB, get the auto generated job key, look up the
> > worker, and then delete him. All goes well, and the process list
> > still looks like what I posted above. I can do this as many times
> > as I want to via IRB and nothing goes wrong. However, when I try to
> > call the same code through a Rails action, Backgroundrb doesn't
> > behave. The browser just sits there spinning. Examining the process
> > list yields:
> >
> > cwarner 80934 92.8  1.3 46936 39752  ??  R    10:45PM   0:02.07
> > ruby: backgroundrb (ruby)
> > cwarner 80804  0.1  1.3 47072 39888  ??  S    10:37PM   0:02.60
> > ruby: backgroundrb (ruby)
> > cwarner 80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 /
> > usr/local/bin/ruby18 /usr/local/bin/mong
> > cwarner 80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 /
> > usr/local/bin/ruby18 /usr/local/bin/mong
> > cwarner 80614  0.0  1.6 57944 50628  ??  R    10:28PM   0:03.51 /
> > usr/local/bin/ruby18 /usr/local/bin/mong
> > cwarner 80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 /
> > usr/local/bin/ruby18 /usr/local/bin/mong
> > cwarner 80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07
> > ruby: backgroundrb_logger (ruby)
> > cwarner 80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06
> > ruby: backgroundrb_results (ruby)
> > cwarner 80936  0.0  0.0  1484   828  p3  S+   10:45PM   0:00.00
> > grep ruby
> >
> > Aha! An extra Backgroundrb process that's spiking the CPU. I've let
> > the browser sit there until it times out, and the Backgroundrb
> > process will still be there, spiking the CPU. If I stop
> > Backgroundrb at this point, the extra process is still there. I
> > have to kill it manually to get rid of it.
> >
> > Something else I've noticed is that, once this process is out
> > there, even if I try creating new workers via IRB, the call to
> > MiddleMan.new_worker just sits there. So this definitely appears to
> > be a problem with the way Backgroundrb and my Mongrel clusters are
> > interacting, but I'm not sure where to even begin. It's frustrating
> > that this isn't happening on the dev box. I've tried to figure out
> > what could be different between the boxes, but other than the Ruby
> > versions, I'm not seeing anything.
> >
> > I'm out of options. Does anyone have any ideas?
> >
> > Thanks,
> > Matthew
> >
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
> >
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
> -- Ezra Zygmuntowicz
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/4fafd351/attachment-0001.html 


More information about the Backgroundrb-devel mailing list