[Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster

Matthew pelargir at gmail.com
Wed Mar 7 14:23:09 EST 2007


Good news. I solved the problem by downgrading to Ruby 1.8.4 to match the
dev box. Backgroundrb immediately started behaving correctly. There must be
some conflict between Backgroundrb and Mongrel when running under Ruby
1.8.5on FreeBSD. I'm running
1.8.5 locally and Backgroundrb worked. Very strange, but I'm very glad to
have this resolved now. Backgroundrb is quite solid on the prod box now.

I have another question, but I'm going to post it in a new thread since it
has nothing to do with this topic.

Matthew


On 3/5/07, Matthew <pelargir at gmail.com> wrote:
>
> Thanks for the reply, Ezra. Just tried that and I'm still experiencing the
> same problem.
>
> As before, running from either script/console or the browser works fine on
> the first go round. I can generate a worker, query it, and delete it. Trying
> to generate a worker the second time is a different story, though. In the
> browser the generate appears to work, but when I try to get a worker for the
> returned key I get a nil object. In script/console, the generate just sits
> there spinning until I kill the backgroundrb process.
>
> Something interesting I noticed as well is the dir that gets created in
> /tmp with the PID on the end does not match the actual PID of my
> backgroundrb process. Is it supposed to?
>
> -bash-2.05b$ ls /tmp/ba*
> /tmp/backgroundrbunix_localhost_2020
> /tmp/backgroundrb.32589
>
> -bash-2.05b$ ps aux | grep backgroundrb
> cwarner 32591  0.0  0.7 21528 20564  ??  S     4:13PM   0:00.17 ruby:
> backgroundrb (ruby)
> cwarner 32592  0.0  0.7 21512 20552  ??  S     4:13PM   0:00.06 ruby:
> backgroundrb_logger (ruby)
> cwarner 32593  0.0  0.7 21468 20560  ??  S     4:13PM   0:00.05 ruby:
> backgroundrb_results (ruby)
>
>
> On an aside, thanks for building a really great tool here. It's simple to
> use and perfect for what I'm trying to do. I'm sure the issue is on my end
> with configuration or something.
>
> Matthew
>
>
> On 3/5/07, Ezra Zygmuntowicz <ezmobius at gmail.com> wrote:
> >
> > Mathew-
> >
> >         This may be the same worker class name length overflow error. If
> > your classname for your worker is too long then slave can fail to
> > create the proper unix domain socket. Can you try to shorten the name
> > of your worker class and try again?
> >
> > -Ezra
> >
> >
> > On Mar 5, 2007, at 12:40 PM, Matthew wrote:
> >
> > > Thanks for the suggestion, Mason. I did that and noticed that each
> > > time I attempted to create a new worker, I got a new entry. This is
> > > what I ended up with in netstat after trying several attempts:
> > >
> > > -bash-2.05b$ netstat -an | grep 2001
> > > c4cb5af0 stream      0      0        0 c8cb5af0        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c8d58a64 stream      0      0        0        0        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c4dcc94c stream      0      0        0 c97ac94c        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c8d20690 stream      0      0        0 c8cb5000        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c8cca348 stream      0      0        0 c4dcc690        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c8cf7b7c stream      0      0        0 c8d209d8        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > > c4cb62bc stream      0      0 c6b63d68        0        0
> > > 0 ///tmp/backgroundrbunix_localhost_2001
> > >
> > > My process list looked like this:
> > >
> > > -bash-2.05b$ ps aux | grep ruby
> > > cwarner 30993 31.9  1.3 46932 39748  ??  R     3:27PM   2: 22.28
> > > ruby: pdf_generator_worker_dde6f63034382c4727
> > > cwarner 31131 31.6  1.3 47344 40160  ??  R     3:30PM   0:26.14
> > > ruby: pdf_generator_worker_031025c9be743ca12e
> > > cwarner 31079 31.5  1.3 47072 39888  ??  R     3:29PM   0: 43.74
> > > ruby: pdf_generator_worker_c617aa5cc57c7b59b7
> > > cwarner 23921  0.1  3.1 103420 97144  ??  S    10:35AM   6:08.78
> > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > > cwarner 23910  0.0  4.5 148668 140424  ??  S    10:35AM  11: 28.06
> > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > > cwarner 23914  0.0  3.1 107792 98560  ??  S    10:35AM   7:13.36
> > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > > cwarner 23918  0.0  3.1 106548 97144  ??  S    10:35AM   6: 39.64
> > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre
> > > cwarner 30836  0.0  1.3 47480 40300  ??  S     3:24PM   0:02.42
> > > ruby: backgroundrb (ruby)
> > > cwarner 30837  0.0  0.7 21512 20552  ??  S     3:24PM   0:00.06
> > > ruby: backgroundrb_logger (ruby)
> > > cwarner 30838  0.0  0.7 21468 20560  ??  S     3:24PM   0:00.06
> > > ruby: backgroundrb_results (ruby)
> > > cwarner 31162  0.0  0.0  1484   832  p3  S+    3:31PM   0:00.00
> > > grep ruby
> > >
> > > Looks like a new pdf_generator_worker process gets created and
> > > immediately freezes each time I try to create a worker. I can't
> > > communicate with any of these workers and have to manually kill
> > > them to get them to go away.
> > >
> > > Anyone else have an idea of what could be going on?
> > >
> > > Matthew
> > >
> > >
> > > On 3/4/07, Mason Hale <masonhale at gmail.com > wrote:Matthew -
> > >
> > > Trying running netstat to see what ports your mongrels and
> > > backgroundrb processes are using. I've run into some problems with
> > > those two trying to use the same ports in past.
> > >
> > > Mason
> > >
> > > On 3/3/07, Matthew < pelargir at gmail.com> wrote:
> > > I've been troubleshooting this problem for several hours now and
> > > haven't been able to figure out what's wrong. I have two boxes, dev
> > > and prod, both are identical except that prod uses Ruby 1.8.5 and
> > > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and
> > > Backroundrb 0.2.1.
> > >
> > > The behavior I'm seeing is this. When I first start Backgroundrb,
> > > this is what the process list looks like:
> > >
> > > cwarner  80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 /
> > > usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 /
> > > usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80614  0.0  1.6 57944 50628  ??  S    10:28PM   0:03.48 /
> > > usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 /
> > > usr/local/bin/ruby18 /usr/local/bin/mon
> > > cwarner  80804  0.0  1.3 46872 39688  ??  S    10:37PM   0:02.43
> > > ruby: backgroundrb (ruby)
> > > cwarner  80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07
> > > ruby: backgroundrb_logger (ruby)
> > > cwarner  80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06
> > > ruby: backgroundrb_results (ruby)
> > >
> > > My four Mongrels in the cluster sitting out there, and the three
> > > backgroundrb processes. I then proceed to generate a worker using
> > > MiddleMan via IRB, get the auto generated job key, look up the
> > > worker, and then delete him. All goes well, and the process list
> > > still looks like what I posted above. I can do this as many times
> > > as I want to via IRB and nothing goes wrong. However, when I try to
> > > call the same code through a Rails action, Backgroundrb doesn't
> > > behave. The browser just sits there spinning. Examining the process
> > > list yields:
> > >
> > > cwarner 80934 92.8  1.3 46936 39752  ??  R    10:45PM   0:02.07
> > > ruby: backgroundrb (ruby)
> > > cwarner 80804  0.1  1.3 47072 39888  ??  S    10:37PM   0:02.60
> > > ruby: backgroundrb (ruby)
> > > cwarner 80608  0.0  1.6 57852 50480  ??  S    10:28PM   0: 03.36 /
> > > usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80611  0.0  1.6 58768 51356  ??  S    10:28PM   0:03.59 /
> > > usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80614  0.0  1.6 57944 50628  ??  R    10:28PM   0:03.51 /
> > > usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80618  0.0  1.6 57384 50060  ??  S    10:28PM   0:03.26 /
> > > usr/local/bin/ruby18 /usr/local/bin/mong
> > > cwarner 80805  0.0  0.7 21512 20548  ??  S    10:37PM   0:00.07
> > > ruby: backgroundrb_logger (ruby)
> > > cwarner 80806  0.0  0.7 21468 20556  ??  S    10:37PM   0:00.06
> > > ruby: backgroundrb_results (ruby)
> > > cwarner 80936  0.0  0.0  1484   828  p3  S+   10:45PM   0:00.00
> > > grep ruby
> > >
> > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let
> > > the browser sit there until it times out, and the Backgroundrb
> > > process will still be there, spiking the CPU. If I stop
> > > Backgroundrb at this point, the extra process is still there. I
> > > have to kill it manually to get rid of it.
> > >
> > > Something else I've noticed is that, once this process is out
> > > there, even if I try creating new workers via IRB, the call to
> > > MiddleMan.new_worker just sits there. So this definitely appears to
> > > be a problem with the way Backgroundrb and my Mongrel clusters are
> > > interacting, but I'm not sure where to even begin. It's frustrating
> > > that this isn't happening on the dev box. I've tried to figure out
> > > what could be different between the boxes, but other than the Ruby
> > > versions, I'm not seeing anything.
> > >
> > > I'm out of options. Does anyone have any ideas?
> > >
> > > Thanks,
> > > Matthew
> > >
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> > >
> > >
> > > _______________________________________________
> > > Backgroundrb-devel mailing list
> > > Backgroundrb-devel at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
> > -- Ezra Zygmuntowicz
> > -- Lead Rails Evangelist
> > -- ez at engineyard.com
> > -- Engine Yard, Serious Rails Hosting
> > -- (866) 518-YARD (9273)
> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/0354fced/attachment-0001.html 


More information about the Backgroundrb-devel mailing list