From david.gurba at gmail.com Fri Mar 2 02:02:54 2007 From: david.gurba at gmail.com (David Gurba) Date: Thu, 1 Mar 2007 23:02:54 -0800 Subject: [Backgroundrb-devel] example standalone error. Message-ID: Hello, Trying to get the example to work in standalone mode. I have the following in a file and am not certain why I get this undefined method error: error: ~/ruby sample_app.rb my sample app :) sample_app.rb:9: undefined method `new_worker' for BackgrounDRb::MiddleMan:Class (NoMethodError) ### code follows ### # sample_app ... try to get stand-alone happiness. #require 'release-0.2.1/lib/backgroundrb' #MiddleManDRbObject.init(:uri =>'drbunix:///tmp/backgroundrbunix_localhost_2000') require 'backgroundrb/middleman' include BackgrounDRb puts 'my sample app :)' key = MiddleMan.new_worker(:class => :example_worker) worker = MiddleMan.worker(key) worker.other_method worker.delete # now that my job is done (and my worker is actually deleted on the server) view # my results ... cause now I'lm interested. p worker.results.to_hash puts 'bye bye :)' I kinda confused because most of the examples online and in /examples assume that one is using Rails for this. Its kind of difficult to see exactly where require or init calls need to go. regards, David G. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070301/81c2a77f/attachment.html From david.gurba at gmail.com Fri Mar 2 02:08:46 2007 From: david.gurba at gmail.com (David Gurba) Date: Thu, 1 Mar 2007 23:08:46 -0800 Subject: [Backgroundrb-devel] example standalone error ... but if i ... Message-ID: If I add .instance in front of new_worker I get the following error ... but for some reason i cant call new_worker directly...? error: ruby sample_app.rb my sample app :) ./backgroundrb/middleman.rb:440:in `ex': undefined method `synchronize' for nil:NilClass (NoMethodError) from ./backgroundrb/middleman.rb:401:in `[]' from ./backgroundrb/middleman.rb:461:in `gen_key' from ./backgroundrb/middleman.rb:173:in `new_worker' from ./backgroundrb/middleman.rb:173:in `new_worker' from sample_app.rb:9 code: key = MiddleMan.instance.new_worker(:class => :example_worker) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070301/e16d927f/attachment.html From ezmobius at gmail.com Fri Mar 2 14:23:26 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 2 Mar 2007 11:23:26 -0800 Subject: [Backgroundrb-devel] example standalone error. In-Reply-To: References: Message-ID: <41DBC608-FB85-4527-8386-C29FA27A1305@brainspl.at> For standalone mode you have to do something like this: MiddleMan = BackgrounDRb::MiddleManDRbObject.init(:uri =>'drbunix:/// tmp/backgroundrbunix_localhost_2000') MiddleMan.new_worker -Ezra On Mar 1, 2007, at 11:02 PM, David Gurba wrote: > Hello, > > Trying to get the example to work in standalone mode. I have the > following in a file and am not certain why I get this undefined > method error: > > error: > ~/ruby sample_app.rb > my sample app :) > sample_app.rb:9: undefined method `new_worker' for > BackgrounDRb::MiddleMan:Class (NoMethodError) > > ### code follows ### > # sample_app ... try to get stand-alone happiness. > #require 'release-0.2.1/lib/backgroundrb' > #MiddleManDRbObject.init(:uri =>'drbunix:///tmp/ > backgroundrbunix_localhost_2000') > > require 'backgroundrb/middleman' > include BackgrounDRb > puts 'my sample app :)' > > key = MiddleMan.new_worker(:class => :example_worker) > worker = MiddleMan.worker(key) > worker.other_method > worker.delete > > # now that my job is done (and my worker is actually deleted on the > server) view > # my results ... cause now I'lm interested. > p worker.results.to_hash > puts 'bye bye :)' > > I kinda confused because most of the examples online and in / > examples assume that one is using Rails for this. Its kind of > difficult to see exactly where require or init calls need to go. > > regards, > David G. > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From pelargir at gmail.com Sat Mar 3 22:54:57 2007 From: pelargir at gmail.com (Matthew) Date: Sat, 3 Mar 2007 22:54:57 -0500 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster Message-ID: I've been troubleshooting this problem for several hours now and haven't been able to figure out what's wrong. I have two boxes, dev and prod, both are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4. Both are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. The behavior I'm seeing is this. When I first start Backgroundrb, this is what the process list looks like: cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:03.36/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: backgroundrb (ruby) cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: backgroundrb_logger (ruby) cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: backgroundrb_results (ruby) My four Mongrels in the cluster sitting out there, and the three backgroundrb processes. I then proceed to generate a worker using MiddleMan via IRB, get the auto generated job key, look up the worker, and then delete him. All goes well, and the process list still looks like what I posted above. I can do this as many times as I want to via IRB and nothing goes wrong. However, when I try to call the same code through a Rails action, Backgroundrb doesn't behave. The browser just sits there spinning. Examining the process list yields: cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: backgroundrb (ruby) cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: backgroundrb (ruby) cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0:03.36/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: backgroundrb_logger (ruby) cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: backgroundrb_results (ruby) cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep ruby Aha! An extra Backgroundrb process that's spiking the CPU. I've let the browser sit there until it times out, and the Backgroundrb process will still be there, spiking the CPU. If I stop Backgroundrb at this point, the extra process is still there. I have to kill it manually to get rid of it. Something else I've noticed is that, once this process is out there, even if I try creating new workers via IRB, the call to MiddleMan.new_worker just sits there. So this definitely appears to be a problem with the way Backgroundrb and my Mongrel clusters are interacting, but I'm not sure where to even begin. It's frustrating that this isn't happening on the dev box. I've tried to figure out what could be different between the boxes, but other than the Ruby versions, I'm not seeing anything. I'm out of options. Does anyone have any ideas? Thanks, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070303/3c154ef6/attachment.html From pardeeb at yahoo.com Sun Mar 4 08:52:56 2007 From: pardeeb at yahoo.com (Brad Pardee) Date: Sun, 4 Mar 2007 05:52:56 -0800 (PST) Subject: [Backgroundrb-devel] Did BackgrounDRb change my apps behavior? Message-ID: <810121.54840.qm@web50909.mail.yahoo.com> I just installed BackgrounDRb the other day. It's working great but after I deployed to production, I noticed a couple of strange things happening: 1) I have the following code in a lib module and after deploying, IS_PRODUCTION was set to false. I'm not sure why this started occurring and the only major change was the installation of BackgrounDRb and its dependencies. Could this have changed the initialization order or something? module Util IS_PRODUCTION = (ENV['RAILS_ENV'] == 'production') 2) I have a runner script which contains some queries like the following: print "Name: " ans = STDIN.gets.chomp Now STDOUT is no longer getting flushed when STDIN.gets is called, so "Name:" doesn't appear until after I enter some input. I realize this may have nothing to do with BackgrounDRb or Slave or Daemons, but thats the only thing I can think of that could have changed this behavior. Thanks for any input! Brad From masonhale at gmail.com Sun Mar 4 11:33:51 2007 From: masonhale at gmail.com (Mason Hale) Date: Sun, 4 Mar 2007 10:33:51 -0600 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: References: Message-ID: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> Matthew - Trying running netstat to see what ports your mongrels and backgroundrb processes are using. I've run into some problems with those two trying to use the same ports in past. Mason On 3/3/07, Matthew wrote: > > I've been troubleshooting this problem for several hours now and haven't > been able to figure out what's wrong. I have two boxes, dev and prod, both > are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4 . Both > are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. > > The behavior I'm seeing is this. When I first start Backgroundrb, this is > what the process list looks like: > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: > backgroundrb (ruby) > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > backgroundrb_results (ruby) > > My four Mongrels in the cluster sitting out there, and the three > backgroundrb processes. I then proceed to generate a worker using MiddleMan > via IRB, get the auto generated job key, look up the worker, and then delete > him. All goes well, and the process list still looks like what I posted > above. I can do this as many times as I want to via IRB and nothing goes > wrong. However, when I try to call the same code through a Rails action, > Backgroundrb doesn't behave. The browser just sits there spinning. Examining > the process list yields: > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: > backgroundrb (ruby) > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: > backgroundrb (ruby) > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > backgroundrb_results (ruby) > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep ruby > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let the > browser sit there until it times out, and the Backgroundrb process will > still be there, spiking the CPU. If I stop Backgroundrb at this point, the > extra process is still there. I have to kill it manually to get rid of it. > > Something else I've noticed is that, once this process is out there, even > if I try creating new workers via IRB, the call to MiddleMan.new_workerjust sits there. So this definitely appears to be a problem with the way > Backgroundrb and my Mongrel clusters are interacting, but I'm not sure where > to even begin. It's frustrating that this isn't happening on the dev box. > I've tried to figure out what could be different between the boxes, but > other than the Ruby versions, I'm not seeing anything. > > I'm out of options. Does anyone have any ideas? > > Thanks, > Matthew > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070304/cc6c1beb/attachment.html From info at siebert-wd.de Sun Mar 4 13:45:12 2007 From: info at siebert-wd.de (Michael Siebert) Date: Sun, 4 Mar 2007 19:45:12 +0100 Subject: [Backgroundrb-devel] BgDRb blocking in ActiveRecord Message-ID: <93690D97-3C9F-4A3E-8227-6717B67B8D19@siebert-wd.de> Hi Folks, I've got a tiny little problem using MySQL-backed ActiveRecord in my worker class. Sometimes it just blocks infinitely when I try to access it. I don't know whether it is in AR itself or in the SQL driver. It just stops doing anything, logging nothing, no exceptions are raised. I don't know if it matters that I call the worker's methods via backgroundrb_schedule.yml. Has anyone experienced a problem alike? Or can you give me a clue what could be wrong? PS: I'm not passing ANY AR objects via DRb ----------------------------- Siebert Michael info at siebert-wd.de --- ACHTUNG potenzieller Amokl?ufer: spiele Killerspiele (Scarface, GTA, UT) schaue Horrorfilme (Wrong Turn, Texas Chainsaw Massacre) h?re Musik von Anarchisten und Dunklen (Rammstein, Tote Hosen, Wizo) h?re Nazi-Musik von den B?hsen Onkelz, bin also auch noch Neonazi war mal auf ner Antifa-Demo war beim B.U.N.D. ich mag Ironie From fred at 82ask.com Sun Mar 4 18:53:35 2007 From: fred at 82ask.com (Frederick Cheung) Date: Sun, 4 Mar 2007 23:53:35 +0000 Subject: [Backgroundrb-devel] BgDRb blocking in ActiveRecord In-Reply-To: <93690D97-3C9F-4A3E-8227-6717B67B8D19@siebert-wd.de> References: <93690D97-3C9F-4A3E-8227-6717B67B8D19@siebert-wd.de> Message-ID: <1C2B3361-C01A-4E6B-9414-4CBAA34A26EE@82ask.com> The native msyql driver is blocking, and blocks the entire process (not just thread) in which it is running for the duration of the query. Would this explain what you're seeing ? Fred On 4 Mar 2007, at 18:45, Michael Siebert wrote: > Hi Folks, > > I've got a tiny little problem using MySQL-backed ActiveRecord in my > worker class. Sometimes it just blocks infinitely when I try to > access it. I don't know whether it is in AR itself or in the SQL > driver. It just stops doing anything, logging nothing, no exceptions > are raised. I don't know if it matters that I call the worker's > methods via backgroundrb_schedule.yml. > > Has anyone experienced a problem alike? Or can you give me a clue > what could be wrong? > > PS: I'm not passing ANY AR objects via DRb > ----------------------------- > Siebert Michael > info at siebert-wd.de > > --- > > ACHTUNG potenzieller Amokl?ufer: > spiele Killerspiele (Scarface, GTA, UT) > schaue Horrorfilme (Wrong Turn, Texas Chainsaw Massacre) > h?re Musik von Anarchisten und Dunklen (Rammstein, Tote Hosen, > Wizo) > h?re Nazi-Musik von den B?hsen Onkelz, bin also auch noch Neonazi > war mal auf ner Antifa-Demo > war beim B.U.N.D. > ich mag Ironie > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From jnutting at gmail.com Mon Mar 5 11:11:06 2007 From: jnutting at gmail.com (Jack Nutting) Date: Mon, 5 Mar 2007 17:11:06 +0100 Subject: [Backgroundrb-devel] nil object / slave server problem Message-ID: I'm just getting started wtih backgroundrb, and unfortunately hit a snag right away... I'm running the latest releases of rails, backgroundrb, slave and daemons. I created a simple worker class, using the generator, which currently does nothing but print the arguments that are passed to do_work. I'm running backgroundrb using './script/backgroundrb run' so I can watch the output. When I try to launch a worker, I get this in the output (in the shell where the backgroundrb script is running): You have a nil object when you didn't expect it! The error occurred while evaluating nil.wait - (NoMethodError) //vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:218:in `new_worker' Looking through middleman.rb, I found that this is triggered by the slave_obj being nil, and see that there's a preceding exception, which I found in backgroundrb_server.log: 20070305-16:45:13 (2929) Loading Worker Class File: //lib/workers/background_message_sender_worker.rb 20070305-16:45:38 (2929) failed to find slave socket - (RuntimeError) 20070305-16:45:38 (2929) /opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in `initialize' FWIW, here's what my backgroundrb.yml looks like: --- :rails_env: development :host: localhost :port: 2000 :acl: :deny: all :allow: localhost 127.0.0.1 :order: deny allow Any ideas? A misconfiguration on my part, perhaps? -- // jack // http://www.nuthole.com From info at siebert-wd.de Mon Mar 5 11:50:28 2007 From: info at siebert-wd.de (Michael Siebert) Date: Mon, 5 Mar 2007 17:50:28 +0100 Subject: [Backgroundrb-devel] BgDRb blocking in ActiveRecord Message-ID: <045FD8E6-5DC6-47BA-B85C-5B8D32067ADF@siebert-wd.de> ... damn i alwys forget to send these replies to the list instead only to the poster... sorry frederick.... Hi again, possibly I expressed myself badly... The problem is not that the DB blocks during a query, which is pretty common, but that it just doesn't return from the AR call. The worker's just frozen! At least it make me think it is. The last log entry by that specific worker is from Sunday 1am (GMT+1). BackgrounDRb just keeps running (there is another worker, not using AR thats working perfectly), but my mailing worker does not do anything. No CPU time, so I think it's somewhere in the middle of a deadlock, but I don't know much about processes and threads. Do I see it correct that BackgrounDRb scheduling does not call the method if it didn't return when it's supposed to run again? (I mean, start foo once an hour, but foo does take longer then an hour today, does bgdrb delay the next call?) i hope someone could help me a little bit with that... cheers Am 05.03.2007 um 00:53 schrieb Frederick Cheung: > The native msyql driver is blocking, and blocks the entire process > (not just thread) in which it is running for the duration of the > query. Would this explain what you're seeing ? > > Fred > > On 4 Mar 2007, at 18:45, Michael Siebert wrote: > >> Hi Folks, >> >> I've got a tiny little problem using MySQL-backed ActiveRecord in my >> worker class. Sometimes it just blocks infinitely when I try to >> access it. I don't know whether it is in AR itself or in the SQL >> driver. It just stops doing anything, logging nothing, no exceptions >> are raised. I don't know if it matters that I call the worker's >> methods via backgroundrb_schedule.yml. >> >> Has anyone experienced a problem alike? Or can you give me a clue >> what could be wrong? >> >> PS: I'm not passing ANY AR objects via DRb >> ----------------------------- >> Siebert Michael >> info at siebert-wd.de >> >> --- >> >> ACHTUNG potenzieller Amokl?ufer: >> spiele Killerspiele (Scarface, GTA, UT) >> schaue Horrorfilme (Wrong Turn, Texas Chainsaw Massacre) >> h?re Musik von Anarchisten und Dunklen (Rammstein, Tote Hosen, >> Wizo) >> h?re Nazi-Musik von den B?hsen Onkelz, bin also auch noch Neonazi >> war mal auf ner Antifa-Demo >> war beim B.U.N.D. >> ich mag Ironie >> >> >> >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > ----------------------------- Siebert Michael info at siebert-wd.de --- ACHTUNG potenzieller Amokl?ufer: spiele Killerspiele (Scarface, GTA, UT) schaue Horrorfilme (Wrong Turn, Texas Chainsaw Massacre) h?re Musik von Anarchisten und Dunklen (Rammstein, Tote Hosen, Wizo) h?re Nazi-Musik von den B?hsen Onkelz, bin also auch noch Neonazi war mal auf ner Antifa-Demo war beim B.U.N.D. ich mag Ironie From david.gurba at gmail.com Mon Mar 5 11:57:58 2007 From: david.gurba at gmail.com (David Gurba) Date: Mon, 05 Mar 2007 08:57:58 -0800 Subject: [Backgroundrb-devel] dynamic status back to client... Message-ID: <45EC4C16.20509@gmail.com> Hi, I have a long worker process I want to capture the output as dynamically as possible and send it to the client. I do the following currently: class SvnWorker < Worker::Base def do_stuff(args) tmp = [] my_project = args[:repo] # Import the contents of a 20 Meg project into a bare bones svn repo. import_status = `svn import #{path_to_extracted_zipdir} file://#{my_project}` tmp = import_status results[:response] = tmp end def status results[:response] end end This only fills results[:response] at the end of the system call ... I would rather dynamically get all the output to stdout the svn import command gives back. Like the following (but doesnt work for some reason): def do_stuff(args) require 'open3' tmp = [] my_project = args[:repo] # Import the contents of a 20 Meg project into a bare bones svn repo. Open3.popen3("svn import #{path_to_extracted_zipdir} file://#{my_project}") do |stdin,stdout,stderr| while txt = stdout.gets tmp << txt results[:response] = tmp end end end But for a reason that eludes me I can't seem to call open ever and if I require anything in my worker class it crashes when I call that workers do_work function. Any thoughts appreciated... regards, David G. From ezmobius at gmail.com Mon Mar 5 14:39:19 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 11:39:19 -0800 Subject: [Backgroundrb-devel] nil object / slave server problem In-Reply-To: References: Message-ID: Jake- This is a problem that probably needs to become a FAQ. WHat's happeneing is the name of your worker is too long and when slave tries to create the unix socket the filename is too long and it fails to fin slave socket. Rename your worker to a smaller name and it will work fine. Or change to use tcp connectiosn instead of unix sockets. Cheers- -Ezra On Mar 5, 2007, at 8:11 AM, Jack Nutting wrote: > I'm just getting started wtih backgroundrb, and unfortunately hit a > snag right away... I'm running the latest releases of rails, > backgroundrb, slave and daemons. > > I created a simple worker class, using the generator, which currently > does nothing but print the arguments that are passed to do_work. I'm > running backgroundrb using './script/backgroundrb run' so I can watch > the output. When I try to launch a worker, I get this in the output > (in the shell where the backgroundrb script is running): > > You have a nil object when you didn't expect it! > The error occurred while evaluating nil.wait - (NoMethodError) > //vendor/plugins/backgroundrb/server/lib/backgroundrb/ > middleman.rb:218:in > `new_worker' > > Looking through middleman.rb, I found that this is triggered by the > slave_obj being nil, and see that there's a preceding exception, which > I found in backgroundrb_server.log: > > 20070305-16:45:13 (2929) Loading Worker Class File: > //lib/workers/background_message_sender_worker.rb > 20070305-16:45:38 (2929) failed to find slave socket - (RuntimeError) > 20070305-16:45:38 (2929) > /opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in > `initialize' > > FWIW, here's what my backgroundrb.yml looks like: > > --- > :rails_env: development > :host: localhost > :port: 2000 > :acl: > :deny: all > :allow: localhost 127.0.0.1 > :order: deny allow > > > Any ideas? A misconfiguration on my part, perhaps? > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From pelargir at gmail.com Mon Mar 5 15:40:27 2007 From: pelargir at gmail.com (Matthew) Date: Mon, 5 Mar 2007 15:40:27 -0500 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> References: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> Message-ID: Thanks for the suggestion, Mason. I did that and noticed that each time I attempted to create a new worker, I got a new entry. This is what I ended up with in netstat after trying several attempts: -bash-2.05b$ netstat -an | grep 2001 c4cb5af0 stream 0 0 0 c8cb5af0 0 0 ///tmp/backgroundrbunix_localhost_2001 c8d58a64 stream 0 0 0 0 0 0 ///tmp/backgroundrbunix_localhost_2001 c4dcc94c stream 0 0 0 c97ac94c 0 0 ///tmp/backgroundrbunix_localhost_2001 c8d20690 stream 0 0 0 c8cb5000 0 0 ///tmp/backgroundrbunix_localhost_2001 c8cca348 stream 0 0 0 c4dcc690 0 0 ///tmp/backgroundrbunix_localhost_2001 c8cf7b7c stream 0 0 0 c8d209d8 0 0 ///tmp/backgroundrbunix_localhost_2001 c4cb62bc stream 0 0 c6b63d68 0 0 0 ///tmp/backgroundrbunix_localhost_2001 My process list looked like this: -bash-2.05b$ ps aux | grep ruby cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2:22.28 ruby: pdf_generator_worker_dde6f63034382c4727 cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 ruby: pdf_generator_worker_031025c9be743ca12e cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0:43.74 ruby: pdf_generator_worker_c617aa5cc57c7b59b7 cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11:28.06 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6:39.64 ruby /usr/local/lib/ruby/gems/1.8/gems/mongre cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 ruby: backgroundrb (ruby) cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 ruby: backgroundrb_logger (ruby) cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 ruby: backgroundrb_results (ruby) cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 grep ruby Looks like a new pdf_generator_worker process gets created and immediately freezes each time I try to create a worker. I can't communicate with any of these workers and have to manually kill them to get them to go away. Anyone else have an idea of what could be going on? Matthew On 3/4/07, Mason Hale wrote: > > Matthew - > > Trying running netstat to see what ports your mongrels and backgroundrb > processes are using. I've run into some problems with those two trying to > use the same ports in past. > > Mason > > On 3/3/07, Matthew wrote: > > > I've been troubleshooting this problem for several hours now and haven't > > been able to figure out what's wrong. I have two boxes, dev and prod, both > > are identical except that prod uses Ruby 1.8.5 and dev uses 1.8.4 . Both > > are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb 0.2.1. > > > > The behavior I'm seeing is this. When I first start Backgroundrb, this > > is what the process list looks like: > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: > > backgroundrb (ruby) > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > backgroundrb_results (ruby) > > > > My four Mongrels in the cluster sitting out there, and the three > > backgroundrb processes. I then proceed to generate a worker using MiddleMan > > via IRB, get the auto generated job key, look up the worker, and then delete > > him. All goes well, and the process list still looks like what I posted > > above. I can do this as many times as I want to via IRB and nothing goes > > wrong. However, when I try to call the same code through a Rails action, > > Backgroundrb doesn't behave. The browser just sits there spinning. Examining > > the process list yields: > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: > > backgroundrb (ruby) > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: > > backgroundrb (ruby) > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > backgroundrb_results (ruby) > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep > > ruby > > > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let the > > browser sit there until it times out, and the Backgroundrb process will > > still be there, spiking the CPU. If I stop Backgroundrb at this point, the > > extra process is still there. I have to kill it manually to get rid of it. > > > > Something else I've noticed is that, once this process is out there, > > even if I try creating new workers via IRB, the call to > > MiddleMan.new_worker just sits there. So this definitely appears to be a > > problem with the way Backgroundrb and my Mongrel clusters are interacting, > > but I'm not sure where to even begin. It's frustrating that this isn't > > happening on the dev box. I've tried to figure out what could be different > > between the boxes, but other than the Ruby versions, I'm not seeing > > anything. > > > > I'm out of options. Does anyone have any ideas? > > > > Thanks, > > Matthew > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/b18ef904/attachment-0001.html From pelargir at gmail.com Mon Mar 5 15:49:25 2007 From: pelargir at gmail.com (Matthew) Date: Mon, 5 Mar 2007 15:49:25 -0500 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: References: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> Message-ID: Something I'm also noticing that may be related is this message. I get it every so often when I try to stop backgroundrb: -bash-2.05b$ script/backgroundrb stop /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in `initialize': No such file or directory - /usr/home/cwarner/rails/log/backgroundrb.pid (Errno::ENOENT) from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in `open' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/pidfile.rb:92:in `pid' from /usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:350:in `stop' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:133:in `stop_all' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in `each' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/application_group.rb:129:in `stop_all' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/controller.rb:74:in `run' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:182:in `run_proc' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in `call' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons/cmdline.rb:105:in `catch_exceptions' from /usr/local/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc' from /usr/home/cwarner/www/html/rails/test/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run' from script/backgroundrb:29 I'm also seeing a file called "backgroundrb.ppid" (yes, with two Ps) in my log dir. "backgroundrb.pid" shows up there when backgroundrb is running, but it gets deleted when I stop. On the other hand, "backgroundrb.ppid" stays there unless I delete it manually. Matthew On 3/5/07, Matthew wrote: > > Thanks for the suggestion, Mason. I did that and noticed that each time I > attempted to create a new worker, I got a new entry. This is what I ended up > with in netstat after trying several attempts: > > -bash-2.05b$ netstat -an | grep 2001 > c4cb5af0 stream 0 0 0 c8cb5af0 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c8d58a64 stream 0 0 0 0 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c4dcc94c stream 0 0 0 c97ac94c 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c8d20690 stream 0 0 0 c8cb5000 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c8cca348 stream 0 0 0 c4dcc690 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c8cf7b7c stream 0 0 0 c8d209d8 0 0 > ///tmp/backgroundrbunix_localhost_2001 > c4cb62bc stream 0 0 c6b63d68 0 0 0 > ///tmp/backgroundrbunix_localhost_2001 > > My process list looked like this: > > -bash-2.05b$ ps aux | grep ruby > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 ruby: > pdf_generator_worker_dde6f63034382c4727 > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 ruby: > pdf_generator_worker_031025c9be743ca12e > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 ruby: > pdf_generator_worker_c617aa5cc57c7b59b7 > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 ruby > /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 ruby > /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 ruby > /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 ruby > /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 ruby: > backgroundrb (ruby) > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 ruby: > backgroundrb_logger (ruby) > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 ruby: > backgroundrb_results (ruby) > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 grep ruby > > Looks like a new pdf_generator_worker process gets created and immediately > freezes each time I try to create a worker. I can't communicate with any of > these workers and have to manually kill them to get them to go away. > > Anyone else have an idea of what could be going on? > > Matthew > > > On 3/4/07, Mason Hale wrote: > > > > Matthew - > > > > Trying running netstat to see what ports your mongrels and backgroundrb > > processes are using. I've run into some problems with those two trying to > > use the same ports in past. > > > > Mason > > > > On 3/3/07, Matthew wrote: > > > > > I've been troubleshooting this problem for several hours now and > > > haven't been able to figure out what's wrong. I have two boxes, dev and > > > prod, both are identical except that prod uses Ruby 1.8.5 and dev uses > > > 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and Backroundrb > > > 0.2.1. > > > > > > The behavior I'm seeing is this. When I first start Backgroundrb, this > > > is what the process list looks like: > > > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48/usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 ruby: > > > backgroundrb (ruby) > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > > backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > > backgroundrb_results (ruby) > > > > > > My four Mongrels in the cluster sitting out there, and the three > > > backgroundrb processes. I then proceed to generate a worker using MiddleMan > > > via IRB, get the auto generated job key, look up the worker, and then delete > > > him. All goes well, and the process list still looks like what I posted > > > above. I can do this as many times as I want to via IRB and nothing goes > > > wrong. However, when I try to call the same code through a Rails action, > > > Backgroundrb doesn't behave. The browser just sits there spinning. Examining > > > the process list yields: > > > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 ruby: > > > backgroundrb (ruby) > > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 ruby: > > > backgroundrb (ruby) > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36/usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59/usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51/usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26/usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 ruby: > > > backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 ruby: > > > backgroundrb_results (ruby) > > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 grep > > > ruby > > > > > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let > > > the browser sit there until it times out, and the Backgroundrb process will > > > still be there, spiking the CPU. If I stop Backgroundrb at this point, the > > > extra process is still there. I have to kill it manually to get rid of it. > > > > > > Something else I've noticed is that, once this process is out there, > > > even if I try creating new workers via IRB, the call to > > > MiddleMan.new_worker just sits there. So this definitely appears to be > > > a problem with the way Backgroundrb and my Mongrel clusters are interacting, > > > but I'm not sure where to even begin. It's frustrating that this isn't > > > happening on the dev box. I've tried to figure out what could be different > > > between the boxes, but other than the Ruby versions, I'm not seeing > > > anything. > > > > > > I'm out of options. Does anyone have any ideas? > > > > > > Thanks, > > > Matthew > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/e1134c2d/attachment.html From ezmobius at gmail.com Mon Mar 5 16:08:14 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 13:08:14 -0800 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: References: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> Message-ID: <145C196C-1573-4191-9109-207DAC87A42F@gmail.com> Mathew- This may be the same worker class name length overflow error. If your classname for your worker is too long then slave can fail to create the proper unix domain socket. Can you try to shorten the name of your worker class and try again? -Ezra On Mar 5, 2007, at 12:40 PM, Matthew wrote: > Thanks for the suggestion, Mason. I did that and noticed that each > time I attempted to create a new worker, I got a new entry. This is > what I ended up with in netstat after trying several attempts: > > -bash-2.05b$ netstat -an | grep 2001 > c4cb5af0 stream 0 0 0 c8cb5af0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8d58a64 stream 0 0 0 0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c4dcc94c stream 0 0 0 c97ac94c 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8d20690 stream 0 0 0 c8cb5000 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8cca348 stream 0 0 0 c4dcc690 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c8cf7b7c stream 0 0 0 c8d209d8 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > c4cb62bc stream 0 0 c6b63d68 0 0 > 0 ///tmp/backgroundrbunix_localhost_2001 > > My process list looked like this: > > -bash-2.05b$ ps aux | grep ruby > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > ruby: pdf_generator_worker_dde6f63034382c4727 > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > ruby: pdf_generator_worker_031025c9be743ca12e > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > ruby: backgroundrb (ruby) > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > ruby: backgroundrb_logger (ruby) > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > ruby: backgroundrb_results (ruby) > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > grep ruby > > Looks like a new pdf_generator_worker process gets created and > immediately freezes each time I try to create a worker. I can't > communicate with any of these workers and have to manually kill > them to get them to go away. > > Anyone else have an idea of what could be going on? > > Matthew > > > On 3/4/07, Mason Hale wrote:Matthew - > > Trying running netstat to see what ports your mongrels and > backgroundrb processes are using. I've run into some problems with > those two trying to use the same ports in past. > > Mason > > On 3/3/07, Matthew wrote: > I've been troubleshooting this problem for several hours now and > haven't been able to figure out what's wrong. I have two boxes, dev > and prod, both are identical except that prod uses Ruby 1.8.5 and > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > Backroundrb 0.2.1. > > The behavior I'm seeing is this. When I first start Backgroundrb, > this is what the process list looks like: > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > usr/local/bin/ruby18 /usr/local/bin/mon > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > ruby: backgroundrb (ruby) > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > ruby: backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > ruby: backgroundrb_results (ruby) > > My four Mongrels in the cluster sitting out there, and the three > backgroundrb processes. I then proceed to generate a worker using > MiddleMan via IRB, get the auto generated job key, look up the > worker, and then delete him. All goes well, and the process list > still looks like what I posted above. I can do this as many times > as I want to via IRB and nothing goes wrong. However, when I try to > call the same code through a Rails action, Backgroundrb doesn't > behave. The browser just sits there spinning. Examining the process > list yields: > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > ruby: backgroundrb (ruby) > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > ruby: backgroundrb (ruby) > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > usr/local/bin/ruby18 /usr/local/bin/mong > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > ruby: backgroundrb_logger (ruby) > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > ruby: backgroundrb_results (ruby) > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > grep ruby > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let > the browser sit there until it times out, and the Backgroundrb > process will still be there, spiking the CPU. If I stop > Backgroundrb at this point, the extra process is still there. I > have to kill it manually to get rid of it. > > Something else I've noticed is that, once this process is out > there, even if I try creating new workers via IRB, the call to > MiddleMan.new_worker just sits there. So this definitely appears to > be a problem with the way Backgroundrb and my Mongrel clusters are > interacting, but I'm not sure where to even begin. It's frustrating > that this isn't happening on the dev box. I've tried to figure out > what could be different between the boxes, but other than the Ruby > versions, I'm not seeing anything. > > I'm out of options. Does anyone have any ideas? > > Thanks, > Matthew > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From michael.dauria at gmail.com Mon Mar 5 16:13:52 2007 From: michael.dauria at gmail.com (Michael D'Auria) Date: Mon, 5 Mar 2007 16:13:52 -0500 Subject: [Backgroundrb-devel] nil object / slave server problem In-Reply-To: References: Message-ID: <1907e2ca0703051313t42920d29r5199d5d4cdd42e80@mail.gmail.com> Is there any way to have Bdrb shrink the filename down automatically? This just seems silly to have to keep an eye out for. On 3/5/07, Ezra Zygmuntowicz wrote: > > Jake- > > This is a problem that probably needs to become a FAQ. WHat's > happeneing is the name of your worker is too long and when slave > tries to create the unix socket the filename is too long and it fails > to fin slave socket. > > Rename your worker to a smaller name and it will work fine. Or > change to use tcp connectiosn instead of unix sockets. > > Cheers- > -Ezra > > On Mar 5, 2007, at 8:11 AM, Jack Nutting wrote: > > > I'm just getting started wtih backgroundrb, and unfortunately hit a > > snag right away... I'm running the latest releases of rails, > > backgroundrb, slave and daemons. > > > > I created a simple worker class, using the generator, which currently > > does nothing but print the arguments that are passed to do_work. I'm > > running backgroundrb using './script/backgroundrb run' so I can watch > > the output. When I try to launch a worker, I get this in the output > > (in the shell where the backgroundrb script is running): > > > > You have a nil object when you didn't expect it! > > The error occurred while evaluating nil.wait - (NoMethodError) > > //vendor/plugins/backgroundrb/server/lib/backgroundrb/ > > middleman.rb:218:in > > `new_worker' > > > > Looking through middleman.rb, I found that this is triggered by the > > slave_obj being nil, and see that there's a preceding exception, which > > I found in backgroundrb_server.log: > > > > 20070305-16:45:13 (2929) Loading Worker Class File: > > //lib/workers/background_message_sender_worker.rb > > 20070305-16:45:38 (2929) failed to find slave socket - (RuntimeError) > > 20070305-16:45:38 (2929) > > /opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in > > `initialize' > > > > FWIW, here's what my backgroundrb.yml looks like: > > > > --- > > :rails_env: development > > :host: localhost > > :port: 2000 > > :acl: > > :deny: all > > :allow: localhost 127.0.0.1 > > :order: deny allow > > > > > > Any ideas? A misconfiguration on my part, perhaps? > > > > -- > > // jack > > // http://www.nuthole.com > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/51c2accb/attachment.html From ezmobius at gmail.com Mon Mar 5 16:20:18 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 13:20:18 -0800 Subject: [Backgroundrb-devel] nil object / slave server problem In-Reply-To: <1907e2ca0703051313t42920d29r5199d5d4cdd42e80@mail.gmail.com> References: <1907e2ca0703051313t42920d29r5199d5d4cdd42e80@mail.gmail.com> Message-ID: <48001DA2-056E-4290-BEF8-BE5A06451CD4@gmail.com> I'm sure there is a way, patches welcome ;) The other way to avoid it is to use the druby protocol instead of unix domain sockets. Then the problem doesn't exist. You have to set protocol: druby to use tcp drb connections as unix domain sockets are the default. I haven't had any time to work on backgroundrb in 3 months or so now so there are some things that need to be fixed. Hopefully I will be picking bdrb up to develop again in another 2-3 weeks or so. Then I will fix the remaining issues. Thanks -Ezra On Mar 5, 2007, at 1:13 PM, Michael D'Auria wrote: > Is there any way to have Bdrb shrink the filename down > automatically? This just seems silly to have to keep an eye out for. > > On 3/5/07, Ezra Zygmuntowicz wrote: Jake- > > This is a problem that probably needs to become a FAQ. WHat's > happeneing is the name of your worker is too long and when slave > tries to create the unix socket the filename is too long and it fails > to fin slave socket. > > Rename your worker to a smaller name and it will work fine. Or > change to use tcp connectiosn instead of unix sockets. > > Cheers- > -Ezra > > On Mar 5, 2007, at 8:11 AM, Jack Nutting wrote: > > > I'm just getting started wtih backgroundrb, and unfortunately hit a > > snag right away... I'm running the latest releases of rails, > > backgroundrb, slave and daemons. > > > > I created a simple worker class, using the generator, which > currently > > does nothing but print the arguments that are passed to do_work. > I'm > > running backgroundrb using './script/backgroundrb run' so I can > watch > > the output. When I try to launch a worker, I get this in the output > > (in the shell where the backgroundrb script is running): > > > > You have a nil object when you didn't expect it! > > The error occurred while evaluating nil.wait - (NoMethodError) > > //vendor/plugins/backgroundrb/server/lib/backgroundrb/ > > middleman.rb:218:in > > `new_worker' > > > > Looking through middleman.rb, I found that this is triggered by the > > slave_obj being nil, and see that there's a preceding exception, > which > > I found in backgroundrb_server.log: > > > > 20070305-16:45:13 (2929) Loading Worker Class File: > > //lib/workers/background_message_sender_worker.rb > > 20070305-16:45:38 (2929) failed to find slave socket - > (RuntimeError) > > 20070305-16:45:38 (2929) > > /opt/local/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in > > `initialize' > > > > FWIW, here's what my backgroundrb.yml looks like: > > > > --- > > :rails_env: development > > :host: localhost > > :port: 2000 > > :acl: > > :deny: all > > :allow: localhost 127.0.0.1 > > :order: deny allow > > > > > > Any ideas? A misconfiguration on my part, perhaps? > > > > -- > > // jack > > // http://www.nuthole.com > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From pelargir at gmail.com Mon Mar 5 16:25:01 2007 From: pelargir at gmail.com (Matthew) Date: Mon, 5 Mar 2007 16:25:01 -0500 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: <145C196C-1573-4191-9109-207DAC87A42F@gmail.com> References: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> <145C196C-1573-4191-9109-207DAC87A42F@gmail.com> Message-ID: Thanks for the reply, Ezra. Just tried that and I'm still experiencing the same problem. As before, running from either script/console or the browser works fine on the first go round. I can generate a worker, query it, and delete it. Trying to generate a worker the second time is a different story, though. In the browser the generate appears to work, but when I try to get a worker for the returned key I get a nil object. In script/console, the generate just sits there spinning until I kill the backgroundrb process. Something interesting I noticed as well is the dir that gets created in /tmp with the PID on the end does not match the actual PID of my backgroundrb process. Is it supposed to? -bash-2.05b$ ls /tmp/ba* /tmp/backgroundrbunix_localhost_2020 /tmp/backgroundrb.32589 -bash-2.05b$ ps aux | grep backgroundrb cwarner 32591 0.0 0.7 21528 20564 ?? S 4:13PM 0:00.17 ruby: backgroundrb (ruby) cwarner 32592 0.0 0.7 21512 20552 ?? S 4:13PM 0:00.06 ruby: backgroundrb_logger (ruby) cwarner 32593 0.0 0.7 21468 20560 ?? S 4:13PM 0:00.05 ruby: backgroundrb_results (ruby) On an aside, thanks for building a really great tool here. It's simple to use and perfect for what I'm trying to do. I'm sure the issue is on my end with configuration or something. Matthew On 3/5/07, Ezra Zygmuntowicz wrote: > > Mathew- > > This may be the same worker class name length overflow error. If > your classname for your worker is too long then slave can fail to > create the proper unix domain socket. Can you try to shorten the name > of your worker class and try again? > > -Ezra > > > On Mar 5, 2007, at 12:40 PM, Matthew wrote: > > > Thanks for the suggestion, Mason. I did that and noticed that each > > time I attempted to create a new worker, I got a new entry. This is > > what I ended up with in netstat after trying several attempts: > > > > -bash-2.05b$ netstat -an | grep 2001 > > c4cb5af0 stream 0 0 0 c8cb5af0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8d58a64 stream 0 0 0 0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c4dcc94c stream 0 0 0 c97ac94c 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8d20690 stream 0 0 0 c8cb5000 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8cca348 stream 0 0 0 c4dcc690 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c8cf7b7c stream 0 0 0 c8d209d8 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > c4cb62bc stream 0 0 c6b63d68 0 0 > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > > My process list looked like this: > > > > -bash-2.05b$ ps aux | grep ruby > > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > > ruby: pdf_generator_worker_dde6f63034382c4727 > > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > > ruby: pdf_generator_worker_031025c9be743ca12e > > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > > ruby: backgroundrb (ruby) > > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > > ruby: backgroundrb_logger (ruby) > > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > > grep ruby > > > > Looks like a new pdf_generator_worker process gets created and > > immediately freezes each time I try to create a worker. I can't > > communicate with any of these workers and have to manually kill > > them to get them to go away. > > > > Anyone else have an idea of what could be going on? > > > > Matthew > > > > > > On 3/4/07, Mason Hale wrote:Matthew - > > > > Trying running netstat to see what ports your mongrels and > > backgroundrb processes are using. I've run into some problems with > > those two trying to use the same ports in past. > > > > Mason > > > > On 3/3/07, Matthew wrote: > > I've been troubleshooting this problem for several hours now and > > haven't been able to figure out what's wrong. I have two boxes, dev > > and prod, both are identical except that prod uses Ruby 1.8.5 and > > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > > Backroundrb 0.2.1. > > > > The behavior I'm seeing is this. When I first start Backgroundrb, > > this is what the process list looks like: > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > usr/local/bin/ruby18 /usr/local/bin/mon > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > > ruby: backgroundrb (ruby) > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > ruby: backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > > > My four Mongrels in the cluster sitting out there, and the three > > backgroundrb processes. I then proceed to generate a worker using > > MiddleMan via IRB, get the auto generated job key, look up the > > worker, and then delete him. All goes well, and the process list > > still looks like what I posted above. I can do this as many times > > as I want to via IRB and nothing goes wrong. However, when I try to > > call the same code through a Rails action, Backgroundrb doesn't > > behave. The browser just sits there spinning. Examining the process > > list yields: > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > > ruby: backgroundrb (ruby) > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > > ruby: backgroundrb (ruby) > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > usr/local/bin/ruby18 /usr/local/bin/mong > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > ruby: backgroundrb_logger (ruby) > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > ruby: backgroundrb_results (ruby) > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > > grep ruby > > > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let > > the browser sit there until it times out, and the Backgroundrb > > process will still be there, spiking the CPU. If I stop > > Backgroundrb at this point, the extra process is still there. I > > have to kill it manually to get rid of it. > > > > Something else I've noticed is that, once this process is out > > there, even if I try creating new workers via IRB, the call to > > MiddleMan.new_worker just sits there. So this definitely appears to > > be a problem with the way Backgroundrb and my Mongrel clusters are > > interacting, but I'm not sure where to even begin. It's frustrating > > that this isn't happening on the dev box. I've tried to figure out > > what could be different between the boxes, but other than the Ruby > > versions, I'm not seeing anything. > > > > I'm out of options. Does anyone have any ideas? > > > > Thanks, > > Matthew > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/4fafd351/attachment-0001.html From jnutting at gmail.com Mon Mar 5 17:13:56 2007 From: jnutting at gmail.com (Jack Nutting) Date: Mon, 5 Mar 2007 23:13:56 +0100 Subject: [Backgroundrb-devel] postgres barfage revisted Message-ID: First of all, I'd like to do what I've seen several others here do: thank Ezra and any other contributors for producing a really great plugin. Even though I'm having some minor problems, this is all so approachable and easy to work with and dig into... I hate to think of the hoops you'd need to jump through (both as creator of such a system and as a user) to do something similar in Java. Now then... I've stumbled across the problem described here: http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/000299.html wherein the native Ruby adaptor for Postgres causes Backgroundrb to crap out with messages like this: ActiveRecord::StatementInvalid: NoMethodError: undefined method `async_query' for #: The author's suggested solutions were to either use the compiled ruby-postgres, or set ActiveRecord::Base.allow_concurrency to false in a start script. Well, the first option seems tricky to me; I haven't succeeded in getting that compiled for Mac OS X yet (granted, I haven't given it too much effort yet), so I figured I'd try the second option. I couldn't find any start script where I could actually make that setting have any effect, and eventually I realized that it's being set to true at worker_rails.rb:21. If I comment out that line, suddenly I'm back in business! What I wonder is if I'm setting myself up for future catastrophe with that change. I figure it's got to be in there for *some* reason. I think I read somewhere that Backgroundrb changed from a multi-thread to a multi-process architecture not too long ago; Is that line, setting allow_concurrency to true, a holdover from the multi-threaded days that's no longer needed? -- // jack // http://www.nuthole.com From david.balatero at gmail.com Mon Mar 5 17:24:37 2007 From: david.balatero at gmail.com (David Balatero) Date: Mon, 5 Mar 2007 14:24:37 -0800 Subject: [Backgroundrb-devel] Scheduled workers only run once unless you call self.delete inside the worker Message-ID: <572c318d0703051424q44d250c9mee4883cbb247182b@mail.gmail.com> I had a worker scheduled to run every minute with backgroundrb_schedules.yml: ebay_runner: :class: :ebay_auction_worker :job_key: :ebay_auction_runner :trigger_type: :cron_trigger :trigger_args: 0 * * * * * * This worker posts an auction to eBay from a queue of auctions every minute. I was having a problem where the worker would run ok the first time, but never any subsequent minutes after that. The way I fixed it was adding a call to self.delete at the end of every do_work() method: ---- class EbayAuctionWorker < BackgrounDRb::Rails # Set this worker to run every minute. attr_accessor :progress, :description def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args @progress = 0 @description = "Checking for eBay auctions and posting" logger.info("Checking to post an auction at #{Time.now.to_s}.") auction = EbayAuction.find(:first, :conditions => ["auction_status = ? AND post_on < ?", EbayAuction::STATUS_STRINGS[:queued], Time.now]) if auction logger.info("--- Posting auction: #{auction.title}") auction.post else logger.info("--- No auctions currently need posting.") end @progress = 100 logger.info("--- Finished auction check.") # Exit the thread, see what happens self.delete end end EbayAuctionWorker.register -------- Does this make sense? I never saw any mention of having to do this with scheduled workers in the documentation. Shouldn't the scheduler automatically clean up the worker with a self.delete, after @progress is >= 100? Thanks, David Balatero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070305/f8254fca/attachment.html From ezmobius at gmail.com Mon Mar 5 17:56:11 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 14:56:11 -0800 Subject: [Backgroundrb-devel] Scheduled workers only run once unless you call self.delete inside the worker In-Reply-To: <572c318d0703051424q44d250c9mee4883cbb247182b@mail.gmail.com> References: <572c318d0703051424q44d250c9mee4883cbb247182b@mail.gmail.com> Message-ID: <998B296F-9C9D-43A2-9551-78A139CF6F58@brainspl.at> You are right you need to self.delete at the end of a worker to destropy it. Bdrb doesn't auto desatroy any workers currently. -Ezra On Mar 5, 2007, at 2:24 PM, David Balatero wrote: > I had a worker scheduled to run every minute with > backgroundrb_schedules.yml: > > ebay_runner: > :class: :ebay_auction_worker > :job_key: :ebay_auction_runner > :trigger_type: :cron_trigger > :trigger_args: 0 * * * * * * > > This worker posts an auction to eBay from a queue of auctions every > minute. I was having a problem where the worker would run ok the > first time, but never any subsequent minutes after that. The way I > fixed it was adding a call to self.delete at the end of every > do_work() method: > > ---- > class EbayAuctionWorker < BackgrounDRb::Rails > # Set this worker to run every minute. > attr_accessor :progress, :description > > def do_work(args) > # This method is called in it's own new thread when you > # call new worker. args is set to :args > > @progress = 0 > @description = "Checking for eBay auctions and posting" > > logger.info("Checking to post an auction at #{Time.now.to_s}.") > auction = EbayAuction.find(:first, :conditions => > ["auction_status = ? AND post_on < ?", EbayAuction::STATUS_STRINGS > [:queued], Time.now]) > if auction > logger.info("--- Posting auction: #{auction.title}") > > auction.post > else > logger.info("--- No auctions currently need posting.") > end > > @progress = 100 > > logger.info("--- Finished auction check.") > > # Exit the thread, see what happens > self.delete > end > end > > EbayAuctionWorker.register > -------- > > > Does this make sense? I never saw any mention of having to do this > with scheduled workers in the documentation. Shouldn't the > scheduler automatically clean up the worker with a self.delete, > after @progress is >= 100? > > Thanks, > David Balatero > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Mar 5 17:57:16 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 14:57:16 -0800 Subject: [Backgroundrb-devel] postgres barfage revisted In-Reply-To: References: Message-ID: Hi- Yeah with the new multi process bdrb you probably need to set allow_concurrency to false. I don't think you will have a problem with it like that. If you do run into a problem then you may have to put a mutex around the calls to your database: @mymutex = Mutex.new @mymutex.synchronize { # calls to database in here. } -Ezra On Mar 5, 2007, at 2:13 PM, Jack Nutting wrote: > First of all, I'd like to do what I've seen several others here do: > thank Ezra and any other contributors for producing a really great > plugin. Even though I'm having some minor problems, this is all so > approachable and easy to work with and dig into... I hate to think of > the hoops you'd need to jump through (both as creator of such a system > and as a user) to do something similar in Java. > > Now then... I've stumbled across the problem described here: > > http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/ > 000299.html > > wherein the native Ruby adaptor for Postgres causes Backgroundrb to > crap out with messages like this: > > ActiveRecord::StatementInvalid: NoMethodError: undefined method > `async_query' for #: > > The author's suggested solutions were to either use the compiled > ruby-postgres, or set ActiveRecord::Base.allow_concurrency to false in > a start script. > > Well, the first option seems tricky to me; I haven't succeeded in > getting that compiled for Mac OS X yet (granted, I haven't given it > too much effort yet), so I figured I'd try the second option. I > couldn't find any start script where I could actually make that > setting have any effect, and eventually I realized that it's being set > to true at worker_rails.rb:21. If I comment out that line, suddenly > I'm back in business! > > What I wonder is if I'm setting myself up for future catastrophe with > that change. I figure it's got to be in there for *some* reason. I > think I read somewhere that Backgroundrb changed from a multi-thread > to a multi-process architecture not too long ago; Is that line, > setting allow_concurrency to true, a holdover from the multi-threaded > days that's no longer needed? > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Mar 5 18:04:15 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 5 Mar 2007 15:04:15 -0800 Subject: [Backgroundrb-devel] postgres barfage revisted In-Reply-To: References: Message-ID: Note that this is only for the postgres bindings afaik. Mysql is fine with allow concurrency = true -Ezra On Mar 5, 2007, at 2:57 PM, Ezra Zygmuntowicz wrote: > Hi- > > Yeah with the new multi process bdrb you probably need to set > allow_concurrency to false. I don't think you will have a problem > with it like that. If you do run into a problem then you may have to > put a mutex around the calls to your database: > > @mymutex = Mutex.new > > @mymutex.synchronize { > # calls to database in here. > } > > > -Ezra > > On Mar 5, 2007, at 2:13 PM, Jack Nutting wrote: > >> First of all, I'd like to do what I've seen several others here do: >> thank Ezra and any other contributors for producing a really great >> plugin. Even though I'm having some minor problems, this is all so >> approachable and easy to work with and dig into... I hate to >> think of >> the hoops you'd need to jump through (both as creator of such a >> system >> and as a user) to do something similar in Java. >> >> Now then... I've stumbled across the problem described here: >> >> http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/ >> 000299.html >> >> wherein the native Ruby adaptor for Postgres causes Backgroundrb to >> crap out with messages like this: >> >> ActiveRecord::StatementInvalid: NoMethodError: undefined method >> `async_query' for #: >> >> The author's suggested solutions were to either use the compiled >> ruby-postgres, or set ActiveRecord::Base.allow_concurrency to >> false in >> a start script. >> >> Well, the first option seems tricky to me; I haven't succeeded in >> getting that compiled for Mac OS X yet (granted, I haven't given it >> too much effort yet), so I figured I'd try the second option. I >> couldn't find any start script where I could actually make that >> setting have any effect, and eventually I realized that it's being >> set >> to true at worker_rails.rb:21. If I comment out that line, suddenly >> I'm back in business! >> >> What I wonder is if I'm setting myself up for future catastrophe with >> that change. I figure it's got to be in there for *some* reason. I >> think I read somewhere that Backgroundrb changed from a multi-thread >> to a multi-process architecture not too long ago; Is that line, >> setting allow_concurrency to true, a holdover from the multi-threaded >> days that's no longer needed? >> >> -- >> // jack >> // http://www.nuthole.com >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From skaar at waste.org Tue Mar 6 09:54:05 2007 From: skaar at waste.org (skaar) Date: Tue, 6 Mar 2007 08:54:05 -0600 Subject: [Backgroundrb-devel] nil object / slave server problem In-Reply-To: <48001DA2-056E-4290-BEF8-BE5A06451CD4@gmail.com> References: <1907e2ca0703051313t42920d29r5199d5d4cdd42e80@mail.gmail.com> <48001DA2-056E-4290-BEF8-BE5A06451CD4@gmail.com> Message-ID: <20070306145405.GA17111@waste.org> > I haven't had any time to work on backgroundrb in 3 months or so now > so there are some things that need to be fixed. Hopefully I will be > picking bdrb up to develop again in another 2-3 weeks or so. Then I > will fix the remaining issues. sadly the same has been the case for me, but I also expect to get back to bdrb in a couple of weeks if all works out. /skaar From masonhale at gmail.com Wed Mar 7 12:48:33 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 7 Mar 2007 11:48:33 -0600 Subject: [Backgroundrb-devel] postgres barfage revisted In-Reply-To: References: Message-ID: <8bca3aa10703070948j55730c53hb1c09ef587011467@mail.gmail.com> This is a new issue in Rails 1.2.x. If allow concurrency = true (in ActiveRecord) and you're using the pure ruby postgres-pr driver, then you will get a NoMethodError when async_exec or async_query is called. I opened a ticket and submitted a patch for this: http://dev.rubyonrails.org/ticket/7727 And it looks like it has been fixed in the rails trunk, but not released yet. You could apply the same fix directly to your local rails install in the interim. Also be aware that the native postgres adapter has some issues on Mac OS X, particularly when used with backgroundrb. This is really a problem for Apple to fix. More details here: http://rubyforge.org/pipermail/backgroundrb-devel/2006-December/000594.html The net result is that on Mac OS X, you are best off using the postgres-pr driver if you are working with backgroundrb, but if you are also using Rails 1.2 -- you'll need to apply the patch referenced above. best regards, Mason On 3/5/07, Ezra Zygmuntowicz wrote: > > > Note that this is only for the postgres bindings afaik. Mysql is > fine with allow concurrency = true > > -Ezra > > > On Mar 5, 2007, at 2:57 PM, Ezra Zygmuntowicz wrote: > > > Hi- > > > > Yeah with the new multi process bdrb you probably need to set > > allow_concurrency to false. I don't think you will have a problem > > with it like that. If you do run into a problem then you may have to > > put a mutex around the calls to your database: > > > > @mymutex = Mutex.new > > > > @mymutex.synchronize { > > # calls to database in here. > > } > > > > > > -Ezra > > > > On Mar 5, 2007, at 2:13 PM, Jack Nutting wrote: > > > >> First of all, I'd like to do what I've seen several others here do: > >> thank Ezra and any other contributors for producing a really great > >> plugin. Even though I'm having some minor problems, this is all so > >> approachable and easy to work with and dig into... I hate to > >> think of > >> the hoops you'd need to jump through (both as creator of such a > >> system > >> and as a user) to do something similar in Java. > >> > >> Now then... I've stumbled across the problem described here: > >> > >> http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/ > >> 000299.html > >> > >> wherein the native Ruby adaptor for Postgres causes Backgroundrb to > >> crap out with messages like this: > >> > >> ActiveRecord::StatementInvalid: NoMethodError: undefined method > >> `async_query' for #: > >> > >> The author's suggested solutions were to either use the compiled > >> ruby-postgres, or set ActiveRecord::Base.allow_concurrency to > >> false in > >> a start script. > >> > >> Well, the first option seems tricky to me; I haven't succeeded in > >> getting that compiled for Mac OS X yet (granted, I haven't given it > >> too much effort yet), so I figured I'd try the second option. I > >> couldn't find any start script where I could actually make that > >> setting have any effect, and eventually I realized that it's being > >> set > >> to true at worker_rails.rb:21. If I comment out that line, suddenly > >> I'm back in business! > >> > >> What I wonder is if I'm setting myself up for future catastrophe with > >> that change. I figure it's got to be in there for *some* reason. I > >> think I read somewhere that Backgroundrb changed from a multi-thread > >> to a multi-process architecture not too long ago; Is that line, > >> setting allow_concurrency to true, a holdover from the multi-threaded > >> days that's no longer needed? > >> > >> -- > >> // jack > >> // http://www.nuthole.com > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Ezra Zygmuntowicz > > -- Lead Rails Evangelist > > -- ez at engineyard.com > > -- Engine Yard, Serious Rails Hosting > > -- (866) 518-YARD (9273) > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/976623e9/attachment-0001.html From Marc at SoftwareHackery.Com Wed Mar 7 13:22:14 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 13:22:14 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? Message-ID: <20070307130913.B41525@me.softwarehackery.com> Hello - I am observing that calls to MiddleMan.worker return only after the worker has completed its work. This puzzles me, and I presume that I am doing something wrong. Can anyone make suggestions? Snippets from my code are: class ReportController < SecurityController ... def create_xml_report(report,start_time) constraints = get_constraints(report,start_time,false) args = { :constraints => constraints, :start_time => start_time, :report_params => report } # For reasons unknown, the following seems to block! session[:xml_report_job] = MiddleMan.new_worker(:class => :xml_report_worker, :args => args) end def get_progress ... worker = MiddleMan.worker(session[job]) end ... end class XmlReportWorker < BackgrounDRb::Worker::RailsBase include DRbUndumped ... def do_work(args) @progress = 0 @report_start_time = args[:start_time] @report_params = args[:report_params] constraints = args[:constraints] dtd_only = args[:dtd_only] io = Tmpfile.new('csv_report','/var/tmp') @path = io.path xml = Builder::XmlMarkup.new(:indent => 2, :target => io) xml.instruct! (dtd_only.nil? or !dtd_only) ? data(xml,constraints) : dtd(xml) io.close @progress = 100 end ... end Within the run_worker, the data method is where the bulk of the time is spent, though to my understanding, what I do in that method should not be able to impact the blocking state of the queries to get_progress. The data method does ALOT of postgresql interaction to extract the data needed to build the report, all of which is read-only. Does anyone have suggestions for what may be causing the blocking situation, and/or how to figure out and fix that issue? Thanks in advance - Marc From Marc at SoftwareHackery.Com Wed Mar 7 14:08:26 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 14:08:26 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <20070307130913.B41525@me.softwarehackery.com> References: <20070307130913.B41525@me.softwarehackery.com> Message-ID: <20070307140712.Q41525@me.softwarehackery.com> I see the I place a comment in my posting in the wrong place. See in context below: On Wed, 7 Mar 2007, Marc Evans wrote: > Hello - > > I am observing that calls to MiddleMan.worker return only after the worker > has completed its work. This puzzles me, and I presume that I am doing > something wrong. Can anyone make suggestions? > > Snippets from my code are: > > class ReportController < SecurityController > ... > def create_xml_report(report,start_time) > constraints = get_constraints(report,start_time,false) > args = { :constraints => constraints, > :start_time => start_time, > :report_params => report > } > # For reasons unknown, the following seems to block! > session[:xml_report_job] = > MiddleMan.new_worker(:class => :xml_report_worker, :args => args) The above line is NOT the problem line... > end > > def get_progress > ... > worker = MiddleMan.worker(session[job]) The above line IS the line that I am finding blocks when called, until the worker is complete with its work. > end > ... > end > > class XmlReportWorker < BackgrounDRb::Worker::RailsBase > include DRbUndumped > ... > def do_work(args) > @progress = 0 > @report_start_time = args[:start_time] > @report_params = args[:report_params] > constraints = args[:constraints] > dtd_only = args[:dtd_only] > io = Tmpfile.new('csv_report','/var/tmp') > @path = io.path > xml = Builder::XmlMarkup.new(:indent => 2, :target => io) > xml.instruct! > (dtd_only.nil? or !dtd_only) ? data(xml,constraints) : dtd(xml) > io.close > @progress = 100 > end > ... > end > > Within the run_worker, the data method is where the bulk of the time is > spent, though to my understanding, what I do in that method should not be > able to impact the blocking state of the queries to get_progress. The data > method does ALOT of postgresql interaction to extract the data needed to > build the report, all of which is read-only. > > Does anyone have suggestions for what may be causing the blocking > situation, and/or how to figure out and fix that issue? > > Thanks in advance - Marc > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From pelargir at gmail.com Wed Mar 7 14:23:09 2007 From: pelargir at gmail.com (Matthew) Date: Wed, 7 Mar 2007 14:23:09 -0500 Subject: [Backgroundrb-devel] Backgroundrb spiking CPU after first run on Mongrel cluster In-Reply-To: References: <8bca3aa10703040833m7d017b8aga650596d9b2acc6a@mail.gmail.com> <145C196C-1573-4191-9109-207DAC87A42F@gmail.com> Message-ID: Good news. I solved the problem by downgrading to Ruby 1.8.4 to match the dev box. Backgroundrb immediately started behaving correctly. There must be some conflict between Backgroundrb and Mongrel when running under Ruby 1.8.5on FreeBSD. I'm running 1.8.5 locally and Backgroundrb worked. Very strange, but I'm very glad to have this resolved now. Backgroundrb is quite solid on the prod box now. I have another question, but I'm going to post it in a new thread since it has nothing to do with this topic. Matthew On 3/5/07, Matthew wrote: > > Thanks for the reply, Ezra. Just tried that and I'm still experiencing the > same problem. > > As before, running from either script/console or the browser works fine on > the first go round. I can generate a worker, query it, and delete it. Trying > to generate a worker the second time is a different story, though. In the > browser the generate appears to work, but when I try to get a worker for the > returned key I get a nil object. In script/console, the generate just sits > there spinning until I kill the backgroundrb process. > > Something interesting I noticed as well is the dir that gets created in > /tmp with the PID on the end does not match the actual PID of my > backgroundrb process. Is it supposed to? > > -bash-2.05b$ ls /tmp/ba* > /tmp/backgroundrbunix_localhost_2020 > /tmp/backgroundrb.32589 > > -bash-2.05b$ ps aux | grep backgroundrb > cwarner 32591 0.0 0.7 21528 20564 ?? S 4:13PM 0:00.17 ruby: > backgroundrb (ruby) > cwarner 32592 0.0 0.7 21512 20552 ?? S 4:13PM 0:00.06 ruby: > backgroundrb_logger (ruby) > cwarner 32593 0.0 0.7 21468 20560 ?? S 4:13PM 0:00.05 ruby: > backgroundrb_results (ruby) > > > On an aside, thanks for building a really great tool here. It's simple to > use and perfect for what I'm trying to do. I'm sure the issue is on my end > with configuration or something. > > Matthew > > > On 3/5/07, Ezra Zygmuntowicz wrote: > > > > Mathew- > > > > This may be the same worker class name length overflow error. If > > your classname for your worker is too long then slave can fail to > > create the proper unix domain socket. Can you try to shorten the name > > of your worker class and try again? > > > > -Ezra > > > > > > On Mar 5, 2007, at 12:40 PM, Matthew wrote: > > > > > Thanks for the suggestion, Mason. I did that and noticed that each > > > time I attempted to create a new worker, I got a new entry. This is > > > what I ended up with in netstat after trying several attempts: > > > > > > -bash-2.05b$ netstat -an | grep 2001 > > > c4cb5af0 stream 0 0 0 c8cb5af0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8d58a64 stream 0 0 0 0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c4dcc94c stream 0 0 0 c97ac94c 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8d20690 stream 0 0 0 c8cb5000 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8cca348 stream 0 0 0 c4dcc690 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c8cf7b7c stream 0 0 0 c8d209d8 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > c4cb62bc stream 0 0 c6b63d68 0 0 > > > 0 ///tmp/backgroundrbunix_localhost_2001 > > > > > > My process list looked like this: > > > > > > -bash-2.05b$ ps aux | grep ruby > > > cwarner 30993 31.9 1.3 46932 39748 ?? R 3:27PM 2: 22.28 > > > ruby: pdf_generator_worker_dde6f63034382c4727 > > > cwarner 31131 31.6 1.3 47344 40160 ?? R 3:30PM 0:26.14 > > > ruby: pdf_generator_worker_031025c9be743ca12e > > > cwarner 31079 31.5 1.3 47072 39888 ?? R 3:29PM 0: 43.74 > > > ruby: pdf_generator_worker_c617aa5cc57c7b59b7 > > > cwarner 23921 0.1 3.1 103420 97144 ?? S 10:35AM 6:08.78 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23910 0.0 4.5 148668 140424 ?? S 10:35AM 11: 28.06 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23914 0.0 3.1 107792 98560 ?? S 10:35AM 7:13.36 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 23918 0.0 3.1 106548 97144 ?? S 10:35AM 6: 39.64 > > > ruby /usr/local/lib/ruby/gems/1.8/gems/mongre > > > cwarner 30836 0.0 1.3 47480 40300 ?? S 3:24PM 0:02.42 > > > ruby: backgroundrb (ruby) > > > cwarner 30837 0.0 0.7 21512 20552 ?? S 3:24PM 0:00.06 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 30838 0.0 0.7 21468 20560 ?? S 3:24PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > cwarner 31162 0.0 0.0 1484 832 p3 S+ 3:31PM 0:00.00 > > > grep ruby > > > > > > Looks like a new pdf_generator_worker process gets created and > > > immediately freezes each time I try to create a worker. I can't > > > communicate with any of these workers and have to manually kill > > > them to get them to go away. > > > > > > Anyone else have an idea of what could be going on? > > > > > > Matthew > > > > > > > > > On 3/4/07, Mason Hale wrote:Matthew - > > > > > > Trying running netstat to see what ports your mongrels and > > > backgroundrb processes are using. I've run into some problems with > > > those two trying to use the same ports in past. > > > > > > Mason > > > > > > On 3/3/07, Matthew < pelargir at gmail.com> wrote: > > > I've been troubleshooting this problem for several hours now and > > > haven't been able to figure out what's wrong. I have two boxes, dev > > > and prod, both are identical except that prod uses Ruby 1.8.5 and > > > dev uses 1.8.4 . Both are running Slave 1.2.0, Daemons 1.0.5, and > > > Backroundrb 0.2.1. > > > > > > The behavior I'm seeing is this. When I first start Backgroundrb, > > > this is what the process list looks like: > > > > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80614 0.0 1.6 57944 50628 ?? S 10:28PM 0:03.48 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > > usr/local/bin/ruby18 /usr/local/bin/mon > > > cwarner 80804 0.0 1.3 46872 39688 ?? S 10:37PM 0:02.43 > > > ruby: backgroundrb (ruby) > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > > > > My four Mongrels in the cluster sitting out there, and the three > > > backgroundrb processes. I then proceed to generate a worker using > > > MiddleMan via IRB, get the auto generated job key, look up the > > > worker, and then delete him. All goes well, and the process list > > > still looks like what I posted above. I can do this as many times > > > as I want to via IRB and nothing goes wrong. However, when I try to > > > call the same code through a Rails action, Backgroundrb doesn't > > > behave. The browser just sits there spinning. Examining the process > > > list yields: > > > > > > cwarner 80934 92.8 1.3 46936 39752 ?? R 10:45PM 0:02.07 > > > ruby: backgroundrb (ruby) > > > cwarner 80804 0.1 1.3 47072 39888 ?? S 10:37PM 0:02.60 > > > ruby: backgroundrb (ruby) > > > cwarner 80608 0.0 1.6 57852 50480 ?? S 10:28PM 0: 03.36 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80611 0.0 1.6 58768 51356 ?? S 10:28PM 0:03.59 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80614 0.0 1.6 57944 50628 ?? R 10:28PM 0:03.51 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80618 0.0 1.6 57384 50060 ?? S 10:28PM 0:03.26 / > > > usr/local/bin/ruby18 /usr/local/bin/mong > > > cwarner 80805 0.0 0.7 21512 20548 ?? S 10:37PM 0:00.07 > > > ruby: backgroundrb_logger (ruby) > > > cwarner 80806 0.0 0.7 21468 20556 ?? S 10:37PM 0:00.06 > > > ruby: backgroundrb_results (ruby) > > > cwarner 80936 0.0 0.0 1484 828 p3 S+ 10:45PM 0:00.00 > > > grep ruby > > > > > > Aha! An extra Backgroundrb process that's spiking the CPU. I've let > > > the browser sit there until it times out, and the Backgroundrb > > > process will still be there, spiking the CPU. If I stop > > > Backgroundrb at this point, the extra process is still there. I > > > have to kill it manually to get rid of it. > > > > > > Something else I've noticed is that, once this process is out > > > there, even if I try creating new workers via IRB, the call to > > > MiddleMan.new_worker just sits there. So this definitely appears to > > > be a problem with the way Backgroundrb and my Mongrel clusters are > > > interacting, but I'm not sure where to even begin. It's frustrating > > > that this isn't happening on the dev box. I've tried to figure out > > > what could be different between the boxes, but other than the Ruby > > > versions, I'm not seeing anything. > > > > > > I'm out of options. Does anyone have any ideas? > > > > > > Thanks, > > > Matthew > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Ezra Zygmuntowicz > > -- Lead Rails Evangelist > > -- ez at engineyard.com > > -- Engine Yard, Serious Rails Hosting > > -- (866) 518-YARD (9273) > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/0354fced/attachment-0001.html From pelargir at gmail.com Wed Mar 7 14:25:12 2007 From: pelargir at gmail.com (Matthew) Date: Wed, 7 Mar 2007 14:25:12 -0500 Subject: [Backgroundrb-devel] Garbage collection kills my logger and results jobs too Message-ID: I wrote the following script to run garbage collection on Backgroundrb workers older than 30 minutes: class WorkerCleaner def self.remove_stale_workers MiddleMan.gc!(Time.now - 60*30) end end I stuck is under /lib in my Rails app and run it using script/runner from a cron job. The only problem I'm seeing is that this script kills my backgroundrb_results and backgroundrb_logger jobs after they've been running for more than 30 minutes. Is this expected? How can I alter this script to ignore the results and logger jobs and let them continue to run while cleaning up anything else that has expired? Thanks, Matthew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/efeba878/attachment.html From masonhale at gmail.com Wed Mar 7 14:27:08 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 7 Mar 2007 13:27:08 -0600 Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <20070307130913.B41525@me.softwarehackery.com> References: <20070307130913.B41525@me.softwarehackery.com> Message-ID: <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> # For reasons unknown, the following seems to block! session[:xml_report_job] = MiddleMan.new_worker(:class => :xml_report_worker, :args => args) The above looks right to me. It should not matter at all what is in the worker class. When you say it is blocking -- how do you know? What is the symptom? Is is just slow? If you create a test worker that loops indefinitely (or for a sufficiently long time, "sleep 120" should do) -- does it that block when you create a new instance with MiddleMan.new_worker? If it is just slow to return, I find there is some lag when spawning a new process, which can make the MiddleMan.new_worker call sluggish to return. But in my experience it does not block until the worker is done. It *does* block until the process is created. A way to speed up the new_worker calls is to set an explicit job_key to call the do_work method on a long running process, rather than starting a new process. Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/6ad37081/attachment.html From Marc at SoftwareHackery.Com Wed Mar 7 14:42:50 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 14:42:50 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> Message-ID: <20070307143846.F41525@me.softwarehackery.com> On Wed, 7 Mar 2007, Mason Hale wrote: > # For reasons unknown, the following seems to block! > session[:xml_report_job] = > MiddleMan.new_worker(:class => :xml_report_worker, :args => args) > > The above looks right to me. It should not matter at all what is in the > worker class. You may be seen my follow up post, where I indicated that I put the comment aboev the wrong line of code. The line that is blocking is the line for MiddleMan.worker. > When you say it is blocking -- how do you know? What is the symptom? Is is > just slow? The symptom is that when I try to obtain the worker in order to then obtain the progress value, the call to MiddleMan.worker blocks until the worker has completed its work. > If you create a test worker that loops indefinitely (or for a sufficiently > long time, "sleep 120" should do) -- does it that block when you create a > new instance with MiddleMan.new_worker? In the example code I sent, I replaced one line in do_work as shown here: # (dtd_only.nil? or !dtd_only) ? data(xml,constraints) : dtd(xml) i=0 while (i < 100) i += 10 sleep 10 @progress = i end By doing so, I no longer experiencing the blocking behavior. The clearly tells me that something in the data method is the direct/indirect cause. > If it is just slow to return, I find there is some lag when spawning a new > process, which can make the MiddleMan.new_worker call sluggish to return. > But in my experience it does not block until the worker is done. It *does* > block until the process is created. > > A way to speed up the new_worker calls is to set an explicit job_key to call > the do_work method on a long running process, rather than starting a new > process. > > Mason > Thanks for your followup message. - Marc From Marc at SoftwareHackery.Com Wed Mar 7 14:55:20 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 14:55:20 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> Message-ID: <20070307145023.I41525@me.softwarehackery.com> On Wed, 7 Mar 2007, Mason Hale wrote: > # For reasons unknown, the following seems to block! > session[:xml_report_job] = > MiddleMan.new_worker(:class => :xml_report_worker, :args => args) > > The above looks right to me. It should not matter at all what is in the > worker class. > > When you say it is blocking -- how do you know? What is the symptom? Is is > just slow? > > If you create a test worker that loops indefinitely (or for a sufficiently > long time, "sleep 120" should do) -- does it that block when you create a > new instance with MiddleMan.new_worker? > > If it is just slow to return, I find there is some lag when spawning a new > process, which can make the MiddleMan.new_worker call sluggish to return. > But in my experience it does not block until the worker is done. It *does* > block until the process is created. > > A way to speed up the new_worker calls is to set an explicit job_key to call > the do_work method on a long running process, rather than starting a new > process. > > Mason I am discovering more information on this, but don't understand the underlying cause still. Below is my data method and a method it calls within scope od do_work: def data(xml,constraints) dtd(xml) @progress = 10 xml.telemetry do telem_to_xml(CacheTelemetry,:cache_telem_to_xml,constraints,xml) @progress = 30 telem_to_xml(CountTelemetry,:count_telem_to_xml,constraints,xml) @progress = 50 telem_to_xml(StreamTelemetry,:stream_telem_to_xml,constraints,xml) @progress = 70 telem_to_xml(PolledTelemetry,:polled_telem_to_xml,constraints,xml) @progress = 90 end end def telem_to_xml(klass,xml_method,constraints,xml) opts = constraints.dup opts[:limit] = 5000 offset = 0 while true do opts[:offset] = offset offset += opts[:limit] found = klass.find_restricted(:all,opts) break if (found.nil? or found.size < 1) found.each { |dct| self.method(xml_method).call(dct,xml) ; sleep 0.001 } end end Notice in the above the "sleep 0.001". Without that sleep, I get the blocking behavior I described in the original post. With it, I get the behavior I would expect, which is that I can retrieve the progress reasonably quickly and repeatedly. Again, any suggestions? - Marc From masonhale at gmail.com Wed Mar 7 16:46:32 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 7 Mar 2007 15:46:32 -0600 Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <20070307145023.I41525@me.softwarehackery.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> Message-ID: <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> On 3/7/07, Marc Evans wrote: > Notice in the above the "sleep 0.001". Without that sleep, I get the > blocking behavior I described in the original post. With it, I get the > behavior I would expect, which is that I can retrieve the progress > reasonably quickly and repeatedly. > > Again, any suggestions? That is very odd. My gut says it must have something to do with locking between threads. I've previously run into thread-related issues around the worker results feature, finding that it is unreliable. This smells like a similar problem. more detail: http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/000638.html http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43 My suggestion would be to store your worker state externally in a database or other store. That "sleep 0.0001" works at all is weird, and I'd be hesitant to rely on it continuing to work. Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/587066bc/attachment.html From ezmobius at gmail.com Wed Mar 7 16:53:10 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 7 Mar 2007 13:53:10 -0800 Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> Message-ID: <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote: > > On 3/7/07, Marc Evans wrote: > Notice in the above the "sleep 0.001". Without that sleep, I get the > blocking behavior I described in the original post. With it, I get the > behavior I would expect, which is that I can retrieve the progress > reasonably quickly and repeatedly. > > Again, any suggestions? > > That is very odd. My gut says it must have something to do with > locking between threads. > I've previously run into thread-related issues around the worker > results feature, finding > that it is unreliable. This smells like a similar problem. > > more detail: > http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/ > 000638.html > http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43 > > My suggestion would be to store your worker state externally in a > database or other store. > > That "sleep 0.0001" works at all is weird, and I'd be hesitant to > rely on it continuing to work. > > Mason > Actually the sleep makes sense. It would seem that your code is blocking the thread which makes all the other threads block. Putting a sleep in there gives the scheduler time to schedule the other threads to be run. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From Marc at SoftwareHackery.Com Wed Mar 7 16:54:50 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 16:54:50 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> Message-ID: <20070307164902.B41525@me.softwarehackery.com> On Wed, 7 Mar 2007, Mason Hale wrote: > On 3/7/07, Marc Evans wrote: > >> Notice in the above the "sleep 0.001". Without that sleep, I get the >> blocking behavior I described in the original post. With it, I get the >> behavior I would expect, which is that I can retrieve the progress >> reasonably quickly and repeatedly. >> >> Again, any suggestions? > > > That is very odd. My gut says it must have something to do with locking > between threads. > I've previously run into thread-related issues around the worker results > feature, finding > that it is unreliable. This smells like a similar problem. > > more detail: > http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/000638.html > http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43 > > My suggestion would be to store your worker state externally in a database > or other store. > > That "sleep 0.0001" works at all is weird, and I'd be hesitant to rely on it > continuing to work. > > Mason > Thanks again for the follow up. I have to agree with your observation. I have also since found that "sleep 0" removes the blocking problem, which further supports your assertions. One last thing I will try, based on other feedback about ruby 1.8.5 problems, is to try going back to 1.8.4 to see if that makes any difference. For reference, I am using the following: FreeBSD-6 AMD Opteron ruby-1.8.5 DBbackgroundRB-0.2.1 - Marc From masonhale at gmail.com Wed Mar 7 17:01:33 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 7 Mar 2007 16:01:33 -0600 Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> Message-ID: <8bca3aa10703071401r319402bie6664e5d53794122@mail.gmail.com> On 3/7/07, Ezra Zygmuntowicz wrote: > > That "sleep 0.0001" works at all is weird, and I'd be hesitant to > > rely on it continuing to work. > > Actually the sleep makes sense. It would seem that your code is > blocking the thread which makes all the other threads block. Putting > a sleep in there gives the scheduler time to schedule the other > threads to be run. > Yeah, I'd buy that. Would also explain 'sleep 0' working. Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070307/aa10c2cc/attachment-0001.html From Marc at SoftwareHackery.Com Wed Mar 7 17:07:52 2007 From: Marc at SoftwareHackery.Com (Marc Evans) Date: Wed, 7 Mar 2007 17:07:52 -0500 (EST) Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> Message-ID: <20070307170359.H41525@me.softwarehackery.com> On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote: > > On Mar 7, 2007, at 1:46 PM, Mason Hale wrote: > >> >> On 3/7/07, Marc Evans wrote: >> Notice in the above the "sleep 0.001". Without that sleep, I get the >> blocking behavior I described in the original post. With it, I get the >> behavior I would expect, which is that I can retrieve the progress >> reasonably quickly and repeatedly. >> >> Again, any suggestions? >> >> That is very odd. My gut says it must have something to do with locking >> between threads. >> I've previously run into thread-related issues around the worker results >> feature, finding >> that it is unreliable. This smells like a similar problem. >> >> more detail: >> http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/000638.html >> http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43 >> >> My suggestion would be to store your worker state externally in a database >> or other store. >> >> That "sleep 0.0001" works at all is weird, and I'd be hesitant to rely on >> it continuing to work. >> >> Mason >> > > Actually the sleep makes sense. It would seem that your code is > blocking the thread which makes all the other threads block. Putting a sleep > in there gives the scheduler time to schedule the other threads to be run. > > Cheers- > > -- Ezra Zygmuntowicz-- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > So, is there a "best practice" method of dealing with this? My worker code is in a tight loop, marshalling a few million records into XML, outputting the marshalled results to a file. I would have expected that each database query (I grab 5000 rows at a time) and each IO write would be a context switch candidate, but aparenetly not. Is there a call I should make to explicitely yield periodically? - Marc From ezmobius at gmail.com Wed Mar 7 17:13:40 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 7 Mar 2007 14:13:40 -0800 Subject: [Backgroundrb-devel] MiddleMan.worker blocks? In-Reply-To: <20070307170359.H41525@me.softwarehackery.com> References: <20070307130913.B41525@me.softwarehackery.com> <8bca3aa10703071127w62339193o23909a047906c4f5@mail.gmail.com> <20070307145023.I41525@me.softwarehackery.com> <8bca3aa10703071346u22421a86sa6428e7b65333940@mail.gmail.com> <1A003C70-0A84-463B-A2CA-7518F1F3C8FD@gmail.com> <20070307170359.H41525@me.softwarehackery.com> Message-ID: <359B608C-323F-44F8-8708-BC170E202D7A@gmail.com> On Mar 7, 2007, at 2:07 PM, Marc Evans wrote: > > On Wed, 7 Mar 2007, Ezra Zygmuntowicz wrote: > >> >> On Mar 7, 2007, at 1:46 PM, Mason Hale wrote: >> >>> On 3/7/07, Marc Evans wrote: >>> Notice in the above the "sleep 0.001". Without that sleep, I get the >>> blocking behavior I described in the original post. With it, I >>> get the >>> behavior I would expect, which is that I can retrieve the progress >>> reasonably quickly and repeatedly. >>> Again, any suggestions? >>> That is very odd. My gut says it must have something to do with >>> locking between threads. >>> I've previously run into thread-related issues around the worker >>> results feature, finding >>> that it is unreliable. This smells like a similar problem. >>> more detail: >>> http://rubyforge.org/pipermail/backgroundrb-devel/2007-January/ >>> 000638.html >>> http://backgroundrb.devjavu.com/projects/backgroundrb/ticket/43 >>> My suggestion would be to store your worker state externally in a >>> database or other store. >>> That "sleep 0.0001" works at all is weird, and I'd be hesitant to >>> rely on it continuing to work. >>> Mason >> >> Actually the sleep makes sense. It would seem that your code is >> blocking the thread which makes all the other threads block. >> Putting a sleep in there gives the scheduler time to schedule the >> other threads to be run. >> >> Cheers- >> >> -- Ezra Zygmuntowicz-- Lead Rails Evangelist >> -- ez at engineyard.com >> -- Engine Yard, Serious Rails Hosting >> -- (866) 518-YARD (9273) >> > > So, is there a "best practice" method of dealing with this? My > worker code is in a tight loop, marshalling a few million records > into XML, outputting the marshalled results to a file. I would have > expected that each database query (I grab 5000 rows at a time) and > each IO write would be a context switch candidate, but aparenetly > not. Is there a call I should make to explicitely yield periodically? > > - Marc Marc- It's hard to say anything about a best practice here. I have never had to do a sleep0 to get other threads scheduled but I am carefull to not use anything that is blocking IO. Ruby's green threads are admittedly weak sauce :( If you do a blocking IO call in one thread *all* threads will block and none will schedule until it finishes. When you say marshalling millions of records to xml I think you may just bee swamping the hell out of the interpreter and its blocking in that thread for some reason. I think in your case putting the sleep 0 in there to get other threads scheduled is a hack at best but I think it's the best hack in this case right now. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From david.gurba at gmail.com Thu Mar 8 03:04:11 2007 From: david.gurba at gmail.com (David Gurba) Date: Thu, 8 Mar 2007 00:04:11 -0800 Subject: [Backgroundrb-devel] worker cant delete? Message-ID: Hi, I get the following error from my client program when I try to call delete after my worker is completed: goten at goten-server:~/projects/svn_working/m29svn_back/app$ ruby backup_app.rb {:done=>true, :txt=>["piping to tmp", "piping done", "getting youngest", "youngest got", "destfile /home/goten/projects/svn_working/svn_tool/backups/backup_test123_2007-03-08_rev1.gz", "backup file created at Thu Mar 08 00:00:04 PST 2007"], :step=>1, :steps=>1} (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1552:in `delete': wrong number of arguments (0 for 1) (ArgumentError) from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1552:in `perform_without_block' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1512:in `perform' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1586:in `main_loop' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1582:in `main_loop' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1578:in `main_loop' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1424:in `run' from (drbunix:///tmp/backgroundrbunix_localhost_2000) (drbunix:///tmp/backgroundrb.6015/svn_worker_3589c0ce3aa090fc5a535718b965af26_0_0.688288742005734) /usr/lib/ruby/1.8/drb/drb.rb:1344:in `initialize' ... 43 levels... from (drbunix:///tmp/backgroundrbunix_localhost_2000) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.5/lib/daemons.rb:181:in `run_proc' from (drbunix:///tmp/backgroundrbunix_localhost_2000) /home/goten/projects/svn_working/m29svn_back/vendor/backgroundrb-0.2.1/server/lib/backgroundrb_server.rb:301:in `run' from (drbunix:///tmp/backgroundrbunix_localhost_2000) ../vendor/backgroundrb-0.2.1/script/backgroundrb:29 from backup_app.rb:14 I am not certain why this is happening ... it seems to happen when I redirect $stdout from child processes of my worker. This does not happen when I call the following in my worker: output = system(cmd); Any help appreciated :) regards, David G. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070308/26e18547/attachment.html From david.gurba at gmail.com Thu Mar 8 03:11:22 2007 From: david.gurba at gmail.com (David Gurba) Date: Thu, 8 Mar 2007 00:11:22 -0800 Subject: [Backgroundrb-devel] cant fully capture child output Message-ID: I cant seem to get the io from a child process of a worker back to my client. There most be something academic I'm missing :/. I looked at examples in Pickaxe and Ruby PLEAC library ... and I believe the following should work: # called from within do_work... def backup(repo) logger.info('SvnWorker backup repo '+repo) tmp = [] my_project = DIRS[:project_repos] + "/#{repo}" dummyfile = 'backuptmp' temp_file = "#{DIRS[:backups]}/#{dummyfile}" svn_cmd = "svnadmin dump #{my_project} | gzip -9 > #{temp_file}" tmp << "piping to tmp" results[:txt] = tmp ### i dont get the output between here ### readme, writeme = IO.pipe pid = fork { # child $stdout = writeme readme.close exec(svn_cmd) } # parent Process.waitpid(pid,0) writeme.close while readme.gets do results[:txt] = $_ end ### i dont get the output between here ### readme.close tmp << "piping done" results[:txt] = tmp tmp << "getting youngest" results[:txt] = tmp youngest = `svnlook youngest #{my_project}` tmp << "youngest got" results[:txt] = tmp rev = "_rev#{youngest.chomp!}" fname = [ "backup_#{repo}_", Time.now.strftime('%Y-%m-%d'), "#{rev}", '.gz' ] destfile = "#{DIRS[:backups]}/#{fname.join}" tmp << "destfile #{destfile}" results[:txt] = tmp FileUtils.mv(temp_file, destfile) tmp << "backup file created at #{Time.now}" results[:txt] = tmp end My previous messages shows taht all data before and after my comments is received by the client. Thanks (again) for any assistance, I'm pretty stumped. I keep plugging away at this and once I get these issues resolved the surrounding app is completed :) [I tested this app prior to using a background job server ... it just timed out on huge projects :/] regards, DavidG. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070308/f72f6093/attachment.html From ezmobius at gmail.com Thu Mar 8 03:48:10 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 8 Mar 2007 00:48:10 -0800 Subject: [Backgroundrb-devel] cant fully capture child output In-Reply-To: References: Message-ID: <90C55B0C-9C2C-4C26-A9D8-3F37B0F60691@brainspl.at> David- Backgroundrb is already doing a lot of forking and redirecting of IO streams. I think you are running into problems because you are forking again in a worker. You can't really do that and still have things work properly. -Ezra On Mar 8, 2007, at 12:11 AM, David Gurba wrote: > I cant seem to get the io from a child process of a worker back to > my client. There most be something academic I'm missing :/. I > looked at examples in Pickaxe and Ruby PLEAC library ... and I > believe the following should work: > > # called from within do_work... > def backup(repo) > logger.info('SvnWorker backup repo '+repo) > tmp = [] > my_project = DIRS[:project_repos] + "/#{repo}" > dummyfile = 'backuptmp' > temp_file = "#{DIRS[:backups]}/#{dummyfile}" > svn_cmd = "svnadmin dump #{my_project} | gzip -9 > #{temp_file}" > tmp << "piping to tmp" > results[:txt] = tmp > > ### i dont get the output between here ### > > readme, writeme = IO.pipe > pid = fork { > # child > $stdout = writeme > readme.close > exec(svn_cmd) > } > # parent > Process.waitpid(pid,0) > writeme.close > while readme.gets do > results[:txt] = $_ > end > > ### i dont get the output between here ### > > readme.close > tmp << "piping done" > results[:txt] = tmp > tmp << "getting youngest" > results[:txt] = tmp > youngest = `svnlook youngest #{my_project}` > tmp << "youngest got" > results[:txt] = tmp > rev = "_rev#{youngest.chomp!}" > fname = [ > "backup_#{repo}_", > Time.now.strftime('%Y-%m-%d'), > "#{rev}", > '.gz' > ] > destfile = "#{DIRS[:backups]}/#{ fname.join}" > tmp << "destfile #{destfile}" > results[:txt] = tmp > FileUtils.mv(temp_file, destfile) > tmp << "backup file created at #{Time.now}" > results[:txt] = tmp > end > > My previous messages shows taht all data before and after my > comments is received by the client. > > Thanks (again) for any assistance, I'm pretty stumped. I keep > plugging away at this and once I get these issues resolved the > surrounding app is completed :) [I tested this app prior to using a > background job server ... it just timed out on huge projects :/] > > regards, > DavidG. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From mwhitt89 at gmail.com Fri Mar 9 17:17:30 2007 From: mwhitt89 at gmail.com (Matthew Whittaker) Date: Fri, 9 Mar 2007 17:17:30 -0500 Subject: [Backgroundrb-devel] Does Bdrd 2.x handle fragment caching still? Message-ID: <50FC6C17-CE06-494B-B631-42106810D223@gmail.com> From the API I was unable to determine if the new architecture for BackgrounDRB still handles fragment caching? If so, how can I use it? I am upgrading to the new architecture from the old one and used this functionality in the old code base. I like the new architecture, thank you. Cheers, Matt From benr at x-cr.com Mon Mar 12 13:47:22 2007 From: benr at x-cr.com (Ben Reubenstein) Date: Mon, 12 Mar 2007 11:47:22 -0600 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment Message-ID: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> Hello ~ I am using the Capistrano recipe outlined in this post to restart my backgroundrb server: http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/000718.html My Version: ~~~~~~~~~~~~ task :before_restart, :roles => :app do restart_backgroundrb end task :start_backgroundrb, :roles => :app do sudo "nohup /var/www/rails/#{application}/current/script/backgroundrb/start " end task :stop_backgroundrb, :roles => :app do sudo "kill -9 `cat /var/www/rails/#{application}/current/log/backgroundrb.pid` 2>/dev/null; true" end task :restart_backgroundrb, :roles => :app do stop_backgroundrb start_backgroundrb end ~~~~~~~~~~~~ The backgroundrb server is stopped and started but hangs at the start_backgroundrb task. Capistrano shows: * executing task start_backgroundrb * executing "sudo nohup /var/www/rails/*************/current/script/backgroundrb/start" servers: ["host"] [host] executing command ** [out :: host] nohup: ignoring input and appending output to `nohup.out' It will hang there and not restart the main web app process. I updated my gems to see if this was a version issue, but am experiencing the same problem. I also tried running this task :after_restart, the web app is successfully restarted, but it still hangs. Thanks in advance for any ideas on how to fix this. ~ Ben -- Ben Reubenstein 303-947-0446 http://www.benr75.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070312/3f321843/attachment.html From ezmobius at gmail.com Mon Mar 12 14:18:22 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 12 Mar 2007 11:18:22 -0700 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> Message-ID: <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> Yeah there are some weird interactions between capistrano and some daemons, the old version of backgroundrb exibits this behavior. That way I get around it is by creating an init.d script, then letting capuse that to restart bdrb. Works like a charm. Here is an init.d script for Gentoo systems, it should be easy to adapt to any other unixy system.: #!/sbin/runscript # Copyright 1999-2004 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v 1.1 2006/07/04 16:58:38 voxus Exp $ depend() { need net use dns after gfs } start() { ebegin "Starting bdrb2" RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ start -d eend $? "Failed to start bdrb2" } stop() { ebegin "Stopping bdrb2" kill -9 `cat /data/toolbox/current/log/ backgroundrb.pid` 2>/dev/null; true eend $? "Failed to stop bdrb2" } Cheers- -Ezra On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > Hello ~ > > I am using the Capistrano recipe outlined in this post to restart > my backgroundrb server: > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > 000718.html > > My Version: > ~~~~~~~~~~~~ > task :before_restart, :roles => :app do > restart_backgroundrb > end > > task :start_backgroundrb, :roles => :app do > sudo "nohup /var/www/rails/#{application}/current/script/ > backgroundrb/start " > end > > task :stop_backgroundrb, :roles => :app do > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > backgroundrb.pid` 2>/dev/null; true" > end > > task :restart_backgroundrb, :roles => :app do > stop_backgroundrb > start_backgroundrb > end > ~~~~~~~~~~~~ > > The backgroundrb server is stopped and started but hangs at the > start_backgroundrb task. Capistrano shows: > > * executing task start_backgroundrb > * executing "sudo nohup /var/www/rails/*************/current/ > script/backgroundrb/start" > servers: ["host"] > [host] executing command > ** [out :: host] nohup: ignoring input and appending output to > `nohup.out' > > It will hang there and not restart the main web app process. I > updated my gems to see if this was a version issue, but am > experiencing the same problem. I also tried running this > task :after_restart, the web app is successfully restarted, but it > still hangs. > > Thanks in advance for any ideas on how to fix this. > > ~ Ben > > -- > Ben Reubenstein > 303-947-0446 > http://www.benr75.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From benr at x-cr.com Mon Mar 12 15:17:30 2007 From: benr at x-cr.com (Ben Reubenstein) Date: Mon, 12 Mar 2007 13:17:30 -0600 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> Message-ID: <44c79a160703121217t165a2a56w19a33aeac528d3e7@mail.gmail.com> Fantastic! This approach worked well and was easy for me to implement as my production servers run Gentoo. Thx! ~ Ben On 3/12/07, Ezra Zygmuntowicz wrote: > > > Yeah there are some weird interactions between capistrano and some > daemons, the old version of backgroundrb exibits this behavior. That > way I get around it is by creating an init.d script, then letting > capuse that to restart bdrb. Works like a charm. Here is an init.d > script for Gentoo systems, it should be easy to adapt to any other > unixy system.: > > #!/sbin/runscript > # Copyright 1999-2004 Gentoo Foundation > # Distributed under the terms of the GNU General Public License v2 > # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v > 1.1 2006/07/04 16:58:38 voxus Exp $ > > > depend() { > need net > use dns > after gfs > } > start() { > ebegin "Starting bdrb2" > RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ > start -d > eend $? "Failed to start bdrb2" > } > > stop() { > ebegin "Stopping bdrb2" > kill -9 `cat /data/toolbox/current/log/ > backgroundrb.pid` 2>/dev/null; true > eend $? "Failed to stop bdrb2" > } > > > > Cheers- > -Ezra > > > On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > > > Hello ~ > > > > I am using the Capistrano recipe outlined in this post to restart > > my backgroundrb server: > > > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > > 000718.html > > > > My Version: > > ~~~~~~~~~~~~ > > task :before_restart, :roles => :app do > > restart_backgroundrb > > end > > > > task :start_backgroundrb, :roles => :app do > > sudo "nohup /var/www/rails/#{application}/current/script/ > > backgroundrb/start " > > end > > > > task :stop_backgroundrb, :roles => :app do > > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > > backgroundrb.pid` 2>/dev/null; true" > > end > > > > task :restart_backgroundrb, :roles => :app do > > stop_backgroundrb > > start_backgroundrb > > end > > ~~~~~~~~~~~~ > > > > The backgroundrb server is stopped and started but hangs at the > > start_backgroundrb task. Capistrano shows: > > > > * executing task start_backgroundrb > > * executing "sudo nohup /var/www/rails/*************/current/ > > script/backgroundrb/start" > > servers: ["host"] > > [host] executing command > > ** [out :: host] nohup: ignoring input and appending output to > > `nohup.out' > > > > It will hang there and not restart the main web app process. I > > updated my gems to see if this was a version issue, but am > > experiencing the same problem. I also tried running this > > task :after_restart, the web app is successfully restarted, but it > > still hangs. > > > > Thanks in advance for any ideas on how to fix this. > > > > ~ Ben > > > > -- > > Ben Reubenstein > > 303-947-0446 > > http://www.benr75.com > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > -- Ben Reubenstein 303-947-0446 http://www.benr75.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070312/5080cb83/attachment.html From jnutting at gmail.com Tue Mar 13 11:06:58 2007 From: jnutting at gmail.com (Jack Nutting) Date: Tue, 13 Mar 2007 16:06:58 +0100 Subject: [Backgroundrb-devel] worker starting twice Message-ID: Hi gang, Thanks to other help I've gotten on this list, I've got backgroundrb up and running using postgres alongside my rails app. I've designed the system to have several eternally-running workers that periodically poll the database to see if there's any work for them to do, and set things up in the config file to make them restart quickly if any of them should crash. One odd thing I've noticed is that one of my workers, with the key "mr", seems to start twice (without the first one having completed). I've got some logging that occurs when each starts: 20070313-15:45:17 (5594) Starting WorkerLogger 20070313-15:45:17 (5595) In ResultsWorker 20070313-15:45:21 (5595) MbReceiver mr started. 20070313-15:45:21 (5595) Sender s started 20070313-15:45:22 (5595) MbSender ms1 started 20070313-15:45:22 (5595) MbSender ms2 started 20070313-15:45:22 (5595) Receiver r started 20070313-15:45:30 (5595) MbReceiver mr started. The relevant chunk of backgroundrb_schedules.yml looks like this: mb_receiver: :class: :mb_receiver :job_key: :mr :worker_method: :do_work :args: :sleep_time: 5 :trigger_args: 0,10,20,30,40,50 * * * * * * The other workers have identical trigger_args, but don't exhibit this behavior. The do_work method just calls another method that loops forever, and I wrapped that call in an exception handler to see if it was crapping out or something, but it doesn't seem to be. FWIW, here's the backgroundrb_server.log from the same period of time: 20070313-15:45:17 (5594) Starting BackgrounDRb Server 20070313-15:45:17 (5594) acl: denyallallowlocalhost 127.0.0.1orderdeny allow 20070313-15:45:17 (5594) temp_dir: /tmp 20070313-15:45:17 (5594) port: 2000 20070313-15:45:17 (5594) worker_dir: /Users/jnutting/work/m_project/lib/workers 20070313-15:45:17 (5594) protocol: drbunix 20070313-15:45:17 (5594) uri: drbunix:///tmp/backgroundrbunix_localhost_2000 20070313-15:45:17 (5594) config: /Users/jnutting/work/m_project/config/backgroundrb.yml 20070313-15:45:17 (5594) rails_env: development 20070313-15:45:17 (5594) socket_dir: /tmp/backgroundrb.5594 20070313-15:45:17 (5594) pool_size: 5 20070313-15:45:17 (5594) host: localhost 20070313-15:45:17 (5594) Starting worker: BackgrounDRb::Worker::WorkerLogger backgroundrb_logger (backgroundrb_logger) () 20070313-15:45:17 (5594) Starting worker: BackgrounDRb::Worker::WorkerResults backgroundrb_results (backgroundrb_results) () 20070313-15:45:17 (5594) Loading Worker Class File: /Users/jnutting/work/m_project/lib/workers/mb_receiver.rb 20070313-15:45:17 (5594) Loading Worker Class File: /Users/jnutting/work/m_project/lib/workers/mb_sender.rb 20070313-15:45:17 (5594) Loading Worker Class File: /Users/jnutting/work/m_project/lib/workers/receiver.rb 20070313-15:45:17 (5594) Loading Worker Class File: /Users/jnutting/work/m_project/lib/workers/sender.rb 20070313-15:45:17 (5594) Loading Sechedule: job_keymrclassmb_receiverargssleep_time5 worker_method_argssleep_time5job_keymrclassmb_receiverargssleep_time5trigger_typecron_triggertrigger_args0,10,20,30,40,50 * * * * * *worker_methoddo_work # 20070313-15:45:17 (5594) Loading Sechedule: job_keysclasssenderargssleep_time5 worker_method_argssleep_time5job_keysclasssenderargssleep_time5trigger_typecron_triggertrigger_args0,10,20,30,40,50 * * * * * *worker_methoddo_work # 20070313-15:45:17 (5594) Loading Sechedule: job_keyms1classmb_senderargssleep_time5 worker_method_argssleep_time5job_keyms1classmb_senderargssleep_time5trigger_typecron_triggertrigger_args0,10,20,30,40,50 * * * * * *worker_methoddo_work # 20070313-15:45:17 (5594) Loading Sechedule: job_keyms2classmb_senderargssleep_time5 worker_method_argssleep_time5job_keyms2classmb_senderargssleep_time5trigger_typecron_triggertrigger_args0,10,20,30,40,50 * * * * * *worker_methoddo_work # 20070313-15:45:17 (5594) Loading Sechedule: job_keyrclassreceiverargssleep_time5 worker_method_argssleep_time5job_keyrclassreceiverargssleep_time5trigger_typecron_triggertrigger_args0,10,20,30,40,50 * * * * * *worker_methoddo_work # 20070313-15:45:20 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:20 +0100 2007, last=Tue Mar 13 15:45:20 +0100 2007> 20070313-15:45:21 (5594) Starting worker: mb_receiver mr (mb_receiver_mr) (sleep_time5) 20070313-15:45:21 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:20 +0100 2007, last=Tue Mar 13 15:45:20 +0100 2007> 20070313-15:45:21 (5594) Starting worker: sender s (sender_s) (sleep_time5) 20070313-15:45:21 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:20 +0100 2007, last=Tue Mar 13 15:45:20 +0100 2007> 20070313-15:45:22 (5594) Starting worker: mb_sender ms1 (mb_sender_ms1) (sleep_time5) 20070313-15:45:22 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:20 +0100 2007, last=Tue Mar 13 15:45:20 +0100 2007> 20070313-15:45:22 (5594) Starting worker: mb_sender ms2 (mb_sender_ms2) (sleep_time5) 20070313-15:45:22 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:20 +0100 2007, last=Tue Mar 13 15:45:20 +0100 2007> 20070313-15:45:22 (5594) Starting worker: receiver r (receiver_r) (sleep_time5) 20070313-15:45:30 (5594) Schedule triggered: # job=#, trigger=#, earliest=Tue Mar 13 15:45:30 +0100 2007, last=Tue Mar 13 15:45:30 +0100 2007> Any ideas? -- // jack // http://www.nuthole.com From david.balatero at gmail.com Tue Mar 13 18:50:09 2007 From: david.balatero at gmail.com (David Balatero) Date: Tue, 13 Mar 2007 15:50:09 -0700 Subject: [Backgroundrb-devel] Scheduled worker dies after about 30-45 runs Message-ID: <572c318d0703131550y44f162a8ofbbbb58e014d6d6a@mail.gmail.com> I have a scheduled worker that runs every minute, checking for new eBay auctions to post. If it finds an auction, then it posts it. If there are no auctions in the database that are queued up, it simply does nothing. I got the worker to finally run as a scheduled worker, every minute. However, after about 30-45 scheduled runs, it simply stops running. My backgroundrb_schedules.yml file: ------ ebay_runner: :class: :ebay_auction_worker :job_key: :ebay_auction_runner :trigger_type: :cron_trigger :trigger_args: 0 * * * * * * ------ My ebay_auction_worker.rb file: ------ class EbayAuctionWorker < BackgrounDRb::Rails # Set this worker to run every minute. attr_accessor :progress, :description def do_work(args) # This method is called in it's own new thread when you # call new worker. args is set to :args @progress = 0 @description = "Checking for eBay auctions and posting" logger.info("Checking to post an auction at #{Time.now.to_s}.") auction = EbayAuction.find(:first, :conditions => ["auction_status = ? AND post_on < ?", EbayAuction::STATUS_STRINGS[:queued], Time.now]) if auction logger.info("--- Posting auction: #{auction.title}") # This method only takes 1-3 seconds tops to run -- it just generates an call to eBay's XML API, and sends it along. auction.post else logger.info("--- No auctions currently need posting.") end @progress = 100 logger.info("--- Finished auction check.") # Exit the thread to cleanup. self.delete end end EbayAuctionWorker.register ------- >From the server logs, I'm getting this output: -------- 20070312-18:14:31 (24026) Starting BackgrounDRb Server 20070312-18:14:31 (24026) load_rails: true 20070312-18:14:31 (24026) uri: drbunix:///tmp/backgroundrbunix_localhost_22222 20070312-18:14:31 (24026) config: /home/david/redclay/config/backgroundrb.yml 20070312-18:14:31 (24026) temp_dir: /tmp 20070312-18:14:31 (24026) database_yml: config/database.yml 20070312-18:14:31 (24026) worker_dir: /home/david/redclay/lib/workers 20070312-18:14:31 (24026) protocol: drbunix 20070312-18:14:31 (24026) socket_dir: /tmp/backgroundrb.24026 20070312-18:14:31 (24026) environment: development 20070312-18:14:31 (24026) host: localhost 20070312-18:14:31 (24026) acl: denyallallowlocalhost 127.0.0.1orderdeny ,allow 20070312-18:14:31 (24026) rails_env: development 20070312-18:14:31 (24026) timer_sleep: 60 20070312-18:14:31 (24026) port: 22222 20070312-18:14:31 (24026) pool_size: 5 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerLogger backgroundrb_logger (backgroundrb_logger) () 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerResults backgroundrb_results (backgroundrb_results) () 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/redclay/lib/workers/variant_addition_worker.rb 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/redclay/lib/workers/variant_remove_worker.rb 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/redclay/lib/workers/variant_subtraction_worker.rb 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/redclay/lib/workers/variant_new_worker.rb 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/redclay/lib/workers/ebay_auction_worker.rb 20070312-18:14:31 (24028) Loading Sechedule: classebay_auction_workerargsjob_keyebay_auction_runner classebay_auction_workertrigger_args0 * * * * * *trigger_typecron_triggerjob_keyebay_auction_runner # 20070312-18:15:00 (24028) Schedule triggered: # job=#, trigger=#, earliest=Mon Mar 12 18:15:00 -0700 2007, last=Mon Mar 12 18:15:00 -0700 2007> 20070312-18:15:01 (24028) Starting worker: ebay_auction_worker ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () ....fast-forward 41 minutes..... 20070312-18:56:00 (24028) Starting worker: ebay_auction_worker ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () 20070312-18:57:00 (24028) Schedule triggered: # job=#, trigger=#, earliest=Mon Mar 12 18:57:00 -0700 2007, last=Mon Mar 12 18:57:00 -0700 2007> 20070312-18:57:00 (24028) failed to find slave socket - (RuntimeError) 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in `initialize' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `new' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:357:in `schedule_worker' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `call' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `run' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `each' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `run' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `loop' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `run' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:95:in `run' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `initialize' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `new' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `run' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb/middleman.rb:106:in `setup' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:306:in `run' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:186:in `call' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:186:in `start_proc' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in `call' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in `call_as_daemon' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:190:in `start_proc' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:226:in `start' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/controller.rb:69:in `run' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:182:in `run_proc' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in `call' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in `catch_exceptions' 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:181:in `run_proc' 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in `run' 20070312-18:57:00 (24028) script/backgroundrb:29 -------------------- It looks like it is dying because a Slave process has somehow killed itself in that timespan. Can I do something about this? I need this worker to be running all the time. Is there something I can do in my code to keep backgroundrb processes from dying? I'd like to figure out how we can resolve this, because it looks like it might be a much more serious problem in the backgroundrb framework. That said, I want to express my gratitude for this framework existing -- I wouldn't have been able to come this far without it. I hope we can figure this out! Thanks, David Balatero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070313/e4b743d9/attachment-0001.html From david.balatero at gmail.com Tue Mar 13 18:53:22 2007 From: david.balatero at gmail.com (David Balatero) Date: Tue, 13 Mar 2007 15:53:22 -0700 Subject: [Backgroundrb-devel] Scheduled worker dies after about 30-45 runs In-Reply-To: <572c318d0703131550y44f162a8ofbbbb58e014d6d6a@mail.gmail.com> References: <572c318d0703131550y44f162a8ofbbbb58e014d6d6a@mail.gmail.com> Message-ID: <572c318d0703131553t4ec8c23dn6f5bd71fbba5e8dc@mail.gmail.com> Also, I just realized I didn't provide any of my system information: I am running Ruby 1.8.5 on Gentoo Linux, with the following gem versions: - rails 1.2.2 - slave 1.2.0 - daemons 1.0.4 - lighttpd 1.3.16 (ssl) Thanks! David On 3/13/07, David Balatero wrote: > > I have a scheduled worker that runs every minute, checking for new eBay > auctions to post. If it finds an auction, then it posts it. If there are no > auctions in the database that are queued up, it simply does nothing. I got > the worker to finally run as a scheduled worker, every minute. However, > after about 30-45 scheduled runs, it simply stops running. > > My backgroundrb_schedules.yml file: > ------ > ebay_runner: > :class: :ebay_auction_worker > :job_key: :ebay_auction_runner > :trigger_type: :cron_trigger > :trigger_args: 0 * * * * * * > ------ > > My ebay_auction_worker.rb file: > ------ > class EbayAuctionWorker < BackgrounDRb::Rails > # Set this worker to run every minute. > attr_accessor :progress, :description > > def do_work(args) > # This method is called in it's own new thread when you > # call new worker. args is set to :args > > @progress = 0 > @description = "Checking for eBay auctions and posting" > > logger.info("Checking to post an auction at #{Time.now.to_s }.") > auction = EbayAuction.find(:first, :conditions => ["auction_status = ? > AND post_on < ?", EbayAuction::STATUS_STRINGS[:queued], Time.now]) > if auction > logger.info("--- Posting auction: #{ auction.title}") > > # This method only takes 1-3 seconds tops to run -- it just > generates an call to eBay's XML API, and sends it along. > auction.post > else > logger.info ("--- No auctions currently need posting.") > end > > @progress = 100 > > logger.info("--- Finished auction check.") > > # Exit the thread to cleanup. > self.delete > end > end > > EbayAuctionWorker.register > ------- > > > From the server logs, I'm getting this output: > -------- > 20070312-18:14:31 (24026) Starting BackgrounDRb Server > 20070312-18:14:31 (24026) load_rails: true > 20070312-18:14:31 (24026) uri: > drbunix:///tmp/backgroundrbunix_localhost_22222 > 20070312-18:14:31 (24026) config: /home/david/redclay/config/backgroundrb > .yml > 20070312-18:14:31 (24026) temp_dir: /tmp > 20070312-18:14:31 (24026) database_yml: config/database.yml > 20070312-18:14:31 (24026) worker_dir: /home/david/redclay/lib/workers > 20070312-18:14:31 (24026) protocol: drbunix > 20070312-18:14:31 (24026) socket_dir: /tmp/backgroundrb.24026 > 20070312-18:14:31 (24026) environment: development > 20070312-18:14:31 (24026) host: localhost > 20070312-18:14:31 (24026) acl: denyallallowlocalhost 127.0.0.1orderdeny > ,allow > 20070312-18:14:31 (24026) rails_env: development > 20070312-18:14:31 (24026) timer_sleep: 60 > 20070312-18:14:31 (24026) port: 22222 > 20070312-18:14:31 (24026) pool_size: 5 > 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerLogger > backgroundrb_logger (backgroundrb_logger) () > 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerResults > backgroundrb_results (backgroundrb_results) () > 20070312-18:14:31 (24028) Loading Worker Class File: > /home/david/redclay/lib/workers/variant_addition_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: > /home/david/redclay/lib/workers/variant_remove_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: > /home/david/redclay/lib/workers/variant_subtraction_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: > /home/david/redclay/lib/workers/variant_new_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: > /home/david/redclay/lib/workers/ebay_auction_worker.rb > 20070312-18:14:31 (24028) Loading Sechedule: > classebay_auction_workerargsjob_keyebay_auction_runner > classebay_auction_workertrigger_args0 * * * * * > *trigger_typecron_triggerjob_keyebay_auction_runner # > > 20070312-18:15:00 (24028) Schedule triggered: # > job=# /server/lib/backgroundrb/middleman.rb:355>, trigger=# @hour=0..23, @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:15:00 -0700 > 2007, last=Mon Mar 12 18:15:00 -0700 2007> > 20070312-18:15:01 (24028) Starting worker: ebay_auction_worker > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > > ....fast-forward 41 minutes..... > > 20070312-18:56:00 (24028) Starting worker: ebay_auction_worker > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > 20070312-18:57:00 (24028) Schedule triggered: # > job=# /server/lib/backgroundrb/middleman.rb:355>, trigger=# @hour=0..23, @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:57:00 -0700 > 2007, last=Mon Mar 12 18:57:00 -0700 2007> > 20070312-18:57:00 (24028) failed to find slave socket - (RuntimeError) > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in > `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/middleman.rb:210:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/middleman.rb:210:in `new_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/thread_pool.rb:36:in `dispatch' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/thread_pool.rb:22:in `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/thread_pool.rb:22:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/thread_pool.rb:22:in `dispatch' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/middleman.rb:199:in `new_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/middleman.rb:357:in `schedule_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:46:in `call' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:46:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:40:in `each' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:40:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:35:in `loop' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:35:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:95:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:94:in `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:94:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/scheduler.rb:94:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb > /server/lib/backgroundrb/middleman.rb:106:in `setup' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:306:in > `run' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:186:in > `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:186:in > `start_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in > `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in > `call_as_daemon' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:190:in > `start_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:226:in > `start' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/controller.rb:69:in > `run' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:182:in > `run_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in > `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in > `catch_exceptions' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:181:in > `run_proc' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/backgroundrb/server/lib/backgroundrb_server.rb:301:in > `run' > 20070312-18:57:00 (24028) script/backgroundrb:29 > -------------------- > > > It looks like it is dying because a Slave process has somehow killed > itself in that timespan. Can I do something about this? I need this worker > to be running all the time. Is there something I can do in my code to keep > backgroundrb processes from dying? I'd like to figure out how we can > resolve this, because it looks like it might be a much more serious problem > in the backgroundrb framework. That said, I want to express my gratitude > for this framework existing -- I wouldn't have been able to come this far > without it. I hope we can figure this out! > > Thanks, > David Balatero > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070313/7d9f1ab5/attachment.html From blucasco at gmail.com Tue Mar 13 18:57:09 2007 From: blucasco at gmail.com (Brian Lucas) Date: Tue, 13 Mar 2007 16:57:09 -0600 Subject: [Backgroundrb-devel] worker starting twice In-Reply-To: Message-ID: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> Hi Jack, I'm experiencing the same problem and wrote about it here earlier. I wonder if it has something to do with thread_pool size or concurrent threads running in brb? Brian ------------- Date: Tue, 13 Mar 2007 16:06:58 +0100 From: "Jack Nutting" Subject: [Backgroundrb-devel] worker starting twice To: backgroundrb-devel at rubyforge.org Message-ID: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi gang, Thanks to other help I've gotten on this list, I've got backgroundrb up and running using postgres alongside my rails app. I've designed the system to have several eternally-running workers that periodically poll the database to see if there's any work for them to do, and set things up in the config file to make them restart quickly if any of them should crash. One odd thing I've noticed is that one of my workers, with the key "mr", seems to start twice (without the first one having completed). I've got some logging that occurs when each starts: 20070313-15:45:17 (5594) Starting WorkerLogger 20070313-15:45:17 (5595) In ResultsWorker 20070313-15:45:21 (5595) MbReceiver mr started. 20070313-15:45:21 (5595) Sender s started 20070313-15:45:22 (5595) MbSender ms1 started 20070313-15:45:22 (5595) MbSender ms2 started 20070313-15:45:22 (5595) Receiver r started 20070313-15:45:30 (5595) MbReceiver mr started. From david.balatero at gmail.com Tue Mar 13 19:03:22 2007 From: david.balatero at gmail.com (David Balatero) Date: Tue, 13 Mar 2007 16:03:22 -0700 Subject: [Backgroundrb-devel] Scheduled worker dies after about 30-45 runs In-Reply-To: <572c318d0703131553t4ec8c23dn6f5bd71fbba5e8dc@mail.gmail.com> References: <572c318d0703131550y44f162a8ofbbbb58e014d6d6a@mail.gmail.com> <572c318d0703131553t4ec8c23dn6f5bd71fbba5e8dc@mail.gmail.com> Message-ID: <572c318d0703131603g69e11005g9b8b5c492341939b@mail.gmail.com> http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/000755.html Is this post relevant to the error I am seeing? Is it just that I can't spawn a worker every minute and expect it to stay sane? On 3/13/07, David Balatero wrote: > > Also, I just realized I didn't provide any of my system information: > > I am running Ruby 1.8.5 on Gentoo Linux, with the following gem versions: > - rails 1.2.2 > - slave 1.2.0 > - daemons 1.0.4 > - lighttpd 1.3.16 (ssl) > > Thanks! > David > > On 3/13/07, David Balatero wrote: > > > > I have a scheduled worker that runs every minute, checking for new eBay > > auctions to post. If it finds an auction, then it posts it. If there are no > > auctions in the database that are queued up, it simply does nothing. I got > > the worker to finally run as a scheduled worker, every minute. However, > > after about 30-45 scheduled runs, it simply stops running. > > > > My backgroundrb_schedules.yml file: > > ------ > > ebay_runner: > > :class: :ebay_auction_worker > > :job_key: :ebay_auction_runner > > :trigger_type: :cron_trigger > > :trigger_args: 0 * * * * * * > > ------ > > > > My ebay_auction_worker.rb file: > > ------ > > class EbayAuctionWorker < BackgrounDRb::Rails > > # Set this worker to run every minute. > > attr_accessor :progress, :description > > > > def do_work(args) > > # This method is called in it's own new thread when you > > # call new worker. args is set to :args > > > > @progress = 0 > > @description = "Checking for eBay auctions and posting" > > > > logger.info("Checking to post an auction at #{Time.now.to_s }.") > > auction = EbayAuction.find(:first, :conditions => ["auction_status = > > ? AND post_on < ?", EbayAuction::STATUS_STRINGS[:queued], Time.now]) > > if auction > > logger.info("--- Posting auction: #{ auction.title}") > > > > # This method only takes 1-3 seconds tops to run -- it just > > generates an call to eBay's XML API, and sends it along. > > auction.post > > else > > logger.info ("--- No auctions currently need posting.") > > end > > > > @progress = 100 > > > > logger.info("--- Finished auction check.") > > > > # Exit the thread to cleanup. > > self.delete > > end > > end > > > > EbayAuctionWorker.register > > ------- > > > > > > From the server logs, I'm getting this output: > > -------- > > 20070312-18:14:31 (24026) Starting BackgrounDRb Server > > 20070312-18:14:31 (24026) load_rails: true > > 20070312-18:14:31 (24026) uri: > > drbunix:///tmp/backgroundrbunix_localhost_22222 > > 20070312-18:14:31 (24026) config: /home/david/redclay/config/ > > backgroundrb.yml > > 20070312-18:14:31 (24026) temp_dir: /tmp > > 20070312-18:14:31 (24026) database_yml: config/database.yml > > 20070312-18:14:31 (24026) worker_dir: /home/david/redclay/lib/workers > > 20070312-18:14:31 (24026) protocol: drbunix > > 20070312-18:14:31 (24026) socket_dir: /tmp/backgroundrb .24026 > > 20070312-18:14:31 (24026) environment: development > > 20070312-18:14:31 (24026) host: localhost > > 20070312-18:14:31 (24026) acl: denyallallowlocalhost 127.0.0.1orderdeny > > ,allow > > 20070312-18:14:31 (24026) rails_env: development > > 20070312-18:14:31 (24026) timer_sleep: 60 > > 20070312-18:14:31 (24026) port: 22222 > > 20070312-18:14:31 (24026) pool_size: 5 > > 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerLogger > > backgroundrb_logger (backgroundrb_logger) () > > 20070312-18:14:31 (24028) Starting worker: BackgrounDRb::Worker::WorkerResults > > backgroundrb_results (backgroundrb_results) () > > 20070312-18:14:31 (24028) Loading Worker Class File: > > /home/david/redclay/lib/workers/variant_addition_worker.rb > > 20070312-18:14:31 (24028) Loading Worker Class File: > > /home/david/redclay/lib/workers/variant_remove_worker.rb > > 20070312-18:14:31 (24028) Loading Worker Class File: > > /home/david/redclay/lib/workers/variant_subtraction_worker.rb > > 20070312-18:14:31 (24028) Loading Worker Class File: > > /home/david/redclay/lib/workers/variant_new_worker.rb > > 20070312-18:14:31 (24028) Loading Worker Class File: > > /home/david/redclay/lib/workers/ebay_auction_worker.rb > > 20070312-18:14:31 (24028) Loading Sechedule: > > classebay_auction_workerargsjob_keyebay_auction_runner > > classebay_auction_workertrigger_args0 * * * * * > > *trigger_typecron_triggerjob_keyebay_auction_runner #< BackgrounDRb::CronTrigger:0xb73ae12c> > > > > 20070312-18:15:00 (24028) Schedule triggered: # > # > > job=# > /server/lib/ backgroundrb/middleman.rb:355>, trigger=# > @hour=0..23, @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:15:00 -0700 > > 2007, last=Mon Mar 12 18:15:00 -0700 2007> > > 20070312-18:15:01 (24028) Starting worker: ebay_auction_worker > > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > > > > ....fast-forward 41 minutes..... > > > > 20070312-18:56:00 (24028) Starting worker: ebay_auction_worker > > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > > 20070312-18:57:00 (24028) Schedule triggered: # > # > > job=# > /server/lib/ backgroundrb/middleman.rb:355>, trigger=# > @hour=0..23, @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:57:00 -0700 > > 2007, last=Mon Mar 12 18:57:00 -0700 2007> > > 20070312-18:57:00 (24028) failed to find slave socket - (RuntimeError) > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/lib/slave.rb:435:in > > `initialize' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `new' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/middleman.rb:357:in > > `schedule_worker' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `call' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `run' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `each' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `run' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `loop' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `run' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:95:in `run' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `initialize' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `new' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `run' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb/middleman.rb:106:in `setup' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb_server.rb:306:in `run' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons- 1.0.4/lib/daemons/application.rb:186:in > > `call' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:186:in > > `start_proc' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in > > `call' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/daemonize.rb:192:in > > `call_as_daemon' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:190:in > > `start_proc' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/application.rb:226:in > > `start' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/controller.rb:69:in > > `run' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:182:in > > `run_proc' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in > > `call' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons/cmdline.rb:94:in > > `catch_exceptions' > > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/lib/daemons.rb:181:in > > `run_proc' > > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > > backgroundrb/server/lib/backgroundrb_server.rb:301:in `run' > > 20070312-18:57:00 (24028) script/ backgroundrb:29 > > -------------------- > > > > > > It looks like it is dying because a Slave process has somehow killed > > itself in that timespan. Can I do something about this? I need this worker > > to be running all the time. Is there something I can do in my code to keep > > backgroundrb processes from dying? I'd like to figure out how we can > > resolve this, because it looks like it might be a much more serious problem > > in the backgroundrb framework. That said, I want to express my gratitude > > for this framework existing -- I wouldn't have been able to come this far > > without it. I hope we can figure this out! > > > > Thanks, > > David Balatero > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070313/6c8a289f/attachment-0001.html From ezmobius at gmail.com Tue Mar 13 19:22:30 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 13 Mar 2007 16:22:30 -0700 Subject: [Backgroundrb-devel] Scheduled worker dies after about 30-45 runs In-Reply-To: <572c318d0703131603g69e11005g9b8b5c492341939b@mail.gmail.com> References: <572c318d0703131550y44f162a8ofbbbb58e014d6d6a@mail.gmail.com> <572c318d0703131553t4ec8c23dn6f5bd71fbba5e8dc@mail.gmail.com> <572c318d0703131603g69e11005g9b8b5c492341939b@mail.gmail.com> Message-ID: <5B603862-3626-4F7A-933F-47EE36BBDD48@brainspl.at> David- For the kind of work you are trying to do you are much better of spawning one worker when the server boots and having it run in a loop, living forever. Cheers- -Ezra On Mar 13, 2007, at 4:03 PM, David Balatero wrote: > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > 000755.html > > Is this post relevant to the error I am seeing? Is it just that I > can't spawn a worker every minute and expect it to stay sane? > > On 3/13/07, David Balatero wrote: Also, > I just realized I didn't provide any of my system information: > > I am running Ruby 1.8.5 on Gentoo Linux, with the following gem > versions: > - rails 1.2.2 > - slave 1.2.0 > - daemons 1.0.4 > - lighttpd 1.3.16 (ssl) > > Thanks! > David > > > On 3/13/07, David Balatero < david.balatero at gmail.com> wrote: I > have a scheduled worker that runs every minute, checking for new > eBay auctions to post. If it finds an auction, then it posts it. If > there are no auctions in the database that are queued up, it simply > does nothing. I got the worker to finally run as a scheduled > worker, every minute. However, after about 30-45 scheduled runs, it > simply stops running. > > My backgroundrb_schedules.yml file: > ------ > ebay_runner: > :class: :ebay_auction_worker > :job_key: :ebay_auction_runner > :trigger_type: :cron_trigger > :trigger_args: 0 * * * * * * > ------ > > My ebay_auction_worker.rb file: > ------ > class EbayAuctionWorker < BackgrounDRb::Rails > # Set this worker to run every minute. > attr_accessor :progress, :description > > def do_work(args) > # This method is called in it's own new thread when you > # call new worker. args is set to :args > > @progress = 0 > @description = "Checking for eBay auctions and posting" > > logger.info("Checking to post an auction at #{Time.now.to_s }.") > auction = EbayAuction.find(:first, :conditions => > ["auction_status = ? AND post_on < ?", EbayAuction::STATUS_STRINGS > [:queued], Time.now]) > if auction > logger.info("--- Posting auction: #{ auction.title}") > > # This method only takes 1-3 seconds tops to run -- it just > generates an call to eBay's XML API, and sends it along. > auction.post > else > logger.info ("--- No auctions currently need posting.") > end > > @progress = 100 > > logger.info("--- Finished auction check.") > > # Exit the thread to cleanup. > self.delete > end > end > > EbayAuctionWorker.register > ------- > > > From the server logs, I'm getting this output: > -------- > 20070312-18:14:31 (24026) Starting BackgrounDRb Server > 20070312-18:14:31 (24026) load_rails: true > 20070312-18:14:31 (24026) uri: drbunix:///tmp/ > backgroundrbunix_localhost_22222 > 20070312-18:14:31 (24026) config: /home/david/redclay/config/ > backgroundrb.yml > 20070312-18:14:31 (24026) temp_dir: /tmp > 20070312-18:14:31 (24026) database_yml: config/database.yml > 20070312-18:14:31 (24026) worker_dir: /home/david/redclay/lib/workers > 20070312-18:14:31 (24026) protocol: drbunix > 20070312-18:14:31 (24026) socket_dir: /tmp/backgroundrb .24026 > 20070312-18:14:31 (24026) environment: development > 20070312-18:14:31 (24026) host: localhost > 20070312-18:14:31 (24026) acl: denyallallowlocalhost > 127.0.0.1orderdeny,allow > 20070312-18:14:31 (24026) rails_env: development > 20070312-18:14:31 (24026) timer_sleep: 60 > 20070312-18:14:31 (24026) port: 22222 > 20070312-18:14:31 (24026) pool_size: 5 > 20070312-18:14:31 (24028) Starting worker: > BackgrounDRb::Worker::WorkerLogger backgroundrb_logger > (backgroundrb_logger) () > 20070312-18:14:31 (24028) Starting worker: > BackgrounDRb::Worker::WorkerResults backgroundrb_results > (backgroundrb_results) () > 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/ > redclay/lib/workers/variant_addition_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/ > redclay/lib/workers/variant_remove_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/ > redclay/lib/workers/variant_subtraction_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/ > redclay/lib/workers/variant_new_worker.rb > 20070312-18:14:31 (24028) Loading Worker Class File: /home/david/ > redclay/lib/workers/ebay_auction_worker.rb > 20070312-18:14:31 (24028) Loading Sechedule: > classebay_auction_workerargsjob_keyebay_auction_runner > classebay_auction_workertrigger_args0 * * * * * > *trigger_typecron_triggerjob_keyebay_auction_runner #< > BackgrounDRb::CronTrigger:0xb73ae12c> > 20070312-18:15:00 (24028) Schedule triggered: # 0xb7956b38> job=# plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:355>, > trigger=# @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:15:00 > -0700 2007, last=Mon Mar 12 18:15:00 -0700 2007> > 20070312-18:15:01 (24028) Starting worker: ebay_auction_worker > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > > ....fast-forward 41 minutes..... > > 20070312-18:56:00 (24028) Starting worker: ebay_auction_worker > ebay_auction_runner (ebay_auction_worker_ebay_auction_runner) () > 20070312-18:57:00 (24028) Schedule triggered: # 0xb7956b38> job=# plugins/backgroundrb/server/lib/ backgroundrb/middleman.rb:355>, > trigger=# @year=nil, @day=1..31, @cron_expr="0 * * * * * *", @sec=[0], > @wday=0..6, @min=0..59, @month=1..12>, earliest=Mon Mar 12 18:57:00 > -0700 2007, last=Mon Mar 12 18:57:00 -0700 2007> > 20070312-18:57:00 (24028) failed to find slave socket - (RuntimeError) > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/slave-1.2.0/ > lib/slave.rb:435:in `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:210:in `new_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/thread_pool.rb:36:in `dispatch' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/thread_pool.rb:22:in `dispatch' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:199:in `new_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:357:in > `schedule_worker' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `call' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:46:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `each' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:40:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `loop' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:35:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:95:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `initialize' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `new' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/scheduler.rb:94:in `run' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb/middleman.rb:106:in `setup' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb_server.rb:306:in `run' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons- > 1.0.4/lib/daemons/application.rb:186:in `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:186:in `start_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/daemonize.rb:192:in `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/daemonize.rb:192:in `call_as_daemon' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:190:in `start_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/application.rb:226:in `start' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/controller.rb:69:in `run' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons.rb:182:in `run_proc' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/cmdline.rb:94:in `call' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons/cmdline.rb:94:in `catch_exceptions' > 20070312-18:57:00 (24028) /usr/lib/ruby/gems/1.8/gems/daemons-1.0.4/ > lib/daemons.rb:181:in `run_proc' > 20070312-18:57:00 (24028) /home/david/redclay/vendor/plugins/ > backgroundrb/server/lib/backgroundrb_server.rb:301:in `run' > 20070312-18:57:00 (24028) script/ backgroundrb:29 > -------------------- > > > It looks like it is dying because a Slave process has somehow > killed itself in that timespan. Can I do something about this? I > need this worker to be running all the time. Is there something I > can do in my code to keep backgroundrb processes from dying? I'd > like to figure out how we can resolve this, because it looks like > it might be a much more serious problem in the backgroundrb > framework. That said, I want to express my gratitude for this > framework existing -- I wouldn't have been able to come this far > without it. I hope we can figure this out! > > Thanks, > David Balatero > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From jnutting at gmail.com Wed Mar 14 10:21:06 2007 From: jnutting at gmail.com (Jack Nutting) Date: Wed, 14 Mar 2007 15:21:06 +0100 Subject: [Backgroundrb-devel] worker starting twice In-Reply-To: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> References: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> Message-ID: On 3/13/07, Brian Lucas wrote: > Hi Jack, I'm experiencing the same problem and wrote about it here earlier. > I wonder if it has something to do with thread_pool size or concurrent > threads running in brb? I don't know, but I know it's really irritating. I think I have a few clues for whoever feels up to looking at this. First of all, the doubled worker is not actually running in two processes. 'ps' shows me only the processes i've specified in the schedules file, not more. Instead, the trigger method is being called twice. Through logging output, I've observed that my loop is in fact running twice simultaneously, presumably from two different threads. Also, I've found a way to determine which of my processes will be doubled: whichever top-level job entry name (not the job_key, but the top-level identifier for a worker) comes alphabetically first, that's the job that I start getting double-invocations on! e.g. if I have a job like this: zzzzz: :class: :my_class :job_key: :my_job1 ... that won't won't be doubled up (some other worker will instead). However, if I change the 'zzzzz' to 'a', this worker will be double-invoked. In the short term, this gives me a workaround since I can have a dummy worker called 'a' that just sleeps, but this is still consuming excess RAM of course. -- // jack // http://www.nuthole.com From masonhale at gmail.com Wed Mar 14 11:49:08 2007 From: masonhale at gmail.com (Mason Hale) Date: Wed, 14 Mar 2007 10:49:08 -0500 Subject: [Backgroundrb-devel] worker starting twice In-Reply-To: References: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> Message-ID: <8bca3aa10703140849i6d54f339g27a1e42915ed06d3@mail.gmail.com> My understanding is that for scheduled workers, the do_work method is called automatically when the worker is created (which provides an opportunity for any initialization, if necessary), and subsequently the :worker_method specified in the backgroundrb_schedules.yml file is called on the specified schedule. If your worker_method is specified as :do_work, then your :do_work method will be called twice, once when the worker is created, and again when the schedule is invoked. Note that these separate calls can occur in separate threads. So if you do use the call to do_work at creation time to do some initialization, you will need to set a flag internally to let other (potentially simultaneous) calls to your :worker_method behave appropriately if called be initialization is complete. Here's the relevant excerpt from the documentation: > Special case: :do_work > > The do_work method of a worker is automatically called when you first > create a worker and we are treating it differently than other methods when > you schedule. > > You can have do_work executed repeatedly in a schedule by specifying > :worker_method => :do_work in schedule_worker. What will happen, is that on > the initial schedule where a worker is created, it will call the built in > call to do_work, then on subsequent trigger events, use the :worker_method. > I believe the documentation should be corrected -- the "on subsequent trigger events" part is not exactly accurate, as the trigger event could be simultaneous, and hence not "subsequent". Mason On 3/14/07, Jack Nutting wrote: > > On 3/13/07, Brian Lucas wrote: > > Hi Jack, I'm experiencing the same problem and wrote about it here > earlier. > > I wonder if it has something to do with thread_pool size or concurrent > > threads running in brb? > > I don't know, but I know it's really irritating. > > I think I have a few clues for whoever feels up to looking at this. > > First of all, the doubled worker is not actually running in two > processes. 'ps' shows me only the processes i've specified in the > schedules file, not more. Instead, the trigger method is being called > twice. Through logging output, I've observed that my loop is in fact > running twice simultaneously, presumably from two different threads. > > Also, I've found a way to determine which of my processes will be > doubled: whichever top-level job entry name (not the job_key, but the > top-level identifier for a worker) comes alphabetically first, that's > the job that I start getting double-invocations on! e.g. if I have a > job like this: > > zzzzz: > :class: :my_class > :job_key: :my_job1 > ... > > that won't won't be doubled up (some other worker will instead). > However, if I change the 'zzzzz' to 'a', this worker will be > double-invoked. In the short term, this gives me a workaround since I > can have a dummy worker called 'a' that just sleeps, but this is still > consuming excess RAM of course. > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070314/562bff13/attachment.html From jnutting at gmail.com Wed Mar 14 17:37:46 2007 From: jnutting at gmail.com (Jack Nutting) Date: Wed, 14 Mar 2007 22:37:46 +0100 Subject: [Backgroundrb-devel] worker starting twice In-Reply-To: <8bca3aa10703140849i6d54f339g27a1e42915ed06d3@mail.gmail.com> References: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> <8bca3aa10703140849i6d54f339g27a1e42915ed06d3@mail.gmail.com> Message-ID: On 3/14/07, Mason Hale wrote: > My understanding is that for scheduled workers, the do_work method is called > automatically when the worker is created (which provides an opportunity for > any initialization, if necessary), and subsequently the :worker_method > specified in the backgroundrb_schedules.yml file is called on the specified > schedule. > > If your worker_method is specified as :do_work, then your :do_work method > will be called twice, once when the worker is created, and again when the > schedule is invoked. Note that these separate calls can occur in separate > threads. So if you do use the call to do_work at creation time to do some > initialization, you will need to set a flag internally to let other > (potentially simultaneous) calls to your :worker_method behave appropriately > if called be initialization is complete. Ahhhh! Now it all makes sense. Thanks! -- // jack // http://www.nuthole.com From skaar at waste.org Wed Mar 14 19:58:26 2007 From: skaar at waste.org (skaar) Date: Wed, 14 Mar 2007 18:58:26 -0500 Subject: [Backgroundrb-devel] worker starting twice In-Reply-To: References: <45f72c1b.57f94f45.5b06.ffffa499@mx.google.com> <8bca3aa10703140849i6d54f339g27a1e42915ed06d3@mail.gmail.com> Message-ID: <20070314235826.GA5282@waste.org> * Jack Nutting (jnutting at gmail.com) [070314 16:26]: > On 3/14/07, Mason Hale wrote: > > My understanding is that for scheduled workers, the do_work method is called > > automatically when the worker is created (which provides an opportunity for > > any initialization, if necessary), and subsequently the :worker_method > > specified in the backgroundrb_schedules.yml file is called on the specified > > schedule. > > > > If your worker_method is specified as :do_work, then your :do_work method > > will be called twice, once when the worker is created, and again when the > > schedule is invoked. Note that these separate calls can occur in separate > > threads. So if you do use the call to do_work at creation time to do some > > initialization, you will need to set a flag internally to let other > > (potentially simultaneous) calls to your :worker_method behave appropriately > > if called be initialization is complete. there is some safeguards in the code against this. If a worker is created, then it will keep track of the fact that it's the initial do_work and _not_ call do_work a second time on the initial schedule. /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From tsailipu at gmail.com Thu Mar 15 18:44:42 2007 From: tsailipu at gmail.com (Philip Lipu Tsai) Date: Thu, 15 Mar 2007 15:44:42 -0700 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> Message-ID: Erza, I am on Centos and bash, Rails 1.2.2, Ruby 1.8.4, BackgrounDrb old version (the 0.1.x version right before 0.2... Is the depend() section peculiar only to Gentoo? If not, anything one needs to do for this? I wrote a script just similar to this (w/o paying attention to the depend() section) and have capistrano before_restart task invoke the "/etc/init.d/backgroundrbd restart" (which is just stop and then start). However, this problem of backgroundrb's exiting after cap task's successful completion occurs (yes, no hanging in cap's process, but backgroundrb quits even though it is started in -d mode; the pid log file is there, but the process is no longer). Prepending with "nohup" in start seems to work, but I thought by using init.d style, one needn't worry about the nohup problem, either. Any insight? (Or is this a 0.1.x problem?) Thanks, Philip On 3/12/07, Ezra Zygmuntowicz wrote: > > > Yeah there are some weird interactions between capistrano and some > daemons, the old version of backgroundrb exibits this behavior. That > way I get around it is by creating an init.d script, then letting > capuse that to restart bdrb. Works like a charm. Here is an init.d > script for Gentoo systems, it should be easy to adapt to any other > unixy system.: > > #!/sbin/runscript > # Copyright 1999-2004 Gentoo Foundation > # Distributed under the terms of the GNU General Public License v2 > # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v > 1.1 2006/07/04 16:58:38 voxus Exp $ > > > depend() { > need net > use dns > after gfs > } > start() { > ebegin "Starting bdrb2" > RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ > start -d > eend $? "Failed to start bdrb2" > } > > stop() { > ebegin "Stopping bdrb2" > kill -9 `cat /data/toolbox/current/log/ > backgroundrb.pid` 2>/dev/null; true > eend $? "Failed to stop bdrb2" > } > > > > Cheers- > -Ezra > > > On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > > > Hello ~ > > > > I am using the Capistrano recipe outlined in this post to restart > > my backgroundrb server: > > > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > > 000718.html > > > > My Version: > > ~~~~~~~~~~~~ > > task :before_restart, :roles => :app do > > restart_backgroundrb > > end > > > > task :start_backgroundrb, :roles => :app do > > sudo "nohup /var/www/rails/#{application}/current/script/ > > backgroundrb/start " > > end > > > > task :stop_backgroundrb, :roles => :app do > > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > > backgroundrb.pid` 2>/dev/null; true" > > end > > > > task :restart_backgroundrb, :roles => :app do > > stop_backgroundrb > > start_backgroundrb > > end > > ~~~~~~~~~~~~ > > > > The backgroundrb server is stopped and started but hangs at the > > start_backgroundrb task. Capistrano shows: > > > > * executing task start_backgroundrb > > * executing "sudo nohup /var/www/rails/*************/current/ > > script/backgroundrb/start" > > servers: ["host"] > > [host] executing command > > ** [out :: host] nohup: ignoring input and appending output to > > `nohup.out' > > > > It will hang there and not restart the main web app process. I > > updated my gems to see if this was a version issue, but am > > experiencing the same problem. I also tried running this > > task :after_restart, the web app is successfully restarted, but it > > still hangs. > > > > Thanks in advance for any ideas on how to fix this. > > > > ~ Ben > > > > -- > > Ben Reubenstein > > 303-947-0446 > > http://www.benr75.com > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070315/34996eb2/attachment.html From benr at x-cr.com Fri Mar 16 12:33:28 2007 From: benr at x-cr.com (Ben Reubenstein) Date: Fri, 16 Mar 2007 10:33:28 -0600 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> Message-ID: <44c79a160703160933s621fd293o737057a4f34445b0@mail.gmail.com> Hi Philip ~ I am having this issue too and have been meaning to post. I am using the init script on Gentoo, and it works perfectly if I am in the machine but when running from capistrano it will report a successful start/stop but the process is nowhere to be found. I have to login to the machine and start it manually. I am running on the latest version so I don't think it is a backgroundrb version issue. ~ Ben On 3/15/07, Philip Lipu Tsai wrote: > > Erza, > > I am on Centos and bash, Rails 1.2.2, Ruby 1.8.4, BackgrounDrb old version > (the 0.1.x version right before 0.2... Is the depend() section peculiar > only to Gentoo? If not, anything one needs to do for this? > > I wrote a script just similar to this (w/o paying attention to the > depend() section) and have capistrano before_restart task invoke the > "/etc/init.d/backgroundrbd restart" (which is just stop and then start). > However, this problem of backgroundrb's exiting after cap task's successful > completion occurs (yes, no hanging in cap's process, but backgroundrb quits > even though it is started in -d mode; the pid log file is there, but the > process is no longer). Prepending with "nohup" in start seems to work, but > I thought by using init.d style, one needn't worry about the nohup > problem, either. Any insight? (Or is this a 0.1.x problem?) > > Thanks, > Philip > > On 3/12/07, Ezra Zygmuntowicz wrote: > > > > > > Yeah there are some weird interactions between capistrano and > > some > > daemons, the old version of backgroundrb exibits this behavior. That > > way I get around it is by creating an init.d script, then letting > > capuse that to restart bdrb. Works like a charm. Here is an init.d > > script for Gentoo systems, it should be easy to adapt to any other > > unixy system.: > > > > #!/sbin/runscript > > # Copyright 1999-2004 Gentoo Foundation > > # Distributed under the terms of the GNU General Public License v2 > > # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v > > 1.1 2006/07/04 16:58:38 voxus Exp $ > > > > > > depend() { > > need net > > use dns > > after gfs > > } > > start() { > > ebegin "Starting bdrb2" > > RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ > > start -d > > eend $? "Failed to start bdrb2" > > } > > > > stop() { > > ebegin "Stopping bdrb2" > > kill -9 `cat /data/toolbox/current/log/ > > backgroundrb.pid` 2>/dev/null; true > > eend $? "Failed to stop bdrb2" > > } > > > > > > > > Cheers- > > -Ezra > > > > > > On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > > > > > Hello ~ > > > > > > I am using the Capistrano recipe outlined in this post to restart > > > my backgroundrb server: > > > > > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > > > 000718.html > > > > > > My Version: > > > ~~~~~~~~~~~~ > > > task :before_restart, :roles => :app do > > > restart_backgroundrb > > > end > > > > > > task :start_backgroundrb, :roles => :app do > > > sudo "nohup /var/www/rails/#{application}/current/script/ > > > backgroundrb/start " > > > end > > > > > > task :stop_backgroundrb, :roles => :app do > > > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > > > backgroundrb.pid` 2>/dev/null; true" > > > end > > > > > > task :restart_backgroundrb, :roles => :app do > > > stop_backgroundrb > > > start_backgroundrb > > > end > > > ~~~~~~~~~~~~ > > > > > > The backgroundrb server is stopped and started but hangs at the > > > start_backgroundrb task. Capistrano shows: > > > > > > * executing task start_backgroundrb > > > * executing "sudo nohup /var/www/rails/*************/current/ > > > script/backgroundrb/start" > > > servers: ["host"] > > > [host] executing command > > > ** [out :: host] nohup: ignoring input and appending output to > > > `nohup.out' > > > > > > It will hang there and not restart the main web app process. I > > > updated my gems to see if this was a version issue, but am > > > experiencing the same problem. I also tried running this > > > task :after_restart, the web app is successfully restarted, but it > > > still hangs. > > > > > > Thanks in advance for any ideas on how to fix this. > > > > > > ~ Ben > > > > > > -- > > > Ben Reubenstein > > > 303-947-0446 > > > http://www.benr75.com > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- Ezra Zygmuntowicz > > -- Lead Rails Evangelist > > -- ez at engineyard.com > > -- Engine Yard, Serious Rails Hosting > > -- (866) 518-YARD (9273) > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Ben Reubenstein 303-947-0446 http://www.benr75.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070316/e7ad6bf3/attachment.html From tsailipu at gmail.com Fri Mar 16 12:58:00 2007 From: tsailipu at gmail.com (Philip Lipu Tsai) Date: Fri, 16 Mar 2007 08:58:00 -0800 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: <44c79a160703160933s621fd293o737057a4f34445b0@mail.gmail.com> References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> <44c79a160703160933s621fd293o737057a4f34445b0@mail.gmail.com> Message-ID: Hello Ben, Erza and you are on Gentoo -- so I thought Erza's script would work on Gentoo. Looks like your try didn't work there, either(?) Because I am on Centos, based on RedHat, I thought there may be some minor differences (the only difference being the depend section that I don't know how to translate.. well, unless I spend more time looking into it.). So I tried putting nohup as such: "RAILS_ENV=production nohup /etc/init.d/backgroundrbd start -d" and it worked. Though it worked, this combination of nohup inside init.d looked a bit unusual (btw, I don't know if setting this RAILS_ENV this way is still necessary for 0.2.x -- it's a weird artifact/bug), so I am curious to see whether Erza or anyone here knows what's up. There must be quite some number of people using backgroundrb and quite some serious about deployment and use capistrano.... So, I hope that we get to discover something more here. ;) On 3/16/07, Ben Reubenstein wrote: > > Hi Philip ~ > > I am having this issue too and have been meaning to post. I am using the > init script on Gentoo, and it works perfectly if I am in the machine but > when running from capistrano it will report a successful start/stop but the > process is nowhere to be found. I have to login to the machine and start it > manually. I am running on the latest version so I don't think it is a > backgroundrb version issue. > > ~ Ben > > On 3/15/07, Philip Lipu Tsai wrote: > > > > Erza, > > > > I am on Centos and bash, Rails 1.2.2, Ruby 1.8.4, BackgrounDrb old > > version (the 0.1.x version right before 0.2... Is the depend() section > > peculiar only to Gentoo? If not, anything one needs to do for this? > > > > I wrote a script just similar to this (w/o paying attention to the > > depend() section) and have capistrano before_restart task invoke the > > "/etc/init.d/backgroundrbd restart" (which is just stop and then start). > > However, this problem of backgroundrb's exiting after cap task's successful > > completion occurs (yes, no hanging in cap's process, but backgroundrb quits > > even though it is started in -d mode; the pid log file is there, but the > > process is no longer). Prepending with "nohup" in start seems to work, but > > I thought by using init.d style, one needn't worry about the nohup > > problem, either. Any insight? (Or is this a 0.1.x problem?) > > > > Thanks, > > Philip > > > > On 3/12/07, Ezra Zygmuntowicz wrote: > > > > > > > > > Yeah there are some weird interactions between capistrano and > > > some > > > daemons, the old version of backgroundrb exibits this behavior. That > > > way I get around it is by creating an init.d script, then letting > > > capuse that to restart bdrb. Works like a charm. Here is an init.d > > > script for Gentoo systems, it should be easy to adapt to any other > > > unixy system.: > > > > > > #!/sbin/runscript > > > # Copyright 1999-2004 Gentoo Foundation > > > # Distributed under the terms of the GNU General Public License v2 > > > # $Header: /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v > > > 1.1 2006/07/04 16:58:38 voxus Exp $ > > > > > > > > > depend() { > > > need net > > > use dns > > > after gfs > > > } > > > start() { > > > ebegin "Starting bdrb2" > > > RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ > > > start -d > > > eend $? "Failed to start bdrb2" > > > } > > > > > > stop() { > > > ebegin "Stopping bdrb2" > > > kill -9 `cat /data/toolbox/current/log/ > > > backgroundrb.pid` 2>/dev/null; true > > > eend $? "Failed to stop bdrb2" > > > } > > > > > > > > > > > > Cheers- > > > -Ezra > > > > > > > > > On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > > > > > > > Hello ~ > > > > > > > > I am using the Capistrano recipe outlined in this post to restart > > > > my backgroundrb server: > > > > > > > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > > > > 000718.html > > > > > > > > My Version: > > > > ~~~~~~~~~~~~ > > > > task :before_restart, :roles => :app do > > > > restart_backgroundrb > > > > end > > > > > > > > task :start_backgroundrb, :roles => :app do > > > > sudo "nohup /var/www/rails/#{application}/current/script/ > > > > backgroundrb/start " > > > > end > > > > > > > > task :stop_backgroundrb, :roles => :app do > > > > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > > > > backgroundrb.pid` 2>/dev/null; true" > > > > end > > > > > > > > task :restart_backgroundrb, :roles => :app do > > > > stop_backgroundrb > > > > start_backgroundrb > > > > end > > > > ~~~~~~~~~~~~ > > > > > > > > The backgroundrb server is stopped and started but hangs at the > > > > start_backgroundrb task. Capistrano shows: > > > > > > > > * executing task start_backgroundrb > > > > * executing "sudo nohup /var/www/rails/*************/current/ > > > > script/backgroundrb/start" > > > > servers: ["host"] > > > > [host] executing command > > > > ** [out :: host] nohup: ignoring input and appending output to > > > > `nohup.out' > > > > > > > > It will hang there and not restart the main web app process. I > > > > updated my gems to see if this was a version issue, but am > > > > experiencing the same problem. I also tried running this > > > > task :after_restart, the web app is successfully restarted, but it > > > > still hangs. > > > > > > > > Thanks in advance for any ideas on how to fix this. > > > > > > > > ~ Ben > > > > > > > > -- > > > > Ben Reubenstein > > > > 303-947-0446 > > > > http://www.benr75.com > > > > _______________________________________________ > > > > Backgroundrb-devel mailing list > > > > Backgroundrb-devel at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- Ezra Zygmuntowicz > > > -- Lead Rails Evangelist > > > -- ez at engineyard.com > > > -- Engine Yard, Serious Rails Hosting > > > -- (866) 518-YARD (9273) > > > > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Ben Reubenstein > 303-947-0446 > http://www.benr75.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070316/ba2274fa/attachment-0001.html From tsailipu at gmail.com Fri Mar 16 13:04:27 2007 From: tsailipu at gmail.com (Philip Lipu Tsai) Date: Fri, 16 Mar 2007 09:04:27 -0800 Subject: [Backgroundrb-devel] Capistrano Hanging on Deployment In-Reply-To: References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> <44c79a160703160933s621fd293o737057a4f34445b0@mail.gmail.com> Message-ID: Typo -- clarifying here for future newcomers, though experienced eyes probably caught this already: That command that I put inside init.d was: "RAILS_ENV=production nohup /[path to my app]/current/script/backgroudrb/start -d" (one doesn't really put "/etc/init.d/[script name] -d" inside the script itself. ;) On 3/16/07, Philip Lipu Tsai wrote: > > Hello Ben, > > Erza and you are on Gentoo -- so I thought Erza's script would work on > Gentoo. Looks like your try didn't work there, either(?) Because I am on > Centos, based on RedHat, I thought there may be some minor differences (the > only difference being the depend section that I don't know how to > translate.. well, unless I spend more time looking into it.). So I tried > putting nohup as such: "RAILS_ENV=production nohup /etc/init.d/backgroundrbd > start -d" and it worked. Though it worked, this combination of nohup inside > init.d looked a bit unusual (btw, I don't know if setting this RAILS_ENV > this way is still necessary for 0.2.x -- it's a weird artifact/bug), so I > am curious to see whether Erza or anyone here knows what's up. There must > be quite some number of people using backgroundrb and quite some serious > about deployment and use capistrano.... So, I hope that we get to discover > something more here. ;) > > On 3/16/07, Ben Reubenstein wrote: > > > > Hi Philip ~ > > > > I am having this issue too and have been meaning to post. I am using > > the init script on Gentoo, and it works perfectly if I am in the machine but > > when running from capistrano it will report a successful start/stop but the > > process is nowhere to be found. I have to login to the machine and start it > > manually. I am running on the latest version so I don't think it is a > > backgroundrb version issue. > > > > ~ Ben > > > > On 3/15/07, Philip Lipu Tsai < tsailipu at gmail.com> wrote: > > > > > > Erza, > > > > > > I am on Centos and bash, Rails 1.2.2, Ruby 1.8.4, BackgrounDrb old > > > version (the 0.1.x version right before 0.2... Is the depend() > > > section peculiar only to Gentoo? If not, anything one needs to do for this? > > > > > > > > > I wrote a script just similar to this (w/o paying attention to the > > > depend() section) and have capistrano before_restart task invoke the > > > "/etc/init.d/backgroundrbd restart" (which is just stop and then start). > > > However, this problem of backgroundrb's exiting after cap task's successful > > > completion occurs (yes, no hanging in cap's process, but backgroundrb quits > > > even though it is started in -d mode; the pid log file is there, but the > > > process is no longer). Prepending with "nohup" in start seems to work, but > > > I thought by using init.d style, one needn't worry about the nohup > > > problem, either. Any insight? (Or is this a 0.1.x problem?) > > > > > > Thanks, > > > Philip > > > > > > On 3/12/07, Ezra Zygmuntowicz wrote: > > > > > > > > > > > > Yeah there are some weird interactions between capistrano > > > > and some > > > > daemons, the old version of backgroundrb exibits this behavior. That > > > > way I get around it is by creating an init.d script, then letting > > > > capuse that to restart bdrb. Works like a charm. Here is an init.d > > > > script for Gentoo systems, it should be easy to adapt to any other > > > > unixy system.: > > > > > > > > #!/sbin/runscript > > > > # Copyright 1999-2004 Gentoo Foundation > > > > # Distributed under the terms of the GNU General Public License v2 > > > > # $Header: > > > > /var/cvsroot/gentoo-x86/www-servers/nginx/files/nginx-r1,v > > > > 1.1 2006/07/04 16:58:38 voxus Exp $ > > > > > > > > > > > > depend() { > > > > need net > > > > use dns > > > > after gfs > > > > } > > > > start() { > > > > ebegin "Starting bdrb2" > > > > RAILS_ENV=demo /data/toolbox/current/script/backgroundrb/ > > > > > > > > start -d > > > > eend $? "Failed to start bdrb2" > > > > } > > > > > > > > stop() { > > > > ebegin "Stopping bdrb2" > > > > kill -9 `cat /data/toolbox/current/log/ > > > > backgroundrb.pid` 2>/dev/null; true > > > > eend $? "Failed to stop bdrb2" > > > > } > > > > > > > > > > > > > > > > Cheers- > > > > -Ezra > > > > > > > > > > > > On Mar 12, 2007, at 10:47 AM, Ben Reubenstein wrote: > > > > > > > > > Hello ~ > > > > > > > > > > I am using the Capistrano recipe outlined in this post to restart > > > > > my backgroundrb server: > > > > > > > > > > http://rubyforge.org/pipermail/backgroundrb-devel/2007-February/ > > > > > 000718.html > > > > > > > > > > My Version: > > > > > ~~~~~~~~~~~~ > > > > > task :before_restart, :roles => :app do > > > > > restart_backgroundrb > > > > > end > > > > > > > > > > task :start_backgroundrb, :roles => :app do > > > > > sudo "nohup /var/www/rails/#{application}/current/script/ > > > > > backgroundrb/start " > > > > > end > > > > > > > > > > task :stop_backgroundrb, :roles => :app do > > > > > sudo "kill -9 `cat /var/www/rails/#{application}/current/log/ > > > > > backgroundrb.pid` 2>/dev/null; true" > > > > > end > > > > > > > > > > task :restart_backgroundrb, :roles => :app do > > > > > stop_backgroundrb > > > > > start_backgroundrb > > > > > end > > > > > ~~~~~~~~~~~~ > > > > > > > > > > The backgroundrb server is stopped and started but hangs at the > > > > > start_backgroundrb task. Capistrano shows: > > > > > > > > > > * executing task start_backgroundrb > > > > > * executing "sudo nohup /var/www/rails/*************/current/ > > > > > script/backgroundrb/start" > > > > > servers: ["host"] > > > > > [host] executing command > > > > > ** [out :: host] nohup: ignoring input and appending output to > > > > > `nohup.out' > > > > > > > > > > It will hang there and not restart the main web app process. I > > > > > updated my gems to see if this was a version issue, but am > > > > > experiencing the same problem. I also tried running this > > > > > task :after_restart, the web app is successfully restarted, but it > > > > > still hangs. > > > > > > > > > > Thanks in advance for any ideas on how to fix this. > > > > > > > > > > ~ Ben > > > > > > > > > > -- > > > > > Ben Reubenstein > > > > > 303-947-0446 > > > > > http://www.benr75.com > > > > > _______________________________________________ > > > > > Backgroundrb-devel mailing list > > > > > Backgroundrb-devel at rubyforge.org > > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > -- Ezra Zygmuntowicz > > > > -- Lead Rails Evangelist > > > > -- ez at engineyard.com > > > > -- Engine Yard, Serious Rails Hosting > > > > -- (866) 518-YARD (9273) > > > > > > > > > > > > _______________________________________________ > > > > Backgroundrb-devel mailing list > > > > Backgroundrb-devel at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > -- > > Ben Reubenstein > > 303-947-0446 > > http://www.benr75.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070316/80718471/attachment.html From bernesco at muteki.it Thu Mar 22 13:53:43 2007 From: bernesco at muteki.it (=?ISO-8859-1?Q?Alessio_Bernesco_L=E0vore?=) Date: Thu, 22 Mar 2007 18:53:43 +0100 Subject: [Backgroundrb-devel] Problems using a internationalization plugin with BackgroundRb In-Reply-To: References: <44c79a160703121047p2a52c07cv62dd99043a21df5e@mail.gmail.com> <63C2A032-55BA-4D12-8B04-3D3C192779D1@brainspl.at> <44c79a160703160933s621fd293o737057a4f34445b0@mail.gmail.com> Message-ID: <0F71D65A-7EFC-46E5-9EF1-086E427976B2@muteki.it> Hi everyone, I've a strange problem during those days, giving me a really bad headache I'm using a worker that has someting like this: for message in RunningMessage.find(:all,:condtions => ["......"]) message.expired end The class RunningMessage use the "localization" plugin in this way; def expired User.find("....").set_message(_("message_expired",Time.now)) end The localization file has a key "message_expired" that looks like: "message_expired","the message expires at %S" When i run the method expired from inside rails, the user receives the translated message, but when the method is invocated by the worker the user receives just the not translated key in the message. I've read something about the needs to re-initialize the plugins inside the workers, but at this time i've initilized the Localization plugin inside the Worker, in the User class, in the Message class, but nothing happens. Some hints? Thanks, alessio. From strass at strasslab.net Wed Mar 28 16:48:07 2007 From: strass at strasslab.net (Yann Klis) Date: Wed, 28 Mar 2007 22:48:07 +0200 Subject: [Backgroundrb-devel] Timeout ? Message-ID: <460AD487.3020004@strasslab.net> Hello folks, I'm looking for a way to stop my workers after a certain amount of time (a timeout). I'm not finding any good tutorial on this subject (just this thread http://rubyforge.org/pipermail/backgroundrb-devel/2006-December/thread.html#600 ). How do you guys implement a timeout mechanisme in your backgroundrb worker ? Thanks in advance, ++ yk From mengkuan at gmail.com Thu Mar 29 12:03:03 2007 From: mengkuan at gmail.com (Meng Kuan) Date: Fri, 30 Mar 2007 00:03:03 +0800 Subject: [Backgroundrb-devel] thread pool Message-ID: Hi, I am trying to limit the number of worker threads running at the same time. Adding ":poolsize: 2" to backgroundrb.yml or starting backgroundrb server like this script/backgroundrb start -- -s 2 does not seem to help. I can still create more than 2 worker threads from a rails app. I tried this on ubuntu dapper - ruby 1.8.4 - backgroundrb 0.2.1 as well as mac osx 10.4.9 - ruby 1.8.5 - backgroundrb 0.2.1 but no luck. Does thread pool work for anyone properly? Or do you use some other methods like implementing your own queue system? Appreciate any hints. Thanks! cheers, mengkuan From masonhale at gmail.com Thu Mar 29 18:26:23 2007 From: masonhale at gmail.com (Mason Hale) Date: Thu, 29 Mar 2007 17:26:23 -0500 Subject: [Backgroundrb-devel] thread pool In-Reply-To: References: Message-ID: <8bca3aa10703291526j2bc78b57w8d85f16f4cb7e352@mail.gmail.com> I thought the parameter was :pool_size (not :poolsize), but either way, I've found it does not limit the number of slave processes spawned in backgroundrb. On 3/29/07, Meng Kuan wrote: > > Hi, > > I am trying to limit the number of worker threads running at the same > time. Adding ":poolsize: 2" to backgroundrb.yml or starting > backgroundrb server like this > > script/backgroundrb start -- -s 2 > > does not seem to help. I can still create more than 2 worker threads > from a rails app. > > I tried this on > > ubuntu dapper > - ruby 1.8.4 > - backgroundrb 0.2.1 > > as well as > > mac osx 10.4.9 > - ruby 1.8.5 > - backgroundrb 0.2.1 > > but no luck. > > Does thread pool work for anyone properly? Or do you use some other > methods like implementing your own queue system? > > Appreciate any hints. Thanks! > > cheers, > mengkuan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070329/39d225f7/attachment-0001.html From mengkuan at gmail.com Thu Mar 29 21:09:07 2007 From: mengkuan at gmail.com (Meng Kuan) Date: Fri, 30 Mar 2007 09:09:07 +0800 Subject: [Backgroundrb-devel] thread pool In-Reply-To: <8bca3aa10703291526j2bc78b57w8d85f16f4cb7e352@mail.gmail.com> References: <8bca3aa10703291526j2bc78b57w8d85f16f4cb7e352@mail.gmail.com> Message-ID: On 30 Mar 2007, at 6:26 AM, Mason Hale wrote: > I thought the parameter was :pool_size (not :poolsize), but either > way, I've found it does not limit the number of slave processes > spawned in backgroundrb. Yes, sorry about the typo. And yes, it does not seem to affect the spawning. So it seems I will be better off implementing my own queueing system? From masonhale at gmail.com Thu Mar 29 21:41:17 2007 From: masonhale at gmail.com (Mason Hale) Date: Thu, 29 Mar 2007 20:41:17 -0500 Subject: [Backgroundrb-devel] thread pool In-Reply-To: References: <8bca3aa10703291526j2bc78b57w8d85f16f4cb7e352@mail.gmail.com> Message-ID: <8bca3aa10703291841m2d5f70fbk1ebca97722914d07@mail.gmail.com> It may be quicker to fix the existing queuing system in backgroundrb rather than implement something new. But I don't know the details of the existing implementation. Skaar? Ezra? any suggestions? Mason On 3/29/07, Meng Kuan wrote: > > On 30 Mar 2007, at 6:26 AM, Mason Hale wrote: > > > I thought the parameter was :pool_size (not :poolsize), but either > > way, I've found it does not limit the number of slave processes > > spawned in backgroundrb. > > Yes, sorry about the typo. And yes, it does not seem to affect the > spawning. So it seems I will be better off implementing my own > queueing system? > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070329/cf2f7755/attachment.html From walter at katipo.co.nz Thu Mar 29 23:18:56 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Fri, 30 Mar 2007 15:18:56 +1200 Subject: [Backgroundrb-devel] example code for file upload Message-ID: I have an archiving application that handles large file uploads. I already use backgroundrb to run a behind the scenes bulk import of records. All good. Thank you Ezra. Now I would like to add long running file uploads to the things that backgroundrb does for me. That's right that ole chestnut. I've seen a lot of mention of this practice around the web, but have yet come across good example code about how this is done. At least not beyond the ajax progress bar aspect. I guess I'm looking for answer to how one gets the stream of the uploading file submitted by the user from mongrel to a backgroundrb before the upload actually completes? Cheers, Walter From ezmobius at gmail.com Fri Mar 30 12:00:59 2007 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 30 Mar 2007 09:00:59 -0700 Subject: [Backgroundrb-devel] example code for file upload In-Reply-To: References: Message-ID: On Mar 29, 2007, at 8:18 PM, Walter McGinnis wrote: > I have an archiving application that handles large file uploads. I > already use backgroundrb to run a behind the scenes bulk import of > records. All good. Thank you Ezra. > > Now I would like to add long running file uploads to the things that > backgroundrb does for me. That's right that ole chestnut. I've seen > a lot of mention of this practice around the web, but have yet come > across good example code about how this is done. At least not beyond > the ajax progress bar aspect. > > I guess I'm looking for answer to how one gets the stream of the > uploading file submitted by the user from mongrel to a backgroundrb > before the upload actually completes? > > Cheers, > Walter > Hey Walter- You don't. BackgrounDrb cannot handle file uploads. You probably want to look at the mongrel_upload_progress gem. Bdrb can do progress bars for tasks it runs but the cotnrol flow of http requests and rails does not allow you to do anything with the file stream until its fully uploaded already so bdrb never has a chance to get involved. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From walter at katipo.co.nz Fri Mar 30 17:36:20 2007 From: walter at katipo.co.nz (Walter McGinnis) Date: Sat, 31 Mar 2007 09:36:20 +1200 Subject: [Backgroundrb-devel] example code for file upload In-Reply-To: References: Message-ID: On Mar 31, 2007, at 4:00 AM, Ezra Zygmuntowicz wrote: > > On Mar 29, 2007, at 8:18 PM, Walter McGinnis wrote: > >> >> I guess I'm looking for answer to how one gets the stream of the >> uploading file submitted by the user from mongrel to a backgroundrb >> before the upload actually completes? >> > > Hey Walter- > > You don't. BackgrounDrb cannot handle file uploads. You probably > want to look at the mongrel_upload_progress gem. Bdrb can do > progress bars for tasks it runs but the cotnrol flow of http > requests and rails does not allow you to do anything with the file > stream until its fully uploaded already so bdrb never has a chance > to get involved. > That makes sense. Thanks for clearing that up. I had already looked at the doc on the mongrel progress bar gem. I'll return to it. My real problem is uploads over crap network connections turning into CPU eating zombie mongrels. Man, I love it when geek speak sounds like a horror movie premise, but in this case it's a real pain. I suspect that the solution for that is going to be mongrel based though. I'll RTFM over in mongrel doc land, but any hints are appreciated. Cheers, Walter