From eden.li at gmail.com Fri Sep 1 02:40:50 2006 From: eden.li at gmail.com (Eden Li) Date: Fri, 1 Sep 2006 14:40:50 +0800 Subject: [Backgroundrb-devel] Unable to start with Capistrano... Message-ID: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> I'm having the same issue. My recipe in config/deploy.rb looks like this: desc "Start backgroundrb" task :backgroundrb_start, :roles => :app do run "cd #{current_path} && sudo rake backgroundrb:start RAILS_ENV=production" end (dev machine) $ cap backgroundrb_start [ ... command finishes without error ...] (server) $ ps auxww | grep backgroundrb [... nothing ...] When I run $ cd /path/to/current && sudo rake backgroundrb:start RAILS_ENV=production directly on my server, backgroundrb starts up perfectly, but whenever run this task remotely, backgroundrb never comes up. Anyone know what's going on? From dvancoevorden at yahoo.com Fri Sep 1 03:26:34 2006 From: dvancoevorden at yahoo.com (David van Coevorden) Date: Fri, 1 Sep 2006 00:26:34 -0700 (PDT) Subject: [Backgroundrb-devel] install backgroundrb Message-ID: <20060901072634.85187.qmail@web30601.mail.mud.yahoo.com> Hi This may be a repeat question on the list, but I could not find an answer. I cannot install backgroundrb via the usual svn method as the firewall does not permit it. Is there a place where I can download the latest version please? Thank you David --------------------------------- Get your own web address for just $1.99/1st yr. We'll help. Yahoo! Small Business. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/f0ae1fc1/attachment.html From jmetillon at gmail.com Fri Sep 1 11:00:18 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Fri, 1 Sep 2006 17:00:18 +0200 Subject: [Backgroundrb-devel] Worker suicide Message-ID: Hi the list people, hi Ezra, I just read from this message http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000134.html that BackgrounDRb will offer a way to kill a worker from within itself. As this message is from 7/25, I wonder if this feature has been added since. If not, is there any way to do it? I tried this: MiddleMan.delete_worker(@_job_key) But it raise: uninitialized constant MiddleMan - (NameError) Thanks Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/c4e742a6/attachment.html From ezmobius at gmail.com Fri Sep 1 11:24:15 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 1 Sep 2006 08:24:15 -0700 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: Message-ID: On Sep 1, 2006, at 8:00 AM, Jonathan M?tillon wrote: > Hi the list people, hi Ezra, > > I just read from this message http://rubyforge.org/pipermail/ > backgroundrb-devel/2006-July/000134.html that BackgrounDRb will > offer a way to kill a worker from within itself. > > As this message is from 7/25, I wonder if this feature has been > added since. If not, is there any way to do it? I tried this: > > MiddleMan.delete_worker (@_job_key) > > But it raise: > > uninitialized constant MiddleMan - (NameError) > > Thanks > > Jonathan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel Jonathan- If your worker is done and not doing any more computation you can just call kill from within your worker. If it is still working on something and you need its thread to be killed also you can do like this: ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/b0f472a5/attachment.html From ezmobius at gmail.com Fri Sep 1 11:30:47 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 1 Sep 2006 08:30:47 -0700 Subject: [Backgroundrb-devel] install backgroundrb In-Reply-To: <20060901072634.85187.qmail@web30601.mail.mud.yahoo.com> References: <20060901072634.85187.qmail@web30601.mail.mud.yahoo.com> Message-ID: On Sep 1, 2006, at 12:26 AM, David van Coevorden wrote: > Hi > > This may be a repeat question on the list, but I could not find an > answer. > I cannot install backgroundrb via the usual svn method as the > firewall does not permit it. Is there a place where I can download > the latest version please? > > Thank you > David > > Get your own web address for just $1.99/1st yr. We'll help. Yahoo! > Small Business. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel Hey David- Here is a copy of the latest version. Have fun! -Ezra ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/8d3f8dcd/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: bgdrb.zip Type: application/zip Size: 23681 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/8d3f8dcd/attachment-0001.zip -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060901/8d3f8dcd/attachment-0003.html From ezmobius at gmail.com Fri Sep 1 11:35:09 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 1 Sep 2006 08:35:09 -0700 Subject: [Backgroundrb-devel] Unable to start with Capistrano... In-Reply-To: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> References: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> Message-ID: <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> On Aug 31, 2006, at 11:40 PM, Eden Li wrote: > I'm having the same issue. > > My recipe in config/deploy.rb looks like this: > > desc "Start backgroundrb" > task :backgroundrb_start, :roles => :app do > run "cd #{current_path} && sudo rake backgroundrb:start > RAILS_ENV=production" > end > > (dev machine) > $ cap backgroundrb_start > [ ... command finishes without error ...] > > (server) > $ ps auxww | grep backgroundrb > [... nothing ...] > > When I run > > $ cd /path/to/current && sudo rake backgroundrb:start > RAILS_ENV=production > > directly on my server, backgroundrb starts up perfectly, but whenever > run this task remotely, backgroundrb never comes up. > > Anyone know what's going on? It seems that for some reason backgroundrb doesn't like being started via the rake command to start. So you need to use the actual start script. Here are my tasks for stopping, starting and restarting. desc <<-DESC Stop the backgroundrb server DESC task :stop_backgroundrb , :roles => :drb do run "#{current_path}/script/backgroundrb/stop" end desc <<-DESC Start the backgroundrb server DESC task :start_backgroundrb , :roles => :drb do run "#{current_path}/script/backgroundrb/start -d" end desc <<-DESC Start the backgroundrb server DESC task :restart_backgroundrb , :roles => :app do stop_backgroundrb start_backgroundrb end Cheers- -Ezra From jmetillon at gmail.com Fri Sep 1 12:33:48 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Fri, 1 Sep 2006 18:33:48 +0200 Subject: [Backgroundrb-devel] Unable to start with Capistrano... In-Reply-To: <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> References: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> Message-ID: On 9/1/06, Ezra Zygmuntowicz wrote: > > Here are my tasks for stopping, starting and restarting. > > desc <<-DESC > Stop the backgroundrb server > DESC > task :stop_backgroundrb , :roles => :drb do > run "#{current_path}/script/backgroundrb/stop" > end > > desc <<-DESC > Start the backgroundrb server > DESC > task :start_backgroundrb , :roles => :drb do > run "#{current_path}/script/backgroundrb/start -d" > end > > desc <<-DESC > Start the backgroundrb server > DESC > task :restart_backgroundrb , :roles => :app do > stop_backgroundrb > start_backgroundrb > end > > > Cheers- > -Ezra Maybe this is overkill but here is a bash script I've wrote to forcefully restart BackgrounDRb: #!/bin/sh if [ -a "log/backgroundrb.pid" ]; then pid=`cat log/backgroundrb.pid` if [[ -n "$pid" ]]; then echo "Killing current BackgrounDRb server (pid: $pid)" kill -9 $pid fi fi echo "Starting the BackgrounDRb server" cd /var/www/123/current log="/tmp/backgroundrb.`date +%s`" echo "Writing to $log" nohup rake backgroundrb:start > $log 2>&1 cat $log rm -f $log It is derived from a script I found while googling, I don't remember who it's from so I can't give credit. Here's what I added to my deploy recipe: task :before_restart, :roles => :app do sudo "#{release_path}/script/backgroundrb/restart" end From gethemant at gmail.com Sun Sep 3 18:25:05 2006 From: gethemant at gmail.com (hemant) Date: Mon, 4 Sep 2006 03:55:05 +0530 Subject: [Backgroundrb-devel] Kill my Thread Message-ID: Following code, should kill the worker thread, if there is a error in connecting to the database, right? but in my case it doesn't and goes on doing other stuff. Of course...there are lots of work arounds for this, but how do i make sure that, the thread is killed at that point of time. begin @dbh = Mysql.real_connect ("localhost","root","","typo_#{username}", at sql_port, at blog_socket) rescue Exception => e @logger.debug("### Error connecting to the database : #{e.to_s}") kill end -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060904/7f3686c5/attachment.html From lapomme00 at gmail.com Mon Sep 4 13:55:31 2006 From: lapomme00 at gmail.com (Brett Walker) Date: Tue, 5 Sep 2006 01:55:31 +0800 Subject: [Backgroundrb-devel] Unable to start with Capistrano... In-Reply-To: References: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> Message-ID: I already had it setup like Ezra, and I could *not* get it to work, even after much gnashing of teeth. The task would always look like it was successful, and the pid file would be sitting there, but the process would be gone. Somehow it must have still been attached to the ssh process. Jonathan's script definitely works. Bummed I need another script, but glad it's automated now. Cheers, Brett On 9/2/06, Jonathan M?tillon wrote: > On 9/1/06, Ezra Zygmuntowicz wrote: > > > > Here are my tasks for stopping, starting and restarting. > > > > desc <<-DESC > > Stop the backgroundrb server > > DESC > > task :stop_backgroundrb , :roles => :drb do > > run "#{current_path}/script/backgroundrb/stop" > > end > > > > desc <<-DESC > > Start the backgroundrb server > > DESC > > task :start_backgroundrb , :roles => :drb do > > run "#{current_path}/script/backgroundrb/start -d" > > end > > > > desc <<-DESC > > Start the backgroundrb server > > DESC > > task :restart_backgroundrb , :roles => :app do > > stop_backgroundrb > > start_backgroundrb > > end > > > > > > Cheers- > > -Ezra > > Maybe this is overkill but here is a bash script I've wrote to > forcefully restart BackgrounDRb: > > #!/bin/sh > > if [ -a "log/backgroundrb.pid" ]; then > pid=`cat log/backgroundrb.pid` > if [[ -n "$pid" ]]; then > echo "Killing current BackgrounDRb server (pid: $pid)" > kill -9 $pid > fi > fi > > echo "Starting the BackgrounDRb server" > cd /var/www/123/current > log="/tmp/backgroundrb.`date +%s`" > echo "Writing to $log" > nohup rake backgroundrb:start > $log 2>&1 > cat $log > rm -f $log > > It is derived from a script I found while googling, I don't remember > who it's from so I can't give credit. > > Here's what I added to my deploy recipe: > > task :before_restart, :roles => :app do > sudo "#{release_path}/script/backgroundrb/restart" > end > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From gethemant at gmail.com Mon Sep 4 20:20:50 2006 From: gethemant at gmail.com (hemant) Date: Tue, 5 Sep 2006 05:50:50 +0530 Subject: [Backgroundrb-devel] Kill my Thread In-Reply-To: References: Message-ID: How about using rubyinline to optimize the worker threads? Currently I have a feeling that worker threads are running really slow. On 9/4/06, hemant wrote: > > > Following code, should kill the worker thread, if there is a error in > connecting to the database, right? > but in my case it doesn't and goes on doing other stuff. Of course...there > are lots of work arounds for this, but how do i make sure that, the thread > is killed at that point of time. > > begin > @dbh = Mysql.real_connect("localhost","root","","typo_#{username}", at sql_port, at blog_socket) > > rescue Exception => e > @logger.debug("### Error connecting to the database : #{e.to_s}") > kill > end > > -- > nothing much to talk > -- nothing much to talk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060905/e4152d0a/attachment.html From eden.li at gmail.com Tue Sep 5 05:15:47 2006 From: eden.li at gmail.com (Eden Li) Date: Tue, 5 Sep 2006 17:15:47 +0800 Subject: [Backgroundrb-devel] Unable to start with Capistrano... In-Reply-To: <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> References: <565dbdd40608312340x1e28a1a5u5ff1b303b8f1b09@mail.gmail.com> <221A0645-D03F-4762-85C6-F6808B9E157A@gmail.com> Message-ID: <565dbdd40609050215g38696b7ahc34026000a295874@mail.gmail.com> Ah, it appears `nohup' is the answer. You if you add nohup before the backgroundrb script invocation in your capistrano recipe, everything works perfect: desc "Start backgroundrb" task :backgroundrb_start, :roles => :app do run "RAILS_ENV=#{rails_env} nohup #{current_path}/script/backgroundrb/start -d > /dev/null 2>&1" end Thanks for posting the script. Maybe the next release of backgroundrb could include capistrano recipes for starting/stopping/restarting the same way mongrel_cluster does? On 9/2/06, Jonathan M?tillon wrote: > Maybe this is overkill but here is a bash script I've wrote to > forcefully restart BackgrounDRb: > > #!/bin/sh > > if [ -a "log/backgroundrb.pid" ]; then > pid=`cat log/backgroundrb.pid` > if [[ -n "$pid" ]]; then > echo "Killing current BackgrounDRb server (pid: $pid)" > kill -9 $pid > fi > fi > > echo "Starting the BackgrounDRb server" > cd /var/www/123/current > log="/tmp/backgroundrb.`date +%s`" > echo "Writing to $log" > nohup rake backgroundrb:start > $log 2>&1 > cat $log > rm -f $log > > It is derived from a script I found while googling, I don't remember > who it's from so I can't give credit. > > Here's what I added to my deploy recipe: > > task :before_restart, :roles => :app do > sudo "#{release_path}/script/backgroundrb/restart" > end From chrisjroos at gmail.com Tue Sep 5 08:33:05 2006 From: chrisjroos at gmail.com (Chris Roos) Date: Tue, 5 Sep 2006 13:33:05 +0100 Subject: [Backgroundrb-devel] Upgrading backgroundrb problems Message-ID: <3a5e51050609050533q510b7afbub499945103480c03@mail.gmail.com> I just ran into the same problem as James Hughes (thread "Rake tasks fail silently, how to debug?"). I'm wondering if the rake backgroundrb:setup task should always overwrite the script/backgroundrb files to be on the safe side. Or at least warn that some already exist. Maybe there's a way to be noisy if the backgroundrb:start task doesn't actually start a backgroundrb process? Just some thoughts. Chris From chrisjroos at gmail.com Tue Sep 5 09:06:25 2006 From: chrisjroos at gmail.com (Chris Roos) Date: Tue, 5 Sep 2006 14:06:25 +0100 Subject: [Backgroundrb-devel] connection closed when creating workers with the same key Message-ID: <3a5e51050609050606rf6f27b5x2ad1e10805614df2@mail.gmail.com> Hi, I was just creating some new workers (MiddleMan.new_worker) with the same job_key. The first worker for a given key is created correctly. Any subsequent requests to create and greeted with the 'connection closed' error. I realise that it's not possible to have two workers with the same key, but maybe a friendlier error message is in order. Chris From chrisjroos at gmail.com Tue Sep 5 10:26:35 2006 From: chrisjroos at gmail.com (Chris Roos) Date: Tue, 5 Sep 2006 15:26:35 +0100 Subject: [Backgroundrb-devel] File processing implementation question Message-ID: <3a5e51050609050726n47eb04c0k3e4da36112089483@mail.gmail.com> Hi, I'm wanting to upload a text file to the server that is processed from a backgroundrb worker. I currently have it working but I'm wondering if there are any recommended approaches for this kind of work? My approach is currently to write the uploaded file to a known place on disk. I pass the filename to my worker which then reads, processes and deletes the file. Are there any improvements I can make to this? Thanks in advance, Chris From chrisjroos at gmail.com Tue Sep 5 12:09:18 2006 From: chrisjroos at gmail.com (Chris Roos) Date: Tue, 5 Sep 2006 17:09:18 +0100 Subject: [Backgroundrb-devel] environment config parameter Message-ID: <3a5e51050609050909x46047a49n32b027c98f8d1629@mail.gmail.com> Am I correct in thinking that this is no longer required? The only place I could see it used (script/start) was commented out. Chris From christopher.k.hall at gmail.com Tue Sep 5 12:09:55 2006 From: christopher.k.hall at gmail.com (Chris Hall) Date: Tue, 5 Sep 2006 12:09:55 -0400 Subject: [Backgroundrb-devel] question about passing array of AR objects to worker Message-ID: got a wierd problem maybe someone can help with. whenever i try to pass an array or AR objects to my worker, i end up with a DRb object in the worker...here's an example class SearchWorker < BackgrounDRb::Rails attr_reader :resources def do_work(resources) logger.info resources.inspect end end class SearchController < ApplicationController def start_search @resources = Resource.find(:all) logger.info @resources.inspect session[:job_key] = MiddleMan.new_worker(:class => :search_worker, :args => @resources) end end class Resource < ActiveRecord::Base include DRbUndumped # attribute, "uri", string end now, my controller logs the array as: [#"http://www.google.com/search?hl=en&q=ruby+on+rails&btnG=Google+Search", "id"=>"1"}>, #"http://eee.tsinghua.edu.cn/", "id"=>"2"}>, #"http://www.digg.com/?s=ruby+on+rails", "id"=>"3"}>, #"http://www.a9.com/ruby%20on%20rails", "id"=>"4"}>, #"http://www.yahoo.com/search?p=ruby+on+rails&fr=FP-tab-web-t500&toggle=1&cop=&ei=UTF-8", "id"=>"5"}>, #"http://search.msn.com/results.aspx?q=ruby+on+rails&FORM=QBHP", "id"=>"6"}>] but the worker logs it as (in backgroundrb.log) as # can someone explain what's going on here? Chris From cbruce at sleeter.com Tue Sep 5 20:56:38 2006 From: cbruce at sleeter.com (Chris Bruce) Date: Tue, 5 Sep 2006 17:56:38 -0700 Subject: [Backgroundrb-devel] Setting Product Env Message-ID: <204BD979E361E64DAF7FAC6996884BA629FD43@tsg-srv-03.sleeter.com> When I am using my rails models in my worker class, it appears the background worker class is always using development no matter what I do. I set background.yml to production. I even have ENV['RAILS_ENV'] ||= 'production uncommented in environment.rb. I don't have the development db even created on my production box, so not sure where it is getting development from. I am using fastcgi and have the environment correctly specified in my fastcgi config. Any ideas? Thanks, Chris From info at siebert-wd.de Wed Sep 6 02:25:42 2006 From: info at siebert-wd.de (Michael Siebert) Date: Wed, 6 Sep 2006 08:25:42 +0200 Subject: [Backgroundrb-devel] question about passing array of AR objects to worker In-Reply-To: References: Message-ID: <32b567370609052325q62ac1173r18169048f37c44e5@mail.gmail.com> its not good to pass AR objects to your worker, that has sth to do with DRb itself, dont know exactly what... a way around that is to pass the ids of the object or somthing else to identify the record. 2006/9/5, Chris Hall : > > got a wierd problem maybe someone can help with. > > whenever i try to pass an array or AR objects to my worker, i end up > with a DRb object in the worker...here's an example > > class SearchWorker < BackgrounDRb::Rails > > attr_reader :resources > > def do_work(resources) > logger.info resources.inspect > end > end > > class SearchController < ApplicationController > > def start_search > @resources = Resource.find(:all) > logger.info @resources.inspect > session[:job_key] = MiddleMan.new_worker(:class => :search_worker, > :args => @resources) > end > end > > class Resource < ActiveRecord::Base > include DRbUndumped > # attribute, "uri", string > end > > now, my controller logs the array as: > > [# @attributes={"uri"=>" > http://www.google.com/search?hl=en&q=ruby+on+rails&btnG=Google+Search", > "id"=>"1"}>, # @attributes={"uri"=>"http://eee.tsinghua.edu.cn/", "id"=>"2"}>, > # @attributes={"uri"=>"http://www.digg.com/?s=ruby+on+rails", > "id"=>"3"}>, # @attributes={"uri"=>"http://www.a9.com/ruby%20on%20rails", > "id"=>"4"}>, # @attributes={"uri"=>" > http://www.yahoo.com/search?p=ruby+on+rails&fr=FP-tab-web-t500&toggle=1&cop=&ei=UTF-8 > ", > "id"=>"5"}>, # @attributes={"uri"=>" > http://search.msn.com/results.aspx?q=ruby+on+rails&FORM=QBHP", > "id"=>"6"}>] > > > but the worker logs it as (in backgroundrb.log) as > > # @uri="druby://localhost:42531"> > > can someone explain what's going on here? > > Chris > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Michael Siebert www.siebert-wd.de - Gedanken lesen www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/b2fe92b9/attachment.html From jmetillon at gmail.com Wed Sep 6 04:07:41 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Wed, 6 Sep 2006 10:07:41 +0200 Subject: [Backgroundrb-devel] Setting Product Env In-Reply-To: <204BD979E361E64DAF7FAC6996884BA629FD43@tsg-srv-03.sleeter.com> References: <204BD979E361E64DAF7FAC6996884BA629FD43@tsg-srv-03.sleeter.com> Message-ID: Chris, Maybe you are using version control and when you deploy to your server, you don't update your newly checked out version so it uses production. That's what happened to me. For example in my deploy recipe I have: run "cat #{release_path}/config/backgroundrb.yml | sed 's#development#production#g' > #{release_path}/config/backgroundrb.yml" Just before the BackgrounDRb restart: sudo "#{release_path}/script/backgroundrb/restart" On 9/6/06, Chris Bruce wrote: > > > When I am using my rails models in my worker class, it appears the > background worker class is always using development no matter what I do. > I set background.yml to production. I even have ENV['RAILS_ENV'] ||= > 'production uncommented in environment.rb. I don't have the development > db even created on my production box, so not sure where it is getting > development from. I am using fastcgi and have the environment correctly > specified in my fastcgi config. > > Any ideas? > > > > Thanks, > > > Chris > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/f19603a9/attachment.html From jmetillon at gmail.com Wed Sep 6 04:11:57 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Wed, 6 Sep 2006 10:11:57 +0200 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: Message-ID: Thanks Ezra that works very well! Can you tell me what means the :: at the start of the line? On 9/1/06, Ezra Zygmuntowicz wrote: > > > On Sep 1, 2006, at 8:00 AM, Jonathan M?tillon wrote: > > Hi the list people, hi Ezra, > > I just read from this message > http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000134.htmlthat BackgrounDRb will offer a way to kill a worker from within itself. > > As this message is from 7/25, I wonder if this feature has been added > since. If not, is there any way to do it? I tried this: > > MiddleMan.delete_worker (@_job_key) > > But it raise: > > uninitialized constant MiddleMan - (NameError) > > Thanks > > Jonathan > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > Jonathan- > > If your worker is done and not doing any more computation you can just > call kill from within your worker. If it is still working on something and > you need its thread to be killed also you can do like this: > > ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key > > -Ezra > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/a17720fd/attachment.html From jmetillon at gmail.com Wed Sep 6 08:00:00 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Wed, 6 Sep 2006 14:00:00 +0200 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: <3a5e51050609060231t5e9c2947x61bc8e1f12a9b68@mail.gmail.com> References: <3a5e51050609060231t5e9c2947x61bc8e1f12a9b68@mail.gmail.com> Message-ID: I wonder why Ezra proposed the delete_worker method if a kill method exists? Maybe it's a brand new feature? i was not aware of it and found nothing about this while googling. On 9/6/06, Chris Roos wrote: > > There is also a kill method for a worker, so I believe (and it seems > to work) that you can just use kill (or self.kill) in place of the > above chain. > > The :: references the top level namespace. > > Chris > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/a9f2f90d/attachment.html From me at seebq.com Wed Sep 6 09:15:24 2006 From: me at seebq.com (Charles Brian Quinn) Date: Wed, 6 Sep 2006 09:15:24 -0400 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: Message-ID: <3a2de0cd0609060615kb461edbpa2d890ce5d667c48@mail.gmail.com> the :: means, start the class/module lookup at the root, so as not to conflict with any class/module namespaces you might already have. If you have a class you've defined called, say Violin, and it has a module or maybe a subclass called String (since violins have strings, get it?), then, if you wanted to create an actual String to deal with letters, you'd have to create it like so: ::String.new("A") because just saying: String.new("A") would find your Violin::String before it found the real String class. it's like in xpath, putting the // will return you to the root of the doc to perform lookups. If you're interested, the Ruby for Rails book has more on this (and uses the same example that I blantantly stole). On 9/6/06, Jonathan M?tillon wrote: > Thanks Ezra that works very well! > > Can you tell me what means the :: at the start of the line? > > > On 9/1/06, Ezra Zygmuntowicz < ezmobius at gmail.com> wrote: > > > > > > > > > > > > > > On Sep 1, 2006, at 8:00 AM, Jonathan M?tillon wrote: > > > > > > > > Hi the list people, hi Ezra, > > > > I just read from this message > http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000134.html > that BackgrounDRb will offer a way to kill a worker from within itself. > > > > As this message is from 7/25, I wonder if this feature has been added > since. If not, is there any way to do it? I tried this: > > > > MiddleMan.delete_worker (@_job_key) > > > > But it raise: > > > > uninitialized constant MiddleMan - (NameError) > > > > Thanks > > > > Jonathan > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > Jonathan- > > > > > > If your worker is done and not doing any more computation you can just > call kill from within your worker. If it is still working on something and > you need its thread to be killed also you can do like this: > > > > > > ::BackgrounDRb::MiddleMan.instance.delete_worker > @_job_key > > > > > > -Ezra > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com From jmetillon at gmail.com Wed Sep 6 09:56:46 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Wed, 6 Sep 2006 15:56:46 +0200 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: <3a2de0cd0609060615kb461edbpa2d890ce5d667c48@mail.gmail.com> References: <3a2de0cd0609060615kb461edbpa2d890ce5d667c48@mail.gmail.com> Message-ID: Thank you Charles! Ruby for Rails is already in my Amazon wish list ;-) On 9/6/06, Charles Brian Quinn wrote: > > the :: means, start the class/module lookup at the root, so as not to > conflict with any class/module namespaces you might already have. > > If you have a class you've defined called, say Violin, and it has a > module or maybe a subclass called String (since violins have strings, > get it?), then, if you wanted to create an actual String to deal with > letters, you'd have to create it like so: > > ::String.new("A") > > because just saying: > > String.new("A") > > would find your Violin::String before it found the real String class. > > it's like in xpath, putting the // will return you to the root of the > doc to perform lookups. > > If you're interested, the Ruby for Rails book has more on this (and > uses the same example that I blantantly stole). > > On 9/6/06, Jonathan M?tillon wrote: > > Thanks Ezra that works very well! > > > > Can you tell me what means the :: at the start of the line? > > > > > > On 9/1/06, Ezra Zygmuntowicz < ezmobius at gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > On Sep 1, 2006, at 8:00 AM, Jonathan M?tillon wrote: > > > > > > > > > > > > Hi the list people, hi Ezra, > > > > > > I just read from this message > > http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000134.html > > that BackgrounDRb will offer a way to kill a worker from within itself. > > > > > > As this message is from 7/25, I wonder if this feature has been added > > since. If not, is there any way to do it? I tried this: > > > > > > MiddleMan.delete_worker (@_job_key) > > > > > > But it raise: > > > > > > uninitialized constant MiddleMan - (NameError) > > > > > > Thanks > > > > > > Jonathan > > > > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > Jonathan- > > > > > > > > > If your worker is done and not doing any more computation you can just > > call kill from within your worker. If it is still working on something > and > > you need its thread to be killed also you can do like this: > > > > > > > > > ::BackgrounDRb::MiddleMan.instance.delete_worker > > @_job_key > > > > > > > > > -Ezra > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > -- > Charles Brian Quinn > self-promotion: www.seebq.com > highgroove studios: www.highgroove.com > slingshot hosting: www.slingshothosting.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/f7bb0bba/attachment.html From cbruce at sleeter.com Wed Sep 6 13:06:57 2006 From: cbruce at sleeter.com (Chris Bruce) Date: Wed, 6 Sep 2006 10:06:57 -0700 Subject: [Backgroundrb-devel] Setting Product Env In-Reply-To: Message-ID: <204BD979E361E64DAF7FAC6996884BA629FD60@tsg-srv-03.sleeter.com> I double and triple checked that. Not sure how it is getting development. Do you know of any way to force it? Chris ________________________________________ From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of Jonathan M?tillon Sent: Wednesday, September 06, 2006 1:08 AM To: Backgroundrb-devel at rubyforge.org Subject: Re: [Backgroundrb-devel] Setting Product Env Chris, Maybe you are using version control and when you deploy to your server, you don't update your newly checked out version so it uses production. That's what happened to me. For example in my deploy recipe I have: run "cat #{release_path}/config/backgroundrb.yml | sed 's#development#production#g' > #{release_path}/config/backgroundrb.yml" Just before the BackgrounDRb restart: sudo "#{release_path}/script/backgroundrb/restart" On 9/6/06, Chris Bruce wrote: When I am using my rails models in my worker class, it appears the background worker class is always using development no matter what I do. I set background.yml to production.??I even have ENV['RAILS_ENV'] ||= 'production uncommented in environment.rb.??I don't have the development db even created on my production box, so not sure where it is getting development from.??I am using fastcgi and have the environment correctly specified in my fastcgi config. Any ideas? Thanks, Chris _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel From ezmobius at gmail.com Wed Sep 6 13:30:02 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 6 Sep 2006 10:30:02 -0700 Subject: [Backgroundrb-devel] question about passing array of AR objects to worker In-Reply-To: References: Message-ID: <570AC30F-5E5B-438D-B3CA-0AA2DE8184BE@gmail.com> Hi ~ On Sep 5, 2006, at 9:09 AM, Chris Hall wrote: > got a wierd problem maybe someone can help with. > > whenever i try to pass an array or AR objects to my worker, i end up > with a DRb object in the worker...here's an example > > class SearchWorker < BackgrounDRb::Rails > > attr_reader :resources > > def do_work(resources) > logger.info resources.inspect > end > end > > class SearchController < ApplicationController > > def start_search > @resources = Resource.find(:all) > logger.info @resources.inspect > session[:job_key] = MiddleMan.new_worker(:class => :search_worker, > :args => @resources) > end > end > > class Resource < ActiveRecord::Base > include DRbUndumped > # attribute, "uri", string > end > > now, my controller logs the array as: > > [# @attributes={"uri"=>"http://www.google.com/search?hl=en&q=ruby+on > +rails&btnG=Google+Search", > "id"=>"1"}>, # @attributes={"uri"=>"http://eee.tsinghua.edu.cn/", "id"=>"2"}>, > # @attributes={"uri"=>"http://www.digg.com/?s=ruby+on+rails", > "id"=>"3"}>, # @attributes={"uri"=>"http://www.a9.com/ruby%20on%20rails", > "id"=>"4"}>, # @attributes={"uri"=>"http://www.yahoo.com/search?p=ruby+on > +rails&fr=FP-tab-web-t500&toggle=1&cop=&ei=UTF-8", > "id"=>"5"}>, # @attributes={"uri"=>"http://search.msn.com/results.aspx?q=ruby+on > +rails&FORM=QBHP", > "id"=>"6"}>] > > > but the worker logs it as (in backgroundrb.log) as > > # > > can someone explain what's going on here? > > Chris Hey Chris- This is the way that DRb works. For any object that you include DRbUndumped into, drb creates a proxy DRbObject on the other side of the wire. So #inspect will tell you its a DRbObject instead of your AR model. But you can just pretend that it really is your AR model and call whatever methods on it that you want and it will work. Its just that #inspect shows th proxy object instead of what the real object is. -Ezra From ezmobius at gmail.com Wed Sep 6 13:30:56 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 6 Sep 2006 10:30:56 -0700 Subject: [Backgroundrb-devel] environment config parameter In-Reply-To: <3a5e51050609050909x46047a49n32b027c98f8d1629@mail.gmail.com> References: <3a5e51050609050909x46047a49n32b027c98f8d1629@mail.gmail.com> Message-ID: <257A0AAE-7ADD-4396-AC23-79DB1911ED90@gmail.com> On Sep 5, 2006, at 9:09 AM, Chris Roos wrote: > Am I correct in thinking that this is no longer required? The only > place I could see it used (script/start) was commented out. > > Chris > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel Hey Chris- You may have discovered a small bug I introduced when I changed :load_rails to actually require env.rb. I will have a look and do an update later today to fix the problem. -Ezra From ezmobius at gmail.com Wed Sep 6 13:32:44 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 6 Sep 2006 10:32:44 -0700 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: <3a5e51050609060231t5e9c2947x61bc8e1f12a9b68@mail.gmail.com> Message-ID: On Sep 6, 2006, at 5:00 AM, Jonathan M?tillon wrote: > I wonder why Ezra proposed the delete_worker method if a kill > method exists? Maybe it's a brand new feature? i was not aware of > it and found nothing about this while googling. > > On 9/6/06, Chris Roos wrote: > There is also a kill method for a worker, so I believe (and it seems > to work) that you can just use kill (or self.kill) in place of the > above chain. > > The :: references the top level namespace. > > Chris > > There is a kill method you can call in your worker. I need to update it however because while it does delete the worker out of the @jobs and @timestamps hashes, it doesnt actualy kill the running thread. There will be a new release soon that addresses this issue. -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060906/ab96af55/attachment.html From mattp at digimonkey.com Wed Sep 6 15:41:53 2006 From: mattp at digimonkey.com (mattp at digimonkey.com) Date: Wed, 06 Sep 2006 14:41:53 -0500 Subject: [Backgroundrb-devel] rescue? Message-ID: <20060906144153.5t9ck2kbkc8wc0go@digimonkey.com> Probably a simple question, but humor me: Let's say I have a page (in Rails) that lets a user type a shell command (say, 'ping blah.com') into a form and it fires off a worker that then updates that page using AJAX (all of this is working so far). How would I correctly wrap the worker in a begin..rescue so that it spits out 'invalid command' when someone types in something incorrect. Right now, I have: def start_working begin @result = IO.popen(@cmd) rescue SystemCallError @result = "Invalid or Failed Command: " + $! end end but in my view, the @result doesn't seem to be accessible to the ajax periodical updater in the same way the @result from the popen is... what am i missing? [mp] From jmetillon at gmail.com Thu Sep 7 08:17:18 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Thu, 7 Sep 2006 14:17:18 +0200 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: <3a5e51050609060231t5e9c2947x61bc8e1f12a9b68@mail.gmail.com> Message-ID: And does ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key actually kill the thread? On 9/6/06, Ezra Zygmuntowicz wrote: > > > On Sep 6, 2006, at 5:00 AM, Jonathan M?tillon wrote: > > I wonder why Ezra proposed the delete_worker method if a kill method > exists? Maybe it's a brand new feature? i was not aware of it and found > nothing about this while googling. > > On 9/6/06, Chris Roos wrote: > > > > There is also a kill method for a worker, so I believe (and it seems > > to work) that you can just use kill (or self.kill) in place of the > > above chain. > > > > The :: references the top level namespace. > > > > Chris > > > > > > There is a kill method you can call in your worker. I need to update it > however because while it does delete the worker out of the @jobs and > @timestamps hashes, it doesnt actualy kill the running thread. There will be > a new release soon that addresses this issue. > > > -Ezra > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060907/84d35067/attachment-0001.html From eden.li at gmail.com Thu Sep 7 12:05:18 2006 From: eden.li at gmail.com (Eden Li) Date: Fri, 8 Sep 2006 00:05:18 +0800 Subject: [Backgroundrb-devel] Setting Product Env Message-ID: <565dbdd40609070905t15cca1f0y492001693f9cfe34@mail.gmail.com> I ran into this problem when my config/database.yml defined no development database. I fixed it by prepending the start command with RAILS_ENV=production ala RAILS_ENV=production script/backgroundrb/start -d I don't know if this will fix your problem since you mention that you're setting RAILS_ENV already in config/environment.rb, but this worked for me. From: "Chris Bruce" > Subject: Re: [Backgroundrb-devel] Setting Product Env > To: > Message-ID: > <204BD979E361E64DAF7FAC6996884BA629FD60 at tsg-srv-03.sleeter.com> > Content-Type: text/plain; charset="iso-8859-1" > > I double and triple checked that. Not sure how it is getting development. Do you know of any way to force it? > > > > Chris > > > ________________________________________ > From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of Jonathan M?tillon > Sent: Wednesday, September 06, 2006 1:08 AM > To: Backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Setting Product Env > > Chris, > > Maybe you are using version control and when you deploy to your server, you don't update your newly checked out version so it uses production. That's what happened to me. For example in my deploy recipe I have: > > run "cat #{release_path}/config/backgroundrb.yml | sed 's#development#production#g' > #{release_path}/config/backgroundrb.yml" > > Just before the BackgrounDRb restart: > > sudo "#{release_path}/script/backgroundrb/restart" > On 9/6/06, Chris Bruce wrote: > > When I am using my rails models in my worker class, it appears the > background worker class is always using development no matter what I do. > I set background.yml to production.??I even have ENV['RAILS_ENV'] ||= > 'production uncommented in environment.rb.??I don't have the development > db even created on my production box, so not sure where it is getting > development from.??I am using fastcgi and have the environment correctly > specified in my fastcgi config. > > Any ideas? > > > > Thanks, > > > Chris From cbruce at sleeter.com Thu Sep 7 12:25:57 2006 From: cbruce at sleeter.com (Chris Bruce) Date: Thu, 7 Sep 2006 09:25:57 -0700 Subject: [Backgroundrb-devel] Setting Product Env In-Reply-To: <565dbdd40609070905t15cca1f0y492001693f9cfe34@mail.gmail.com> Message-ID: <204BD979E361E64DAF7FAC6996884BA629FD87@tsg-srv-03.sleeter.com> Thanks guys. That worked. Not sure why it wasn't picking it up correctly. Anyways, thanks again. Chris -----Original Message----- From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of Eden Li Sent: Thursday, September 07, 2006 9:05 AM To: backgroundrb-devel at rubyforge.org Subject: Re: [Backgroundrb-devel] Setting Product Env I ran into this problem when my config/database.yml defined no development database. I fixed it by prepending the start command with RAILS_ENV=production ala RAILS_ENV=production script/backgroundrb/start -d I don't know if this will fix your problem since you mention that you're setting RAILS_ENV already in config/environment.rb, but this worked for me. From: "Chris Bruce" > Subject: Re: [Backgroundrb-devel] Setting Product Env > To: > Message-ID: > <204BD979E361E64DAF7FAC6996884BA629FD60 at tsg-srv-03.sleeter.com> > Content-Type: text/plain; charset="iso-8859-1" > > I double and triple checked that. Not sure how it is getting development. Do you know of any way to force it? > > > > Chris > > > ________________________________________ > From: backgroundrb-devel-bounces at rubyforge.org [mailto:backgroundrb-devel-bounces at rubyforge.org] On Behalf Of Jonathan M?tillon > Sent: Wednesday, September 06, 2006 1:08 AM > To: Backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] Setting Product Env > > Chris, > > Maybe you are using version control and when you deploy to your server, you don't update your newly checked out version so it uses production. That's what happened to me. For example in my deploy recipe I have: > > run "cat #{release_path}/config/backgroundrb.yml | sed 's#development#production#g' > #{release_path}/config/backgroundrb.yml" > > Just before the BackgrounDRb restart: > > sudo "#{release_path}/script/backgroundrb/restart" > On 9/6/06, Chris Bruce wrote: > > When I am using my rails models in my worker class, it appears the > background worker class is always using development no matter what I do. > I set background.yml to production.??I even have ENV['RAILS_ENV'] ||= > 'production uncommented in environment.rb.??I don't have the development > db even created on my production box, so not sure where it is getting > development from.??I am using fastcgi and have the environment correctly > specified in my fastcgi config. > > Any ideas? > > > > Thanks, > > > Chris _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel From cbruce at sleeter.com Thu Sep 7 12:55:40 2006 From: cbruce at sleeter.com (Chris Bruce) Date: Thu, 7 Sep 2006 09:55:40 -0700 Subject: [Backgroundrb-devel] Monitor Script Message-ID: <204BD979E361E64DAF7FAC6996884BA629FD8A@tsg-srv-03.sleeter.com> I was thinking it would be nice to have an extra script to monitor the process and restart it if it wasn't running. Basically a watchdog script that you can run as a cron job if you want. I can put one together if someone hasn't already? Chris From ezmobius at gmail.com Thu Sep 7 13:06:41 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 7 Sep 2006 10:06:41 -0700 Subject: [Backgroundrb-devel] Worker suicide In-Reply-To: References: <3a5e51050609060231t5e9c2947x61bc8e1f12a9b68@mail.gmail.com> Message-ID: <00D13949-30F8-41D6-A6FD-4AC2AB553353@gmail.com> Yes. I'm sorry for the confusion about that. -Ezra On Sep 7, 2006, at 5:17 AM, Jonathan M?tillon wrote: > And does > > ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key > > actually kill the thread? > > > On 9/6/06, Ezra Zygmuntowicz < ezmobius at gmail.com> wrote: > > On Sep 6, 2006, at 5:00 AM, Jonathan M?tillon wrote: > >> I wonder why Ezra proposed the delete_worker method if a kill >> method exists? Maybe it's a brand new feature? i was not aware of >> it and found nothing about this while googling. >> >> On 9/6/06, Chris Roos wrote: >> There is also a kill method for a worker, so I believe (and it seems >> to work) that you can just use kill (or self.kill) in place of the >> above chain. >> >> The :: references the top level namespace. >> >> Chris >> >> > > There is a kill method you can call in your worker. I need to > update it however because while it does delete the worker out of > the @jobs and @timestamps hashes, it doesnt actualy kill the > running thread. There will be a new release soon that addresses > this issue. > > > -Ezra > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060907/54b84fd8/attachment.html From ezmobius at gmail.com Thu Sep 7 13:16:24 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 7 Sep 2006 10:16:24 -0700 Subject: [Backgroundrb-devel] Monitor Script In-Reply-To: <204BD979E361E64DAF7FAC6996884BA629FD8A@tsg-srv-03.sleeter.com> References: <204BD979E361E64DAF7FAC6996884BA629FD8A@tsg-srv-03.sleeter.com> Message-ID: <33A935CA-8CBB-4A51-89A8-800DDC049AE5@gmail.com> On Sep 7, 2006, at 9:55 AM, Chris Bruce wrote: > I was thinking it would be nice to have an extra script to monitor the > process and restart it if it wasn't running. Basically a watchdog > script that you can run as a cron job if you want. > > I can put one together if someone hasn't already? > > > Chris That would be great if you wanted to come up with something. I use runit or daemontools to monitor bgdrb on my servers but a pure ruby daemon to do the same would be nice. -Ezra From rails at advany.com Thu Sep 7 13:18:50 2006 From: rails at advany.com (Abdur-Rahman Advany) Date: Thu, 07 Sep 2006 19:18:50 +0200 Subject: [Backgroundrb-devel] Impossible to get stable? In-Reply-To: <8715E247-7DF9-46FE-8AC3-43BCA608A291@gmail.com> References: <284E7D6B-A5E6-48D1-B912-D7CFFB23DD32@contuitive.com> <1FD6BA47-DD3A-4A1C-98FA-0E17BA8B391D@gmail.com> <2D2D37F4-5FE5-4C96-9A3C-A69C64F8D3AA@gmail.com> <8715E247-7DF9-46FE-8AC3-43BCA608A291@gmail.com> Message-ID: <4500547A.50103@advany.com> Hey Eric, Would you mind sharing this code with me? I am experiencing the same problems... Abdul Ezra Zygmuntowicz wrote: > Hey Eric- > > Yeah I think your approach is valid. Would you mind sending me the > code you used for this experiment? I would like to run some tests > myself. I will add some unit tests to try and see what the upper > limits of threaded AR usage is. > > Thanks- > -Ezra > > On Aug 20, 2006, at 5:20 AM, Erik Morton wrote: > > >> I'm planning on using Backgroundrb to run background processes for a >> fairly large all-Rails application that is launching in two weeks. >> I've been getting a bit paranoid about how ActiveRecord would react >> with possibly as many as twenty workers chugging away at the same >> time on the same tables. I put together a dead-simple "test" and the >> results were, I think, very positive. (The OS is OSX 10.4.7) >> >> Setup: One worker class that queries a MySQL database (4.1.18) n >> times and then calls ActiveRecord::Base.connection.disconnect! and >> kill. The worker executes the "SHOW PROCESSLIST" query, which returns >> a list of active MySQL processes, and logs the number of active >> connections along with the to_s of the connection and the thread. >> >> The test script uses MiddleMan to instantiate 100 workers passing in >> the worker's number as the number (10, 11, 12...110) of times the >> worker should loop and query the database. I took this approach so >> that the workers would be closing connections while other threads had >> connections open. >> >> Goals: 1) verify that each Thread has his own connection. 2) verify >> that each worker can call disconnect! to close their own connection >> to free it up for other workers. >> >> Here's a snippet of the log file. >> >> worker number, ActiveRecord::Base.object_id, >> ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active >> MySQL connections >> 25, 3705192, #> 0x277ba04>, #, 12 >> 21, 3705192, #> 0x277a730>, #, 12 >> 18, 3705192, #> 0x26c86e8>, #, 12 >> 17, 3705192, #> 0x26d61d0>, #, 12 >> 16, 3705192, #> 0x26e06d0>, #, 12 >> 15, 3705192, #> 0x26e7098>, #, 12 >> 23, 3705192, #> 0x2784f8c>, #, 11 >> >> In this rather unscientific way I have positively confirmed (I think) >> that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4, each worker gets >> his own thread with his own connection, and can freely close his >> connection without adversely affecting any other thread. Any thoughts >> on this approach? >> >> I made one small change to swap out the SHOW PROCESS list query with >> MyModel.find :first and the results were the same -- each worker had >> his own thread with his own connection that he can close without >> affecting any other threads. >> >> Finally, I altered the start script to set allow concurrency to >> false, ran the test and not surprisingly all heck broke loose: >> >> 39, 3705192, #> 0x26fdca8>, #, 1 >> 25, 3705192, #> 0x26fdca8>, #, 1 >> 86, 3705192,#> 0x26fdca8>,#, >> 54, 3705192, #> 0x26fdca8>, #, 1 >> 87, 3705192,#> 0x26fdca8>,#, >> Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; - >> (ActiveRecord::StatementInvalid) >> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/ >> active_record/connection_adapters/abstract_adapter.rb:120:in `log' >> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/ >> active_record/connection_adapters/mysql_adapter.rb:184:in `execute' >> >> Different threads but the same connection and the first worker that >> closes his connection ruins the whole party. >> >> Are results like the above consistent with the intention of >> Backgroundrb? Is my approach was at least somewhat valid? >> >> Erik >> >> On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote: >> >> >>> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote: >>> >>> >>>> I have concluded that it is impossible to get this background >>>> process stable. I am out of ideas and could really use some help. >>>> >>>> Here is what I have: >>>> >>>> 50 workers, just running an infinite loop that constantly calls >>>> save! on a model. Keep in mind this is just for testing purposes. >>>> I catch all exceptions and put them in the log. >>>> >>>> I am tried setting allow_concurrency to true and false. Neither of >>>> them is stable. For both true and false I have got the following >>>> errors: >>>> >>>> #>>> exceeded; try restarting transaction: >>>> >>>> #>>> gone away: >>>> >>>> What have you found to be most stable? Setting allow concurrency >>>> to false and just putting a Mutex.synchronize object around all >>>> database calls? Or setting it true? >>>> >>>> Thank You, >>>> Ben Johnson >>>> E: bjohnson at contuitive.com >>>> O: 800-341-6826 >>>> M: 817-229-4258 >>>> >>> Hey Ben- >>> >>> I feel your pain. I have come to the conclusion that ActiveRecord >>> is just not thread safe no matter what you do. My best suggestion >>> is to use either dbi or mysql bindings directly in your workers. >>> These are thread safe and while you would have to write some sql >>> yourself, it would solve your issue. I get the feeling that >>> railscore is not very interested in making rails thread safe at >>> this point and pushing as many db connections in as many threads as >>> you are doing is more the ActiveRecord can handle. >>> >>> I know its not what you want to hear but its the situation we have >>> to deal with. >>> >>> -Ezra >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > From ezmobius at gmail.com Thu Sep 7 13:31:16 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 7 Sep 2006 10:31:16 -0700 Subject: [Backgroundrb-devel] Impossible to get stable? In-Reply-To: <4500547A.50103@advany.com> References: <284E7D6B-A5E6-48D1-B912-D7CFFB23DD32@contuitive.com> <1FD6BA47-DD3A-4A1C-98FA-0E17BA8B391D@gmail.com> <2D2D37F4-5FE5-4C96-9A3C-A69C64F8D3AA@gmail.com> <8715E247-7DF9-46FE-8AC3-43BCA608A291@gmail.com> <4500547A.50103@advany.com> Message-ID: <7F1BF0AD-9A08-4743-B56E-3FCD1119EF9F@gmail.com> Hey Abdur- Here is the code that he sent me when I requested it. -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrbtests.tar.gz Type: application/x-gzip Size: 1350 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060907/0e6e75bd/attachment-0001.gz -------------- next part -------------- -Ezra On Sep 7, 2006, at 10:18 AM, Abdur-Rahman Advany wrote: > Hey Eric, > > Would you mind sharing this code with me? I am experiencing the same > problems... > > Abdul > > Ezra Zygmuntowicz wrote: >> Hey Eric- >> >> Yeah I think your approach is valid. Would you mind sending me the >> code you used for this experiment? I would like to run some tests >> myself. I will add some unit tests to try and see what the upper >> limits of threaded AR usage is. >> >> Thanks- >> -Ezra >> >> On Aug 20, 2006, at 5:20 AM, Erik Morton wrote: >> >> >>> I'm planning on using Backgroundrb to run background processes for a >>> fairly large all-Rails application that is launching in two weeks. >>> I've been getting a bit paranoid about how ActiveRecord would react >>> with possibly as many as twenty workers chugging away at the same >>> time on the same tables. I put together a dead-simple "test" and the >>> results were, I think, very positive. (The OS is OSX 10.4.7) >>> >>> Setup: One worker class that queries a MySQL database (4.1.18) n >>> times and then calls ActiveRecord::Base.connection.disconnect! and >>> kill. The worker executes the "SHOW PROCESSLIST" query, which >>> returns >>> a list of active MySQL processes, and logs the number of active >>> connections along with the to_s of the connection and the thread. >>> >>> The test script uses MiddleMan to instantiate 100 workers passing in >>> the worker's number as the number (10, 11, 12...110) of times the >>> worker should loop and query the database. I took this approach so >>> that the workers would be closing connections while other threads >>> had >>> connections open. >>> >>> Goals: 1) verify that each Thread has his own connection. 2) verify >>> that each worker can call disconnect! to close their own connection >>> to free it up for other workers. >>> >>> Here's a snippet of the log file. >>> >>> worker number, ActiveRecord::Base.object_id, >>> ActiveRecord::Base.connection.to_s, Thread.current.to_s, # of active >>> MySQL connections >>> 25, 3705192, #>> 0x277ba04>, #, 12 >>> 21, 3705192, #>> 0x277a730>, #, 12 >>> 18, 3705192, #>> 0x26c86e8>, #, 12 >>> 17, 3705192, #>> 0x26d61d0>, #, 12 >>> 16, 3705192, #>> 0x26e06d0>, #, 12 >>> 15, 3705192, #>> 0x26e7098>, #, 12 >>> 23, 3705192, #>> 0x2784f8c>, #, 11 >>> >>> In this rather unscientific way I have positively confirmed (I >>> think) >>> that on Mac OSX 10.4.7, MySQL 4.1.18 and AR 1.14.4, each worker >>> gets >>> his own thread with his own connection, and can freely close his >>> connection without adversely affecting any other thread. Any >>> thoughts >>> on this approach? >>> >>> I made one small change to swap out the SHOW PROCESS list query with >>> MyModel.find :first and the results were the same -- each worker had >>> his own thread with his own connection that he can close without >>> affecting any other threads. >>> >>> Finally, I altered the start script to set allow concurrency to >>> false, ran the test and not surprisingly all heck broke loose: >>> >>> 39, 3705192, #>> 0x26fdca8>, #, 1 >>> 25, 3705192, #>> 0x26fdca8>, #, 1 >>> 86, 3705192,#>> 0x26fdca8>,#, >>> 54, 3705192, #>> 0x26fdca8>, #, 1 >>> 87, 3705192,#>> 0x26fdca8>,#, >>> Mysql::Error: MySQL server has gone away: SHOW PROCESSLIST; - >>> (ActiveRecord::StatementInvalid) >>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/ >>> active_record/connection_adapters/abstract_adapter.rb:120:in `log' >>> /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/ >>> active_record/connection_adapters/mysql_adapter.rb:184:in `execute' >>> >>> Different threads but the same connection and the first worker that >>> closes his connection ruins the whole party. >>> >>> Are results like the above consistent with the intention of >>> Backgroundrb? Is my approach was at least somewhat valid? >>> >>> Erik >>> >>> On Aug 20, 2006, at 12:39 AM, Ezra Zygmuntowicz wrote: >>> >>> >>>> On Aug 18, 2006, at 6:20 PM, Ben Johnson wrote: >>>> >>>> >>>>> I have concluded that it is impossible to get this background >>>>> process stable. I am out of ideas and could really use some help. >>>>> >>>>> Here is what I have: >>>>> >>>>> 50 workers, just running an infinite loop that constantly calls >>>>> save! on a model. Keep in mind this is just for testing purposes. >>>>> I catch all exceptions and put them in the log. >>>>> >>>>> I am tried setting allow_concurrency to true and false. Neither of >>>>> them is stable. For both true and false I have got the following >>>>> errors: >>>>> >>>>> #>>>> exceeded; try restarting transaction: >>>>> >>>>> #>>>> gone away: >>>>> >>>>> What have you found to be most stable? Setting allow concurrency >>>>> to false and just putting a Mutex.synchronize object around all >>>>> database calls? Or setting it true? >>>>> >>>>> Thank You, >>>>> Ben Johnson >>>>> E: bjohnson at contuitive.com >>>>> O: 800-341-6826 >>>>> M: 817-229-4258 >>>>> >>>> Hey Ben- >>>> >>>> I feel your pain. I have come to the conclusion that ActiveRecord >>>> is just not thread safe no matter what you do. My best suggestion >>>> is to use either dbi or mysql bindings directly in your workers. >>>> These are thread safe and while you would have to write some sql >>>> yourself, it would solve your issue. I get the feeling that >>>> railscore is not very interested in making rails thread safe at >>>> this point and pushing as many db connections in as many threads as >>>> you are doing is more the ActiveRecord can handle. >>>> >>>> I know its not what you want to hear but its the situation we have >>>> to deal with. >>>> >>>> -Ezra >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >>> >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Fri Sep 8 00:11:22 2006 From: gethemant at gmail.com (hemant) Date: Fri, 8 Sep 2006 09:41:22 +0530 Subject: [Backgroundrb-devel] disable logging messages to development.log Message-ID: All the active record....read and writes....from backgroundrb gets logged into development.log files..created by rails...is there any way to avoid this? Its driving me nuts....because once bulk read/write from backgroundrb starts i am not able to trace my normal rails logs. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060908/40d5cbb0/attachment.html From jmetillon at gmail.com Fri Sep 8 03:37:07 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Fri, 8 Sep 2006 09:37:07 +0200 Subject: [Backgroundrb-devel] Breakpointer and BackgrounDRb Message-ID: The last question from Hemant raised this one for me: Is there a way to use the breakpointer with BackgrounDRb? I tried to set breakpoints inside of a worker but nothing happens in the breakpointer. As they are two different standalone servers, maybe they can't communicate? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060908/79bc64fc/attachment.html From wborgon at gmail.com Fri Sep 8 15:22:36 2006 From: wborgon at gmail.com (Wolfgang Borgon) Date: Fri, 8 Sep 2006 12:22:36 -0700 Subject: [Backgroundrb-devel] Access worker outside of Rails? Message-ID: <63edcbc90609081222v7555f478ta184883fe3479b80@mail.gmail.com> Hi all, Wondering if it's possible to access a worker from outside of Rails -- I've tried simply including: require RAILS_ROOT + '/vendor/plugins/backgroundrb/backgroundrb.rb' BackgrounDRb.MiddleMan.get_worker(key) but, I get an error that the 'get_worker' method is undefined. Any ideas? Thanks a lot, W -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060908/2d4dc172/attachment.html From joshuabates at gmail.com Fri Sep 8 16:36:10 2006 From: joshuabates at gmail.com (Joshua Bates) Date: Fri, 8 Sep 2006 14:36:10 -0600 Subject: [Backgroundrb-devel] Passing blocks to a worker Message-ID: I'm trying to implement a callback system between my controller, and a worker. The goal being that I could send to the client via Juggernaut when an action takes place in the worker. Something like this. worker.subscribe :action do render :update, :juggernaut => "channel" do |page| .... end end There seems to be a couple problems with doing this. First off, the binding of the block that gets sent from the controller is no longer around by the time the block is called from the worker. I could get around that one, but I'm not able to get a block to run even when trying to call it with instance_eval. Is anything like this doable, or am I just barking up the wrong tree? joshua -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060908/fa7eaaa3/attachment.html From ezmobius at gmail.com Fri Sep 8 17:24:28 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 8 Sep 2006 14:24:28 -0700 Subject: [Backgroundrb-devel] Access worker outside of Rails? In-Reply-To: <63edcbc90609081222v7555f478ta184883fe3479b80@mail.gmail.com> References: <63edcbc90609081222v7555f478ta184883fe3479b80@mail.gmail.com> Message-ID: <52965518-E0BC-4910-A386-46987A435FC2@gmail.com> On Sep 8, 2006, at 12:22 PM, Wolfgang Borgon wrote: > Hi all, > Wondering if it's possible to access a worker from outside of Rails > -- I've tried simply including: > > require RAILS_ROOT + '/vendor/plugins/backgroundrb/backgroundrb.rb' > BackgrounDRb.MiddleMan.get_worker (key) > > but, I get an error that the 'get_worker' method is undefined. Any > ideas? > Thanks a lot, > W To use it from outside of rails you need to start up a drb client in your script. So soemthing like this: #!/usr/bin/env ruby require "drb" DRb.start_service MiddleMan = DRbObject.new(nil, "druby://localhost:22222") MiddleMan.get_worker(:somekey) Cheers- -Ezra From bill.walton at charter.net Tue Sep 12 10:45:08 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 12 Sep 2006 09:45:08 -0500 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> Message-ID: <041c01c6d67a$0b202cd0$6401a8c0@Presario> Hi Ezra, Continuing our conversation from the Rails list. Ezra Zygmuntowicz wrote: > Bill Walton wrote: > > ... but could it work like.... > > > > 1) from the rails app, start a worker for a session > > with a timeout setting and a controller/action argument > > 2) every time the visitor interacts with the rails app, > > reset the worker's timeout > > 3) when the timeout occurs, the worker invokes the > > controller/action in the rails app that was passed in at > > startup. > Something like this could work. The main issue is that > backgroundrb doesn't use http so it can't really call a > controller method in your app for you. You could just > put a similar action in your worker class and call it there. I'm not sure I'm understanding you here. Are you saying put the 'cleanup' action that deletes the user-entered records and files in the worker? If so, doesn't that go against the DRY principle? Or are you saying to move it from where it is now (in one of the app's controllers) into a worker? I'm definitely confused by your statement that "backgroundrb doesn't use http so it can't really call a controller method in your app". I understood that workers could respond to Ajax requests. Did I misunderstand? To my limited understanding, if a worker's job is to respond to Ajax requests, that means I can think of a backgroundrb worker as just another of the app's controllers, albeit a 'special' one because it's running in parallel. Am I 'courting danger' with this simplistic view? Now that we've talked about it a little, I guess if it *is* ok to think about a worker as one of the app's controllers, then the worker really is where I want to put the cleanup code. I'm also assuming that a worker 'knows' what session it's part of. Is that right? Does any of the way I'm thinking about this tell you I'm heading off 'into the weeds'? > Ok all that being said, BackgrounDRb can do the same > thing for you. [...] If you decide to go the BackgrounDRb > route then please join the mailing list for it and I'll do my > best to help you along. Well.... here I am! ;-) And thank you. You may yet convince me that the script/cron route is the way to go. But first, I need to better understand what you're telling me. In the interest of full disclosure, until finding RoR around the first of this year, I've not done any serious development in a dozen years. I've been managing it, but not doing it. A lot has changed and I'm finding there are lots of details I need to learn. Best regards, Bill *Mailing list: http://rubyforge.org/mailman/listinfo/backgroundrb-devel Cheers- -Ezra --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. To post to this group, send email to rubyonrails-talk at googlegroups.com To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe at googlegroups.com For more options, visit this group at http://groups.google.com/group/rubyonrails-talk -~----------~----~----~----~------~----~------~--~--- From ezmobius at gmail.com Tue Sep 12 14:56:49 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 12 Sep 2006 11:56:49 -0700 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? In-Reply-To: <041c01c6d67a$0b202cd0$6401a8c0@Presario> References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> Message-ID: Hi Bill~ On Sep 12, 2006, at 7:45 AM, Bill Walton wrote: > Hi Ezra, > > Continuing our conversation from the Rails list. Welcome ;) > > Ezra Zygmuntowicz wrote: > > > I'm not sure I'm understanding you here. Are you saying put the > 'cleanup' > action that deletes the user-entered records and files in the > worker? If > so, doesn't that go against the DRY principle? Or are you saying > to move it > from where it is now (in one of the app's controllers) into a worker? > > I'm definitely confused by your statement that "backgroundrb > doesn't use > http so it can't really call a controller method in your app". I > understood > that workers could respond to Ajax requests. Did I misunderstand? > > To my limited understanding, if a worker's job is to respond to Ajax > requests, that means I can think of a backgroundrb worker as just > another of > the app's controllers, albeit a 'special' one because it's running in > parallel. Am I 'courting danger' with this simplistic view? > > Now that we've talked about it a little, I guess if it *is* ok to > think > about a worker as one of the app's controllers, then the worker > really is > where I want to put the cleanup code. I'm also assuming that a worker > 'knows' what session it's part of. Is that right? > > Does any of the way I'm thinking about this tell you I'm heading > off 'into > the weeds'? Yes you have a few incorrect notions, let's see if I can clear things up for you a bit. Your backgroundrb worker are not rails controllers at all and they do not directly respond to ajax calls. In your rails app you have access to the MiddleMan object that you can use to start workers in the drb server and ping them for results later during the ajax requests. But the http requests have to come into a normal rails controller, the in a controller action you can use MiddleMan to grab a handle on your remote worker object and call methods or get vars from it. So yes if you are going to be using backgroundrb to do this cleanup project I would move the cleanup code into a worker class and remove it from your main app. Then in your rails app when you need to call the cleanup worker, you use the MiddleMan to do so. And then in the cleanup worker you could have a timer that wakes up every X minutes and does the cleanup for users who left their session dangling. So a worker class might look like this: class CleanupWorker < BackgrounDRb::Rails repeat_every 15.minutes first_run Time.now def do_work(args) # pull the id out of args and do the cleanup # or if there is no args then just look for session older then X minutes and clean them up. end end Then you would want to autostart this worker so it runs on boot up. So you would add this to your backgroundrb.yml config file: autostart: 1: job_key: cleanup class: cleanup_worker And then to access this from your rails app when someone logs out explicitely # in a controller or model method def cleanup(user_id) MiddleMan.new_worker :class => :cleanup_worker, :args => user_id, :ttl => 120 end So now when you call that action, the drb server will create a new cleaup worker and pass in the user_id so you can cleanup after that user. > > > >> Ok all that being said, BackgrounDRb can do the same >> thing for you. [...] If you decide to go the BackgrounDRb >> route then please join the mailing list for it and I'll do my >> best to help you along. > > Well.... here I am! ;-) And thank you. You may yet convince me > that the > script/cron route is the way to go. But first, I need to better > understand > what you're telling me. In the interest of full disclosure, until > finding > RoR around the first of this year, I've not done any serious > development in > a dozen years. I've been managing it, but not doing it. A lot has > changed > and I'm finding there are lots of details I need to learn. > > Best regards, > Bill So IU hope that explains it a bit better. You could go either way with this. BackgrounDRb will certainly work for what you need to do. But since this is more like a fire and forget type of job, it is less moving parts to just use a cron job and a script/runner script that fires the cleanup code. Its up to you which direction to take. I'll try to help as much as I can. Cheers- -Ezra From bill.walton at charter.net Wed Sep 13 10:38:36 2006 From: bill.walton at charter.net (Bill Walton) Date: Wed, 13 Sep 2006 09:38:36 -0500 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> Message-ID: <066d01c6d742$4c42d6e0$6401a8c0@Presario> Hi Ezra, Ezra Zygmuntowicz wrote: >> Continuing our conversation from the Rails list. > > Welcome ;) Thanks! >> Does any of the way I'm thinking about this tell you I'm heading off >> 'into the weeds'? > > Yes you have a few incorrect notions, I was afraid of that ;-) > let's see if I can clear things up for you a bit. I really appreciate it. I should warn you though. Sometimes even *I* am surprised by my density ;-) > Your backgroundrb worker are not rails controllers at all and they do not > directly respond to ajax calls. In your rails app you have access to the > MiddleMan object that you can use to start workers in the drb server and > ping them for results later during the ajax requests. If I can ping the worker from the controller and they can respond and my controller can receive and process their response, and they're sitting out there running and waiting to respond, I'm not getting why they couldn't *initiate* a 'conversation'. Is this a 'we don't have a tool built to do that' situation? Or is it 'the Rails architecture doesn't allow for that kind of interprocess communication' situation? > So IU hope that explains it a bit better. I have a much better understanding of how to use BackgrounDRb. Thank you. But I'm still confused ;-( I think, maybe, my confusion comes from associating (in my mind) BackgrounDRb and dRuby (drb) as discussed in "Programming Ruby". The DRb class description (p.670) says "Although expressed in terms of clients and servers, once the initial connection is established, the protocol is effectively symmetrical: either side can invoke methods in objects on the other side." Could you shed some light on this for me? Thanks! Bill From info at siebert-wd.de Wed Sep 13 14:13:10 2006 From: info at siebert-wd.de (Michael Siebert) Date: Wed, 13 Sep 2006 20:13:10 +0200 Subject: [Backgroundrb-devel] A new QueueWorker class In-Reply-To: <44ED2E82.8070800@lemstra.ca> References: <44ED2E82.8070800@lemstra.ca> Message-ID: <32b567370609131113n30a5fc1bwb09ea30e705cc8d@mail.gmail.com> Hey David, I'm currently trying out your QueueWorker and I am experiencing a little problem: It seems the worker (my worker) is never started. Just to make sure I got it right: MiddleMan[:queue].queue_job({ :in_filename => self.getFullPath, :out_filename => self.getPreviewPath, :width => 300, :height => 600 } ) This should put another instance of my worker into the queue and run it, if there are enough threads "free". is that how you meant it? I modified the configuration a little bit to fit my needs: autostart: :queue: class: queue_worker args: :num_child_workers: 1 :child_class: :generate_preview_worker :reQ_on_finish: false :singleton: true Any ideas what i might be doing wrong? regards, Micha 2006/8/24, David Lemstra : > > Hello all, > I've come up w/ a worker class that manages queued jobs using a fixed > number of child workers. Well, that's not quite true -- a new worker is > spawned for each job, but you set the total number that may exist at once. > > There are three components: > 1) queue_worker.rb: The singleton worker that manages the child workers. > You probably want to auto start this. Make sure you give it > :singleton=>true. Read this file for the methods to interact with your > children. (ie. queue_job(), delete_job(), job_progress() ) > > 2) backgroundrb_rails_queue.rb: The super class for the "child workers" > (and uses backgroundrb_rails.rb in turn). This file needs to be included > in background.rb > > 3) Your child worker, which should be a subclass of > BackgrounDRb::RailsQueue, is otherwise the about same as normal > If it's a big loop, you probably want to use terminate?() on each > iteration and update @progress. Use suicide() at the end to make room > for the next child. > > Options: (probably in your backgroundrb.yml) > autostart: > :queue_key: > class: queue_worker > args: > :num_child_workers: 2 > :child_class: :cost_calculator_worker > :reQ_on_finish: true > :singleton: true > > :queue_key can be changed to what you want, but it is the permanent key > of the QueueWorker > :num_child_workers: is up to you! > :child_class: your worker class you want as child workers. > :reQ_on_finish: do you want results to be stored in the queue until you > call job_progress!() ? > Note: to be able to access your child jobs w/ the QueueWorker methods, > include a unique :id in your {args} when you queue_job({args}) > > I'll attach the files. If they don't go through, I'll resend as text. > > BTW, This works well enough for me, but I'm learning as I go too, so no > guarantees :) I don't use the fancy timing options, so ymmmv for > :next_start and :interval. > > Let me know if you find any issues (though I'm off-line for a week after > this post). I'm wondering myself if it might be better to reuse child > workers instead of re-spawning new ones. Another day maybe. > > cheers, > David Lemstra > > > # Put your code that runs your task inside the do_work method > # it will be run automatically in a thread. You have access to > # all of your rails models if you set load_rails to true in the > # config file. You also get @logger inside of this class by default. > require 'monitor.rb' > > class QueueWorker < BackgrounDRb::Rails > > attr_reader :q, :id_hash, :completed > def initialize(key, args={}) > super(key,args) > @num_child_workers = args[:num_child_workers] ? > args[:num_child_workers] : 1 > @child_workers = Array.new(@num_child_workers) {|i| > Hash[:job_key,nil,:args,nil,:s_thread,nil, :s_mutex,Mutex.new, :child, i] > } > > @q = [] > @q.extend(MonitorMixin) > @q_loaded_cv = @q.new_cond > @id_hash = {} > @id_hash_mutex = Mutex.new > > raise ArgumentError unless args.has_key?(:child_class) > @child_class = args[:child_class] > @reQ_on_finish = args[:reQ_on_finish] || false > @completed = 0 > end > > def queue_job(args) > return nil if @id_hash && args[:id] && @id_hash.has_key?(args[:id]) > @q.synchronize do > @q.push args > @id_hash_mutex.synchronize { @id_hash[args[:id]] = > {:status=>:queued, :job_key=>nil, :results=>nil } } if args[:id] > @q_loaded_cv.signal > end > return true > end > > def job_in_progress?(job_id) > @id_hash.has_key?(job_id) > end > > def job_status?(job_id) > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > return @id_hash[job_key][:status] > end > end > > def job_progress(job_id) > report_hsh = {} > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > report_hsh[:status] = @id_hash[job_id][:status] > report_hsh[:progress] = case @id_hash[job_id][:status] > when :queued then > ahead = 0 > @q.each_index {|i| if @q[i][:id] == job_id then ahead = i; > break; end } > ahead > when :running > w = self[@id_hash[job_id][:job_key]] > w.nil? ? nil : w.progress > when :done > @id_hash[job_id][:results] > else nil > end > end > return report_hsh > end > > def job_progress!(job_id) > report_hsh = {} > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > report_hsh[:status] = @id_hash[job_id][:status] > report_hsh[:progress] = case @id_hash[job_id][:status] > when :queued then > ahead = 0 > @q.each_index {|i| if @q[i][:id] == job_id then ahead = i; > break; end } > ahead > when :running > w = self[@id_hash[job_id][:job_key]] > if w.nil? > @id_hash.delete(job_id)[:results] > else > w.progress > end > when :done > @id_hash.delete(job_id)[:results] > else nil > end > end > return report_hsh > end > > def delete_job(job_id) > args = nil > @q.synchronize do @id_hash_mutex.synchronize do > args = @id_hash[job_id] > return true if args.nil? > if args[:status] == :queued > @q.delete_if {|h| h[:id] == job_id } > @id_hash.delete(job_id) > return true > elsif args[:status] == :done > @id_hash.delete(job_id) > return true > end > end > end > ::BackgrounDRb::MiddleMan.instance.delete_worker(args[:job_key]) if > args[:status] == :running > return true > end > > def do_work(args) > # You probably don't want to mess with this method unless you know > what's what. > @child_workers.each do |child_hash| > child_hash[:s_thread] = Thread.start do > loop do > # Wait for a new job in the @q > child_hash[:s_mutex].synchronize do > # get the Q mutex and wait for a job > @q.synchronize do > tl = Thread.list > @q_loaded_cv.wait_while { @q.empty? } > child_hash[:args] = @q.shift > if child_hash[:args][:id] > @id_hash_mutex.synchronize do > @id_hash[child_hash[:args][:id]][:status] = :running > child_hash[:job_key] = > spawn_worker({:args=>child_hash[:args],:class=>@child_class}) > @id_hash[child_hash[:args][:id]][:job_key] = > child_hash[:job_key] > end > else > child_hash[:job_key] = > spawn_worker(job_args.merge(:class=>@child_class)) > end > end > > self[child_hash[:job_key]].thread[:DQ_request].wait(child_hash[:s_mutex]) > # grab and store the results > if child_hash[:args][:id] > @id_hash_mutex.synchronize do > if @reQ_on_finish > r = self[child_hash[:job_key]].results > @id_hash[child_hash[:args][:id]][:results] = r if r > @id_hash[child_hash[:args][:id]][:status] = :done > @id_hash[child_hash[:args][:id]][:done_at] = Time.now > else > @id_hash.delete(child_hash[:args][:id]) > end > end > end > self[child_hash[:job_key]].thread[:DQed].signal > @completed += 1 > [:args,:job_key].each {|k| child_hash[k] = nil } > end > # Loop back and wait for the job_key to get killed again.... > end > end > end > end > > private > > def [](key) > # Use jobs to avoid the access time update w/ [] > ::BackgrounDRb::MiddleMan.instance.jobs[key] > end > > end > > > module BackgrounDRb > > class RailsQueue < BackgrounDRb::Rails > attr_reader :progress > def initialize(key, args={}) > super(key,args) > @job_ctrl = true > end > > def start_process > return if schedule_first_run && schedule_first_run.to_i > > Time.now.to_i > @thread = Thread.new do > Thread.current[:safe_to_kill] = ConditionVariable.new > Thread.current[:kill] = false > Thread.current[:DQ_request] = ConditionVariable.new > Thread.current[:DQed] = ConditionVariable.new > Thread.current[:mutex] = Mutex.new > begin > Thread.current[:mutex].synchronize do > do_work(@args) > end > rescue Exception => e > @logger.error "#{ e.message } - (#{ e.class })" << "\n" << ( > e.backtrace or []).join("\n") > end > end > @next_start = @interval.from_now if schedule_repeat > end > > def results > # Overwrite this method and set reQ_on_finish = true (in the queue > worker args) > # to have a process put it's results in back in the queue > # for pickup before being killed > nil > end > > def before_DQ(args=nil) > # stub method that gets called before dequeue is run. > # Overwrite in your class instance > true > end > > def terminate(args=nil) > do_DQ(args) > super(args) > end > > def suicide(args=nil) > do_DQ(args) > kill > Thread.exit > end > > private > def do_DQ(args=nil) > before_DQ(args) > Thread.current[:DQ_request].signal > Thread.current[:DQed].wait(Thread.current[:mutex]) > end > end > end > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- Michael Siebert www.siebert-wd.de - Gedanken lesen www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060913/74d55a1c/attachment-0001.html From ezmobius at gmail.com Thu Sep 14 13:00:35 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 14 Sep 2006 10:00:35 -0700 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? In-Reply-To: <066d01c6d742$4c42d6e0$6401a8c0@Presario> References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> <066d01c6d742$4c42d6e0$6401a8c0@Presario> Message-ID: On Sep 13, 2006, at 7:38 AM, Bill Walton wrote: > Hi Ezra, > > Ezra Zygmuntowicz wrote: > >>> Continuing our conversation from the Rails list. >> >> Welcome ;) > > Thanks! > >>> Does any of the way I'm thinking about this tell you I'm heading >>> off 'into the weeds'? >> >> Yes you have a few incorrect notions, > > I was afraid of that ;-) > >> let's see if I can clear things up for you a bit. > > I really appreciate it. I should warn you though. Sometimes even > *I* am surprised by my density ;-) > >> Your backgroundrb worker are not rails controllers at all and they >> do not directly respond to ajax calls. In your rails app you have >> access to the MiddleMan object that you can use to start workers >> in the drb server and ping them for results later during the ajax >> requests. > > If I can ping the worker from the controller and they can respond > and my controller can receive and process their response, and > they're sitting out there running and waiting to respond, I'm not > getting why they couldn't *initiate* a 'conversation'. Is this a > 'we don't have a tool built to do that' situation? Or is it 'the > Rails architecture doesn't allow for that kind of interprocess > communication' situation? I'm just not sure what you are trying to accomplish with this. I think you are making it way more difficult ;) The way backgroundrb works is mainly a one way thing. You kick off the workers from your controller and then you can ping the worker later from your controller to get status or results from it. If you really feel you need to call back to your controller from the bgdrb server, you could use net/http in ruby's standard library top make a http request back to the rails app. The problem with trying to talk back to rails from the drb server is that rails is not persistent. Each request instantiates a new controller object. So if you try to callback to rails from bdrb, you aren't garanteed to get the same controller object to work with. So bdrb works by letting rails control it. You kick off your worker classes with a command in your controller and then you store a job key in your sessions. Then on later requests in rails you can use this job key to grab a handle on your worker object and call methods on it for results. > > > >> So IU hope that explains it a bit better. > > I have a much better understanding of how to use BackgrounDRb. > Thank you. But I'm still confused ;-( > > I think, maybe, my confusion comes from associating (in my mind) > BackgrounDRb and dRuby (drb) as discussed in "Programming Ruby". > The DRb class description (p.670) says "Although expressed in terms > of clients and servers, once the initial connection is established, > the protocol is effectively symmetrical: either side can invoke > methods in objects on the other side." Could you shed some light > on this for me? Yes BackgrounDRb uses drb to do its work. But the way it is designed is for use with the way rails works. Its a persistent daemon that runs your jobs. Your rails controllers and models are all instantiated on each request so they don't persist and thats why you need a session. So yes its true that the way drb works is that both ends effectively become servers and clients. But because rails is a web app, and the way request/response and object instantiation in rails works, you can't really call back to rails from backgroundrb. Because the place you would be trying to call back to won't exist after its request is served. Cheers- -Ezra From ezmobius at gmail.com Thu Sep 14 13:39:11 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 14 Sep 2006 10:39:11 -0700 Subject: [Backgroundrb-devel] Status of your worker pool In-Reply-To: <32b567370609141025r4630a611j3091f5d776df61bf@mail.gmail.com> References: <32b567370609141025r4630a611j3091f5d776df61bf@mail.gmail.com> Message-ID: On Sep 14, 2006, at 10:25 AM, Michael Siebert wrote: > Hey Ezra, > how is the status of your worker pool? i need some thing like this > ASAP and i can't get David Lemstra's QueueWorker to work, so: whats > the status? do you have some "beta" lying around on your hard > drive? I dont want to spend my time on that if you're coming up > with that in a short time, so i'd be happy if I could get my hands > on some pre-release-worker-pool-thingie :) > > kind regards, > Micha > > -- > Michael Siebert > > www.siebert-wd.de - Gedanken lesen > www.stellar-legends.de - Weltraum-Browsergame im Alpha-Stadium Hey Michael- Yeah I do have a beta here. It is working very well. What I did to make it easier to refactor was to grab an older version. One that didn't have any of the Timing stuff or the Threading stuff for ConditionVariables and all that. Once I got it down to its basic format, I was able to write a nice ThreadPool class and then use that in the MiddleMan. So it is way cleaner now. the middleman holds a thread pool and does all the thread management so the workers don't have to. It made the code smaller and much more robust. Here is what I have been playing with. http://pastie.caboo.se/13279 Its not integrated with the rest of the bdrb infrastructure yet, but you can play with it. Do this: save that file as bdrb.rb and then run it like this: $ irb -r bdrb.rb You will see it run and output what it is doing while it runs. If you look at the very bottom of the file you will see where i instantiate it with a thread pool size of 3 and then try to spawn 15 workers. You can see them go into the pool and wait until other threads finish so they can get their turn to run. Let me know what you think. I like how lightweight it is right now so I am debating what features I want to put back in. In fact this may become BackgrounDRb2. That way I can make it nicer now that I know how everyone uses it. But this will maybe change the interface a little bit so I will leave the current release available for folks that already buitl a ton on top of it. I also am working on a way to have 'slaves' . This would mean that if you passed an extra flag to the new_worker method, the drb server will actually fork a new process for your worker to work in, and it will setup a heartbeat between the drb server and the slave so that when you kill the drb server, all slaves will die because they don't get the heartbeat. Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060914/eadd9360/attachment.html From bill.walton at charter.net Thu Sep 14 13:57:40 2006 From: bill.walton at charter.net (Bill Walton) Date: Thu, 14 Sep 2006 12:57:40 -0500 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> <066d01c6d742$4c42d6e0$6401a8c0@Presario> Message-ID: <00f501c6d827$45953b90$6401a8c0@Presario> Hi Ezra, Ezra Zygmuntowicz wrote: > > I'm just not sure what you are trying to accomplish with this. My bad. In 'normal' session management the goal is to shut off visitor access after a set period of time. Data removal is a secondary concern. I'm working to a requirement that says "shutting off visitor access and removing the data they've entered from the system need to happen at the same time, and both need to happen when we say they're going to happen." Sweeping the system with a cron job doesn't accomplish that. If, for example, the timeout is set for 5 minutes, and the cron job runs every five minutes, some visitors will time out in 5 minutes and some will time out in 9:59 minutes. > I think you are making it way more difficult ;) What do you think now? If there's an easier way, I'm dying to be educated! ;-) > If you really feel you need to call back to your controller from the bgdrb > server, you could use net/http in ruby's standard library top make a http > request back to the rails app. Now we're talkin'!!! I'll dig into it. > The problem with trying to talk back to rails from the drb server is that > rails is not persistent. Each request instantiates a new controller > object. I understand. I was hoping that if I started a DRb server from within the controller to listen for a call from a remote client, that Rails would somehow keep the controller 'alive' at some level. I guess not. You've given me some hope though by pointing me to the net/http library. My understanding is that Rails knows what objects to use based on a session id that gets passed as part of the message header. Is that right? If it is, then it seems like if I send a logout message with the 'right' header, Rails won't know whether or not the request actually came from the visitor clicking the logout button. I guess, in the end, what I'm trying to do is an 'authorized' session hijacking by an surrogate whose only job is to make sure everything gets cleaned up as required. If I've said anything else that doesn't make sense, please feel free to 'slap me around' a little ;-) I'll be back, with more questions I'm sure ;-), after I've studied the net/http library. Thanks! Bill From ezmobius at gmail.com Thu Sep 14 14:29:40 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 14 Sep 2006 11:29:40 -0700 Subject: [Backgroundrb-devel] Status of your worker pool In-Reply-To: <32b567370609141121s5840e5d8ndd19f8680568b966@mail.gmail.com> References: <32b567370609141025r4630a611j3091f5d776df61bf@mail.gmail.com> <32b567370609141121s5840e5d8ndd19f8680568b966@mail.gmail.com> Message-ID: <45324222-41E9-4929-8A9C-0772D56A1D6F@gmail.com> On Sep 14, 2006, at 11:21 AM, Michael Siebert wrote: > > > 2006/9/14, Ezra Zygmuntowicz : > Yeah I do have a beta here. It is working very well. What I did to > make it easier to refactor was to grab an older version. One that > didn't have any of the Timing stuff or the Threading stuff for > ConditionVariables and all that. Once I got it down to its basic > format, I was able to write a nice ThreadPool class and then use > that in the MiddleMan. So it is way cleaner now. the middleman > holds a thread pool and does all the thread management so the > workers don't have to. It made the code smaller and much more robust. > > looks nice and is exactly what i need right now. the mor i think > about your aproach of cleaning it all up, the more i agree. first > thing i thought was: "hey then my periodically one-tima-a-day- > running workers wouldnt do it any more!" but isnt it the rails way > to let the right tools do the right job? > when it comes to robustness: i have a backgroundrb processrunning > since 24th of july! cool ;) I do think it needed cleaning up, it got a bit of feature bloat I think. > > > I like how lightweight it is right now so I am debating what > features I want to put back in. In fact this may become > BackgrounDRb2. That way I can make it nicer now that I know how > everyone uses it. But this will maybe change the interface a little > bit so I will leave the current release available for folks that > already buitl a ton on top of it. > > looks very good and simple but one feature i think _must_ make it > in bgdrb2: some autostarting mechanism (both for backgroundrb and > workers). possibly some thing to hook into the rails startup - i > think init.rb would be the place of choice: check the existance of > log/bgrdb.pid, and if not, touch bgdrb.pid and start it. that could > event work when launching multiple rails instances Yeah this is a tough one. I would be great if there was a way to have it started when you start rails, but I think it will be easier to keep it separate from rails startup. But what I will do is include a set of scripts or rake tasks that first start the drb server and then start rails. Maybe put this script in /script/ and call it to start everything at once in the right order. And autostart workers will still be in there too. > > I also am working on a way to have 'slaves' . This would mean that > if you passed an extra flag to the new_worker method, the drb > server will actually fork a new process for your worker to work in, > and it will setup a heartbeat between the drb server and the slave > so that when you kill the drb server, all slaves will die because > they don't get the heartbeat. > > what is the difference between a slave and a normal worker? i guess > threading? so that thread-unsafe things can work in their own > process or am i wrong? Well yes it has to do with threads and horsepower. Sometimes you need to do really compute intense tasks in a worker like image manipulation. So instead of sharing one process with all running workers, you can make some workers that end up running in their own process to take advantage of multiple cpu's and more power from more ruby processes. I am going to integrate this so you don;t even have to think about it. You can just pass an extra flag and it will run in another proc, but you will use it the same way from the MiddleMan. > > kind regards > Micha > Cheers- -Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060914/881f64bf/attachment.html From bill.walton at charter.net Thu Sep 14 16:00:58 2006 From: bill.walton at charter.net (Bill Walton) Date: Thu, 14 Sep 2006 15:00:58 -0500 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? References: <029601c6d504$7075ed80$6401a8c0@Presario> <030301c6d59d$474713a0$6401a8c0@Presario> <6821828E-F658-442C-9B1F-0D193A135F03@gmail.com> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> <066d01c6d742$4c42d6e0$6401a8c0@Presario> <00f501c6d827$45953b90$6401a8c0@Presario> <32b567370609141131t5460dccbr9b4d6236471b1734@mail.gmail.com> Message-ID: <012701c6d838$7f4d5320$6401a8c0@Presario> Hi Michael, Michael Siebert wrote: > 2006/9/14, Bill Walton : > > I'm working to a requirement that says "shutting off visitor > > access and removing the data they've entered from the > > system need to happen at the same time, and both need > > to happen when we say they're going to happen." > unfortunately this aproach doesnt delete the data if the user > just leaves the site and doesnt come back. Exactly. > here you could use a cronjob of some sort. Which, as I noted earlier, would not satisfy the last portion of the requirement above. > every other thing would by a hackery of some type Precisely what this requirement is about protecting against. > think your problem is that you absolutely wanted to use > bgdrb and did not see other obvious solutions like what > i said... I understand that interpretation ... > but possibly thats just crap too :) I wouldn't call it crap. I would say that this is a very unusual requirement that seems to require that I take a different approach than I would under normal circumstances. > the session id is stored in a cookie and that is transfered > in the header. slight difference : In Ruby (as opposed to Rails), is there an easy way for an application to explicitly set the cookie to a specific value? I can't seem to zero in on how to do that. Thanks! Bill From david at lemstra.ca Thu Sep 14 17:41:53 2006 From: david at lemstra.ca (David Lemstra) Date: Thu, 14 Sep 2006 17:41:53 -0400 Subject: [Backgroundrb-devel] A new QueueWorker class In-Reply-To: <32b567370609131113n30a5fc1bwb09ea30e705cc8d@mail.gmail.com> References: <44ED2E82.8070800@lemstra.ca> <32b567370609131113n30a5fc1bwb09ea30e705cc8d@mail.gmail.com> Message-ID: <4509CCA1.7000006@lemstra.ca> Michael, It all looks good to me (but some error messages might help). Maybe superclass :generate_preview_worker with BackgrounDRb::Rails first and make sure it works on its own w/o the Q baggage. Maybe setting :id manually in the arg hash is needed. I'm not sure if I've tried it without. David Lemstra Michael Siebert wrote: > Hey David, > I'm currently trying out your QueueWorker and I am experiencing a little > problem: > It seems the worker (my worker) is never started. Just to make sure I > got it right: > > MiddleMan[:queue].queue_job({ > :in_filename => self.getFullPath, > :out_filename => self.getPreviewPath, > :width => 300, > :height => 600 > > } > ) > > This should put another instance of my worker into the queue and run it, > if there are enough threads "free". is that how you meant it? > I modified the configuration a little bit to fit my needs: > > autostart: > :queue: > class: queue_worker > args: > :num_child_workers: 1 > :child_class: :generate_preview_worker > :reQ_on_finish: false > :singleton: true > > Any ideas what i might be doing wrong? > > regards, > Micha > > 2006/8/24, David Lemstra >: > > Hello all, > I've come up w/ a worker class that manages queued jobs using a fixed > number of child workers. Well, that's not quite true -- a new worker is > spawned for each job, but you set the total number that may exist at > once. > > There are three components: > 1) queue_worker.rb: The singleton worker that manages the child > workers. > You probably want to auto start this. Make sure you give it > :singleton=>true. Read this file for the methods to interact with your > children. (ie. queue_job(), delete_job(), job_progress() ) > > 2) backgroundrb_rails_queue.rb: The super class for the "child workers" > (and uses backgroundrb_rails.rb in turn). This file needs to be included > in background.rb > > 3) Your child worker, which should be a subclass of > BackgrounDRb::RailsQueue, is otherwise the about same as normal > If it's a big loop, you probably want to use terminate?() on each > iteration and update @progress. Use suicide() at the end to make room > for the next child. > > Options: (probably in your backgroundrb.yml) > autostart: > :queue_key: > class: queue_worker > args: > :num_child_workers: 2 > :child_class: :cost_calculator_worker > :reQ_on_finish: true > :singleton: true > > :queue_key can be changed to what you want, but it is the permanent key > of the QueueWorker > :num_child_workers: is up to you! > :child_class: your worker class you want as child workers. > :reQ_on_finish: do you want results to be stored in the queue until you > call job_progress!() ? > Note: to be able to access your child jobs w/ the QueueWorker methods, > include a unique :id in your {args} when you queue_job({args}) > > I'll attach the files. If they don't go through, I'll resend as text. > > BTW, This works well enough for me, but I'm learning as I go too, so no > guarantees :) I don't use the fancy timing options, so ymmmv for > :next_start and :interval. > > Let me know if you find any issues (though I'm off-line for a week > after > this post). I'm wondering myself if it might be better to reuse child > workers instead of re-spawning new ones. Another day maybe. > > cheers, > David Lemstra > > > # Put your code that runs your task inside the do_work method > # it will be run automatically in a thread. You have access to > # all of your rails models if you set load_rails to true in the > # config file. You also get @logger inside of this class by default. > require 'monitor.rb ' > > class QueueWorker < BackgrounDRb::Rails > > attr_reader :q, :id_hash, :completed > def initialize(key, args={}) > super(key,args) > @num_child_workers = args[:num_child_workers] ? > args[:num_child_workers] : 1 > @child_workers = Array.new(@num_child_workers) {|i| > Hash[:job_key,nil,:args,nil,:s_thread,nil, :s_mutex,Mutex.new, > :child, i] } > > @q = [] > @q.extend(MonitorMixin) > @q_loaded_cv = @q.new_cond > @id_hash = {} > @id_hash_mutex = Mutex.new > > raise ArgumentError unless args.has_key?(:child_class) > @child_class = args[:child_class] > @reQ_on_finish = args[:reQ_on_finish] || false > @completed = 0 > end > > def queue_job(args) > return nil if @id_hash && args[:id] && @id_hash.has_key?(args[:id]) > @q.synchronize do > @q.push args > @id_hash_mutex.synchronize { @id_hash[args[:id]] = > {:status=>:queued, :job_key=>nil, :results=>nil } } if args[:id] > @q_loaded_cv.signal > end > return true > end > > def job_in_progress?(job_id) > @id_hash.has_key?(job_id) > end > > def job_status?(job_id) > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > return @id_hash[job_key][:status] > end > end > > def job_progress(job_id) > report_hsh = {} > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > report_hsh[:status] = @id_hash[job_id][:status] > report_hsh[:progress] = case @id_hash[job_id][:status] > when :queued then > ahead = 0 > @q.each_index {|i| if @q[i][:id] == job_id then ahead = i; > break; end } > ahead > when :running > w = self[@id_hash[job_id][:job_key]] > w.nil? ? nil : w.progress > when :done > @id_hash[job_id][:results] > else nil > end > end > return report_hsh > end > > def job_progress!(job_id) > report_hsh = {} > @id_hash_mutex.synchronize do > return nil unless @id_hash.has_key?(job_id) > report_hsh[:status] = @id_hash[job_id][:status] > report_hsh[:progress] = case @id_hash[job_id][:status] > when :queued then > ahead = 0 > @q.each_index {|i| if @q[i][:id] == job_id then ahead = i; > break; end } > ahead > when :running > w = self[@id_hash[job_id][:job_key]] > if w.nil? > @id_hash.delete(job_id)[:results] > else > w.progress > end > when :done > @id_hash.delete(job_id)[:results] > else nil > end > end > return report_hsh > end > > def delete_job(job_id) > args = nil > @q.synchronize do @id_hash_mutex.synchronize do > args = @id_hash[job_id] > return true if args.nil? > if args[:status] == :queued > @q.delete_if {|h| h[:id] == job_id } > @id_hash.delete(job_id) > return true > elsif args[:status] == :done > @id_hash.delete(job_id) > return true > end > end > end > ::BackgrounDRb::MiddleMan.instance.delete_worker(args[:job_key]) > if args[:status] == :running > return true > end > > def do_work(args) > # You probably don't want to mess with this method unless you > know what's what. > @child_workers.each do |child_hash| > child_hash[:s_thread] = Thread.start do > loop do > # Wait for a new job in the @q > child_hash[:s_mutex].synchronize do > # get the Q mutex and wait for a job > @q.synchronize do > tl = Thread.list > @q_loaded_cv.wait_while { @ q.empty? } > child_hash[:args] = @q.shift > if child_hash[:args][:id] > @id_hash_mutex.synchronize do > @id_hash[child_hash[:args][:id]][:status] = :running > child_hash[:job_key] = > spawn_worker({:args=>child_hash[:args],:class=>@child_class}) > @id_hash[child_hash[:args][:id]][:job_key] = > child_hash[:job_key] > end > else > child_hash[:job_key] = > spawn_worker(job_args.merge(:class=>@child_class)) > end > end > self[child_hash[:job_key]].thread[:DQ_request].wait(child_hash[:s_mutex]) > > # grab and store the results > if child_hash[:args][:id] > @id_hash_mutex.synchronize do > if @reQ_on_finish > r = self[child_hash[:job_key]].results > @id_hash[child_hash[:args][:id]][:results] = r if r > @id_hash[child_hash[:args][:id]][:status] = :done > @id_hash[child_hash[:args][:id]][:done_at] = Time.now > else > @id_hash.delete(child_hash[:args][:id]) > end > end > end > self[child_hash[:job_key]].thread[:DQed].signal > @completed += 1 > [:args,:job_key].each {|k| child_hash[k] = nil } > end > # Loop back and wait for the job_key to get killed again.... > end > end > end > end > > private > > def [](key) > # Use jobs to avoid the access time update w/ [] > ::BackgrounDRb::MiddleMan.instance.jobs[key] > end > > end > > > module BackgrounDRb > > class RailsQueue < BackgrounDRb::Rails > attr_reader :progress > def initialize(key, args={}) > super(key,args) > @job_ctrl = true > end > > def start_process > return if schedule_first_run && schedule_first_run.to_i > > Time.now.to_i > @thread = Thread.new do > Thread.current[:safe_to_kill] = ConditionVariable.new > Thread.current[:kill] = false > Thread.current[:DQ_request] = ConditionVariable.new > Thread.current[:DQed] = ConditionVariable.new > Thread.current[:mutex] = Mutex.new > begin > Thread.current[:mutex].synchronize do > do_work(@args) > end > rescue Exception => e > @logger.error "#{ e.message } - (#{ e.class })" << "\n" << > (e.backtrace or []).join("\n") > end > end > @next_start = @interval.from_now if schedule_repeat > end > > def results > # Overwrite this method and set reQ_on_finish = true (in the > queue worker args) > # to have a process put it's results in back in the queue > # for pickup before being killed > nil > end > > def before_DQ(args=nil) > # stub method that gets called before dequeue is run. > # Overwrite in your class instance > true > end > > def terminate(args=nil) > do_DQ(args) > super(args) > end > > def suicide(args=nil) > do_DQ(args) > kill > Thread.exit > end > > private > def do_DQ(args=nil) > before_DQ(args) > Thread.current[:DQ_request].signal > Thread.current[:DQed].wait(Thread.current[:mutex]) > end > end > end > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -- > Michael Siebert > > > www.siebert-wd.de - Gedanken lesen > www.stellar-legends.de - > Weltraum-Browsergame im Alpha-Stadium -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | David Lemstra B.Eng.Mgt., M.A.Sc. | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| | david at lemstra.ca | 4339 Harrison Rd. Binbrook | | (905)-692-3687 | Ontario, Canada. L0R1C0 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | "If it ain't broke, it doesn't have enough | | features yet" ~Scott Adams | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From christopher.k.hall at gmail.com Fri Sep 15 09:54:07 2006 From: christopher.k.hall at gmail.com (Chris Hall) Date: Fri, 15 Sep 2006 09:54:07 -0400 Subject: [Backgroundrb-devel] unable to figure out how to get backgroundrb.yml to be used when starting backgroundrb Message-ID: i ran rake backgroundrb:setup and modified the default config/backgroundrb.yml file as: --- port: "22223" timer_sleep: 60 load_rails: true environment: production host: localhost database_yml: config/database.yml acl: deny: all allow: localhost 127.0.0.1 order: deny,allow and my database.yml production: adapter: mysql database: chip2_production username: xxxxx password: xxxxx socket: /var/lib/mysql/mysql.sock i found that rake backgroundrb:start starts the server with the default values so i attempted to run: script/backgroundrb/start --config-file config/backgroundrb.yml but i get the following error: AM2005:/srv/www/railsapps/chip2-beta-demo_091506 # script/backgroundrb/start --config-file config/backgroundrb.yml /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/active_record/connection_adapters/abstract/connection_specification.rb:194:in `establish_connection': development database is not configured (ActiveRecord::AdapterNotSpecified) from /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/active_record/connection_adapters/abstract/connection_specification.rb:185:in `establish_connection' from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/initializer.rb:182:in `initialize_database' from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/initializer.rb:84:in `process' from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/initializer.rb:42:in `run' from ./script/backgroundrb/../../config/environment.rb:13 from /usr/lib/ruby/1.8/rubygems/custom_require.rb:21:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:147:in `require' from script/backgroundrb/start:68 if backgroundrb is to use the backgroundrb.ylm config file, why would it be trying to use the development database settings? it's as if it's not even reading the config file. Chris From gethemant at gmail.com Fri Sep 15 10:29:58 2006 From: gethemant at gmail.com (hemant) Date: Fri, 15 Sep 2006 19:59:58 +0530 Subject: [Backgroundrb-devel] unable to figure out how to get backgroundrb.yml to be used when starting backgroundrb In-Reply-To: References: Message-ID: > > if backgroundrb is to use the backgroundrb.ylm config file, why would > it be trying to use the development database settings? it's as if > it's not even reading the config file. > > Chris Not sure, why is that happening. But till Ezra comes, you can directly modify the start script in #{RAILS_ROOT}/script/backgroudrb/start. There should be a hash called "defaults", just change your environment to production there and you should be ready to go. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060915/aab3dc1c/attachment.html From bill.walton at charter.net Fri Sep 15 12:08:48 2006 From: bill.walton at charter.net (Bill Walton) Date: Fri, 15 Sep 2006 11:08:48 -0500 Subject: [Backgroundrb-devel] Could a BackgrounDrb worker do this? References: <029601c6d504$7075ed80$6401a8c0@Presario> <03d701c6d5e3$1e365ed0$6401a8c0@Presario> <8AD0F963-06FA-45AE-9A97-16E3500580D6@gmail.com> <041c01c6d67a$0b202cd0$6401a8c0@Presario> <066d01c6d742$4c42d6e0$6401a8c0@Presario> <00f501c6d827$45953b90$6401a8c0@Presario> <32b567370609141131t5460dccbr9b4d6236471b1734@mail.gmail.com> <012701c6d838$7f4d5320$6401a8c0@Presario> <32b567370609142334r5c345488ye99ada8144a80bee@mail.gmail.com> Message-ID: <02eb01c6d8e1$3acc3320$6401a8c0@Presario> Hi Michael, Michael Siebert wrote: > erm... i have an idea... > use activerecord session store and when the > user logs in, create a bgdrb worker. this worker > is responsible for deleting the users session and > data after a certain amount of time (the timeout). That's pretty much Ezra suggested earlier. > now, in a before_filter, get your worker (hope > you saved the worker id) and update the timeout > somehow. Yep. > would that aproach satisfy all of your requirements? I'm still getting my arms around how it would work, exactly, but I think this is the track I'm going to take. The worker will be responsible for deleting the session whether the visitor explicitly logs out, times out with their browser still open, or abandons the session by closing/redirecting the browser. I've not done much with processing in the model and that's probably a material part of why I'm having trouble with getting my arms around how this would work. As long as I can delete the files referenced in the DB before the DB record that identifies them gets deleted it should work. Thanks for your help! Bill From ezmobius at gmail.com Fri Sep 15 12:58:28 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 15 Sep 2006 09:58:28 -0700 Subject: [Backgroundrb-devel] unable to figure out how to get backgroundrb.yml to be used when starting backgroundrb In-Reply-To: References: Message-ID: <20D917A5-03E0-4ECB-B442-C2B298583D10@brainspl.at> Hey Chris- Until I release the new version this weekend , you can fix this bug by un-commenting line 69 in the start script. Soryy, that was my fault as I commited without remembering to uncomment that line. New version of the whole plugin will be out this weekend. -Ezra On Sep 15, 2006, at 6:54 AM, Chris Hall wrote: > i ran rake backgroundrb:setup and modified the default > config/backgroundrb.yml file as: > > --- > port: "22223" > timer_sleep: 60 > load_rails: true > environment: production > host: localhost > database_yml: config/database.yml > acl: > deny: all > allow: localhost 127.0.0.1 > order: deny,allow > > and my database.yml > > production: > adapter: mysql > database: chip2_production > username: xxxxx > password: xxxxx > socket: /var/lib/mysql/mysql.sock > > > > i found that rake backgroundrb:start starts the server with the > default values so i attempted to run: > > script/backgroundrb/start --config-file config/backgroundrb.yml > > but i get the following error: > > AM2005:/srv/www/railsapps/chip2-beta-demo_091506 # > script/backgroundrb/start --config-file config/backgroundrb.yml > /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/active_record/ > connection_adapters/abstract/connection_specification.rb:194:in > `establish_connection': development database is not configured > (ActiveRecord::AdapterNotSpecified) > from /usr/lib/ruby/gems/1.8/gems/activerecord-1.14.4/lib/ > active_record/connection_adapters/abstract/ > connection_specification.rb:185:in > `establish_connection' > from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/ > initializer.rb:182:in > `initialize_database' > from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/ > initializer.rb:84:in > `process' > from /usr/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/ > initializer.rb:42:in > `run' > from ./script/backgroundrb/../../config/environment.rb:13 > from /usr/lib/ruby/1.8/rubygems/custom_require.rb:21:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:147:in > `require' > from script/backgroundrb/start:68 > > if backgroundrb is to use the backgroundrb.ylm config file, why would > it be trying to use the development database settings? it's as if > it's not even reading the config file. > > Chris > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From bill.walton at charter.net Mon Sep 18 10:25:49 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 18 Sep 2006 09:25:49 -0500 Subject: [Backgroundrb-devel] Problem with install on IR/Windows Message-ID: <496401c6db2e$571350a0$6401a8c0@Presario> I'm getting an error I don't understand when I try to install the BackgrounDRb plugin. I installed the plugin yesterday, or thought I did, and tried this morning to start the daemon. Got an error and so tried to reinstall using: >ruby script\plugin install svn://rubyforge.org//var/svn/backgroundrb --force The first line of the reponse says: svn: 'C:\InstantRails\rails_apps\sandbox\vendor\plugins' is not a working copy Anybody got any idea what I'm doing wrong? Thanks! Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060918/afa3e3be/attachment-0001.html From gethemant at gmail.com Mon Sep 18 23:51:05 2006 From: gethemant at gmail.com (hemant) Date: Tue, 19 Sep 2006 09:21:05 +0530 Subject: [Backgroundrb-devel] Problem with install on IR/Windows In-Reply-To: <496401c6db2e$571350a0$6401a8c0@Presario> References: <496401c6db2e$571350a0$6401a8c0@Presario> Message-ID: On 9/18/06, Bill Walton wrote: > > > I'm getting an error I don't understand when I try to install the > BackgrounDRb plugin. > > I installed the plugin yesterday, or thought I did, and tried this morning > to start the daemon. Got an error and so tried to reinstall using: > >ruby script\plugin install > svn://rubyforge.org//var/svn/backgroundrb --force > > The first line of the reponse says: > svn: 'C:\InstantRails\rails_apps\sandbox\vendor\plugins' is > not a working copy > There is not much you can do there, except: 1. Go to vendor plugins directory 2. and do: svn co svn://rubyforge.org//var/svn/backgroundrb Its almost as good as the plugin/install aproach. -- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary. From gethemant at gmail.com Tue Sep 19 08:48:46 2006 From: gethemant at gmail.com (hemant) Date: Tue, 19 Sep 2006 18:18:46 +0530 Subject: [Backgroundrb-devel] worker threads written in rubyinline Message-ID: Hey Ezra, I asked this on ruby-talk, but probably you missed it. I am already using Backgroundrb for several other stuff. But i am curious, that how it will work with rubyinline? Ruby threads, should handle it alright..theoretically(since backgroundrb uses ruby threads) But, i would just ask opinions of Ezra on this. Because i feel, stuff at my end needs some real optimization. These worker threads are damn slow and i need to make them fast. I have some computing intensive workers and i need to optimize them. -- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary. From bobo.beware at gmail.com Thu Sep 21 08:44:01 2006 From: bobo.beware at gmail.com (sherwin velasco) Date: Thu, 21 Sep 2006 20:44:01 +0800 Subject: [Backgroundrb-devel] backgroundrb scalability Message-ID: i'm in a team that is currently developing a web 2.0 application that really needs to scale in the very near future. we are using backgroundrb for some of the server tasks. currently, we have two servers (in RubyOnRails) that connects to a single backgroundrb daemon. is there a way that we can run multiple backgroundrb and have the servers do a round robin on them to retrieve a worker using a key that was saved in the session? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060921/34c19d25/attachment.html From ezmobius at gmail.com Thu Sep 21 21:47:58 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 21 Sep 2006 18:47:58 -0700 Subject: [Backgroundrb-devel] backgroundrb scalability In-Reply-To: References: Message-ID: On Sep 21, 2006, at 5:44 AM, sherwin velasco wrote: > i'm in a team that is currently developing a web 2.0 application > that really needs to scale in the very near future. > we are using backgroundrb for some of the server tasks. > currently, we have two servers (in RubyOnRails) that connects to a > single backgroundrb daemon. > is there a way that we can run multiple backgroundrb and have the > servers do a round robin on them to retrieve a worker using a key > that was saved in the session? > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel Hey Sherwin- This could be coded up to work for you. You would basically write a class that wraps multiple middleman clients and tries to get the workers by checking for MiddleMan.jobs.keys.include? session [:jobkey] . So you could have MiddleMan1, MiddleMan2..MiddleManN . And just make a wrapper that iterates over each midleman checking to see if it contains the job you are looking for and using the right middleman. This would not be too hard to come up with and I might add something to support this soon. I do have a completele new version of the whole plugin cooking. I hope to get it released this weekend but we shall see. The new version will support spawning multiple processes and managing those through the same interface. -Ezra From awatercn at gmail.com Fri Sep 22 00:22:28 2006 From: awatercn at gmail.com (a water) Date: Fri, 22 Sep 2006 12:22:28 +0800 Subject: [Backgroundrb-devel] how about the global data when multiple backgroundrbs ? Message-ID: <29f5f8f20609212122p6cb5af84v439a34bce7b6a282@mail.gmail.com> Ezra, Suppose a chatroom application, a RailsApp + 2 backgroundrbs: MiddleMan1& MiddleMan2, running in 2 machines. When Chater1 login, RailsApp call MiddleMan1 to get something about Chater1 from db to memory, like his contacts or other personal settings. When Chater2 login, RailsApp call MiddleMan2 do the same thing. But who is reponsible for the Global Data? like a Online Chaters List, or Active Chat Room List ? I think, for the MiddleMan1 & MiddleMan2 located in different machines, global data should not be there. So global data in RailsApp as Global Variables? But how about RailsApp also running in 2+ servers ? It seems the possible ways to store global data is db or network file system ? Hope for suggestions, thanks. >>On Sep 21, 2006, at 5:44 AM, sherwin velasco wrote: >> i'm in a team that is currently developing a web 2.0 application that >> really needs to scale in the very near future. >> we are using backgroundrb for some of the server tasks. >> currently, we have two servers (in RubyOnRails) that connects to a >> single backgroundrb daemon. >> is there a way that we can run multiple backgroundrb and have the >> servers do a round robin on them to retrieve a worker using a key that >> was saved in the session? >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > Hey Sherwin- > > This could be coded up to work for you. You would basically write a class that wraps multiple middleman clients and tries to get the workers by checking for MiddleMan.jobs.keys.include? session [:jobkey] . So you could have MiddleMan1, MiddleMan2..MiddleManN . >And just make a wrapper that iterates over each midleman checking to see if it contains the job you are looking for and using the right middleman. > This would not be too hard to come up with and I might add something to support this soon. I do have a completele new version of the whole plugin cooking. >I hope to get it released this weekend but we shall see. The new version will support spawning multiple processes and managing those through the same interface. >-Ezra -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060922/2f3d6122/attachment.html From omer at bitwine.com Sat Sep 23 14:03:26 2006 From: omer at bitwine.com (Omer Enbar) Date: Sat, 23 Sep 2006 20:03:26 +0200 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method Message-ID: Hey. I have a very annoying problem, and was wondering what is wrong. Suppose I have backgroundrb running, and then I have an action in some controller. In the action I define a worker. When leaving the action, suddenly the connection to backgroundrb is lost: DRb URI: druby://localhost:22222 Pid: 3976 Autostart... done druby://localhost:42531 - # druby://localhost:42531 - # ... When I'm still in the action, for instance using 'sleep 10'. As long I'm inside the action, there's no problem with the worker. Why is that? Why would the connection to the worker be lost? Thanks, Omer. From ezmobius at gmail.com Sat Sep 23 23:57:09 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sat, 23 Sep 2006 20:57:09 -0700 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: References: Message-ID: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> Hey Omer- On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: > Hey. > > I have a very annoying problem, and was wondering what is wrong. > Suppose I have backgroundrb running, and then I have an action in some > controller. In the action I define a worker. When leaving the action, > suddenly the connection to backgroundrb is lost: > DRb URI: druby://localhost:22222 > Pid: 3976 > Autostart... > done > druby://localhost:42531 - # connect(2)> > druby://localhost:42531 - # connect(2)> > ... > > When I'm still in the action, for instance using 'sleep 10'. As long > I'm inside the action, there's no problem with the worker. > > Why is that? > Why would the connection to the worker be lost? > > > Thanks, > Omer. Can you paste the code in question? I am not sure what you are trying to do that is causing you trouble. -Ezra From ezmobius at gmail.com Sun Sep 24 03:39:02 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 24 Sep 2006 00:39:02 -0700 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: References: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> Message-ID: <1A112BA3-9A96-4280-B431-0021403DC595@gmail.com> Hey Omer- Ok I see your problem. You don;t want the workers to be defined in environment.rb like that, its not the way bdrb works. The workers are required into the drb server when it starts and not on the rails side of things. Can you explain what it is you are trying to accomplish and why you need to dynamically create the worker class like that? I;m curious and maybe I can help you come up with a better way of doing this. Also the next release of backgroundrb is coming soon ? ;) And now it uses multiple processes instead of running workers in one multithreaded ruby process. This makes it much much more scalable and also keeps the same interface! -Ezra On Sep 24, 2006, at 12:32 AM, Omer Enbar wrote: > Hey. > > I've got this in environment.rb: > > # define a worker class on the run with 'run_every 1.second, method' > # in backgroundrb there's only definition, while in the server it is > being spawned > def run_every delta, meth, singleton = true > worker_class_name = "Worker#{Digest::SHA1.hexdigest(meth.to_s)}" > if not Object.constants.member? worker_class_name > Object.class_eval <<-METHOD_CODE > class #{worker_class_name} < BackgrounDRb::Rails > > repeat_every #{delta} > first_run Time.now > > def do_work(args) > log :debug, "\#{Time.now} running: \#{args[:name]}" > args[:method].call > rescue => e > puts "\#{e}" > # kill > end > end > METHOD_CODE > puts worker_class_name > end > MiddleMan.new_worker(:class => worker_class_name, > :singleton => singleton, > :args => {:method => meth, :name => meth.to_s}) > unless $0 =~ /backgroundrb/ > end > > def blah > puts "blah!" > end > > def run_workers > run_every 1.second, Object.method(:blah) > rescue DRb::DRbConnError > run_backgroundrb > end > > > Now I have an action, support /localhost/test/drb > > def drb > run_workers > sleep 5 > end > > When I run the action, I'll start seeing 'blah' for 5 seconds, and > once the execution leaves method drb, I'll have a connection error > druby://localhost:42531 - # connect(2) > as printed in the body of the worker. > > Thanks for the quick reply. > Omer. > > > On 9/24/06, Ezra Zygmuntowicz wrote: >> Hey Omer- >> >> On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: >> >> > Hey. >> > >> > I have a very annoying problem, and was wondering what is wrong. >> > Suppose I have backgroundrb running, and then I have an action >> in some >> > controller. In the action I define a worker. When leaving the >> action, >> > suddenly the connection to backgroundrb is lost: >> > DRb URI: druby://localhost:22222 >> > Pid: 3976 >> > Autostart... >> > done >> > druby://localhost:42531 - #> > connect(2)> >> > druby://localhost:42531 - #> > connect(2)> >> > ... >> > >> > When I'm still in the action, for instance using 'sleep 10'. As >> long >> > I'm inside the action, there's no problem with the worker. >> > >> > Why is that? >> > Why would the connection to the worker be lost? >> > >> > >> > Thanks, >> > Omer. >> >> Can you paste the code in question? I am not sure what you >> are >> trying to do that is causing you trouble. >> >> -Ezra >> From jedgecombe at carolina.rr.com Sun Sep 24 09:52:46 2006 From: jedgecombe at carolina.rr.com (Jason Edgecombe) Date: Sun, 24 Sep 2006 09:52:46 -0400 Subject: [Backgroundrb-devel] how about the global data when multiple backgroundrbs ? In-Reply-To: <29f5f8f20609212122p6cb5af84v439a34bce7b6a282@mail.gmail.com> References: <29f5f8f20609212122p6cb5af84v439a34bce7b6a282@mail.gmail.com> Message-ID: <45168DAE.9060005@carolina.rr.com> Hi, The answer is that nobody is responsible unless you code them to be. The easiest solution is to use the DB. I'm guessing you could also make one middleman the "master" that holds all of the globals. The DB gives the best data persistence. Since you're already using a two server set up, I would recommend checking out memcached at www.danga.org. If you care about data persistence, then use the DB with memcached as a cache. Otherwise, just use memcached my itself. Some other options might be Rinda or a third plan drb process. Sincerely, Jason a water wrote: > > Ezra, > Suppose a chatroom application, a RailsApp + 2 backgroundrbs: > MiddleMan1& MiddleMan2, running in 2 machines. > > When Chater1 login, RailsApp call MiddleMan1 to get something about > Chater1 from db to memory, like his contacts or other personal settings. > > When Chater2 login, RailsApp call MiddleMan2 do the same thing. > > But who is reponsible for the Global Data? like a Online Chaters > List, or Active Chat Room List ? > > I think, for the MiddleMan1 & MiddleMan2 located in different > machines, global data should not be there. > > So global data in RailsApp as Global Variables? But how about > RailsApp also running in 2+ servers ? > > It seems the possible ways to store global data is db or network > file system ? > > Hope for suggestions, thanks. > > >>On Sep 21, 2006, at 5:44 AM, sherwin velasco wrote: > > >> i'm in a team that is currently developing a web 2.0 application that > >> really needs to scale in the very near future. > >> we are using backgroundrb for some of the server tasks. > >> currently, we have two servers (in RubyOnRails) that connects to a > >> single backgroundrb daemon. > >> is there a way that we can run multiple backgroundrb and have the > >> servers do a round robin on them to retrieve a worker using a key that > >> was saved in the session? > >> > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > Hey Sherwin- > > > > This could be coded up to work for you. You would basically write a > class that wraps multiple middleman clients and tries to get the > workers by checking for MiddleMan.jobs.keys.include ? session > [:jobkey] . So you could have MiddleMan1, MiddleMan2..MiddleManN . > >And just make a wrapper that iterates over each midleman checking to > see if it contains the job you are looking for and using the right > middleman. > > > This would not be too hard to come up with and I might add something > to support this soon. I do have a completele new version of the whole > plugin cooking. >I hope to get it released this weekend but we shall > see. The new version will support spawning multiple processes and > managing those through the same interface. > > > >-Ezra > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From eimorton at gmail.com Sun Sep 24 11:21:09 2006 From: eimorton at gmail.com (Erik Morton) Date: Sun, 24 Sep 2006 11:21:09 -0400 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB Message-ID: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> I'm having trouble getting backgroundrb up and running in a production environment. Here's my backgroundrb.yml: --- port: "22222" timer_sleep: 60 load_rails: true environment: production host: XXX.XX.XXX.50 database_yml: config/database.yml acl: deny: all allow: localhost 127.0.0.1 XXX.XX.XXX.51 order: deny,allow I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ start -d I can then telnet localhost 22222 and connect to the drb server However when I: telnet XXX.XX.XXX.50 22222 from the same machine I get connection refused. Does this sound like an ACL issue or perhaps my iptables? Thanks in advance, Erik From ezmobius at gmail.com Sun Sep 24 16:40:22 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 24 Sep 2006 13:40:22 -0700 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB In-Reply-To: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> References: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> Message-ID: <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> Hey Eric- On Sep 24, 2006, at 8:21 AM, Erik Morton wrote: > I'm having trouble getting backgroundrb up and running in a > production environment. > > Here's my backgroundrb.yml: > --- > port: "22222" > timer_sleep: 60 > load_rails: true > environment: production > host: XXX.XX.XXX.50 > database_yml: config/database.yml > acl: > deny: all > allow: localhost 127.0.0.1 XXX.XX.XXX.51 > order: deny,allow > > I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ > start -d > I can then telnet localhost 22222 and connect to the drb server > However when I: telnet XXX.XX.XXX.50 22222 from the same machine I > get connection refused. > > Does this sound like an ACL issue or perhaps my iptables? > > Thanks in advance, > Erik It could be an acl issue but I rather think it might be a network issue, drb is fidgety with NAT and firewalls. IS the other server behind NAT or a firewall? If so you may want to look at DRbFire, its an add on to drb to help it navigate firewalls and nat. Cheers- -Ezra From eimorton at gmail.com Sun Sep 24 16:59:48 2006 From: eimorton at gmail.com (Erik Morton) Date: Sun, 24 Sep 2006 16:59:48 -0400 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB In-Reply-To: <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> References: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> Message-ID: <17303987-9147-418D-BCBA-3AB15CF9FE18@gmail.com> Interesting. I have a three server setup at a colo and they are all running out of the box firewalls from RedHat EL4. Normally I would put my Mongrels on the "app" server with BackgroundRB. In this case however my app server is a bit overloaded with a HylaFAX installation, so I have my mongrels on the "web" server. Would you consider it a best practice to keep the mongrels and BackgroundRB on the same server? Erik On Sep 24, 2006, at 4:40 PM, Ezra Zygmuntowicz wrote: > Hey Eric- > > On Sep 24, 2006, at 8:21 AM, Erik Morton wrote: > >> I'm having trouble getting backgroundrb up and running in a >> production environment. >> >> Here's my backgroundrb.yml: >> --- >> port: "22222" >> timer_sleep: 60 >> load_rails: true >> environment: production >> host: XXX.XX.XXX.50 >> database_yml: config/database.yml >> acl: >> deny: all >> allow: localhost 127.0.0.1 XXX.XX.XXX.51 >> order: deny,allow >> >> I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ >> start -d >> I can then telnet localhost 22222 and connect to the drb server >> However when I: telnet XXX.XX.XXX.50 22222 from the same machine I >> get connection refused. >> >> Does this sound like an ACL issue or perhaps my iptables? >> >> Thanks in advance, >> Erik > > It could be an acl issue but I rather think it might be a network > issue, drb is fidgety with NAT and firewalls. IS the other server > behind NAT or a firewall? If so you may want to look at DRbFire, > its an add on to drb to help it navigate firewalls and nat. > > > Cheers- > -Ezra From ezmobius at gmail.com Sun Sep 24 17:07:39 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 24 Sep 2006 14:07:39 -0700 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB In-Reply-To: <17303987-9147-418D-BCBA-3AB15CF9FE18@gmail.com> References: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> <17303987-9147-418D-BCBA-3AB15CF9FE18@gmail.com> Message-ID: Hey- Hrmm, if all your servers are in the same subnet or vlan then you should be able to put the drb anywhere. Mongrel on the web server is fine if the app server is loaded. ANd drb should work fine wherever you put it. I'm not sure what the problems you are having are all about, I have a 5 server setup with one of the servers only running drb processes and it talks to the other 4 servers in the subnet fine. It will be quicker if you put the mongrels and drb on the same server, but I don't think it should meka to much difference on a local network. -Ezra On Sep 24, 2006, at 1:59 PM, Erik Morton wrote: > Interesting. I have a three server setup at a colo and they are all > running out of the box firewalls from RedHat EL4. Normally I would > put my Mongrels on the "app" server with BackgroundRB. In this case > however my app server is a bit overloaded with a HylaFAX > installation, so I have my mongrels on the "web" server. > > Would you consider it a best practice to keep the mongrels and > BackgroundRB on the same server? > > Erik > On Sep 24, 2006, at 4:40 PM, Ezra Zygmuntowicz wrote: > >> Hey Eric- >> >> On Sep 24, 2006, at 8:21 AM, Erik Morton wrote: >> >>> I'm having trouble getting backgroundrb up and running in a >>> production environment. >>> >>> Here's my backgroundrb.yml: >>> --- >>> port: "22222" >>> timer_sleep: 60 >>> load_rails: true >>> environment: production >>> host: XXX.XX.XXX.50 >>> database_yml: config/database.yml >>> acl: >>> deny: all >>> allow: localhost 127.0.0.1 XXX.XX.XXX.51 >>> order: deny,allow >>> >>> I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ >>> start -d >>> I can then telnet localhost 22222 and connect to the drb server >>> However when I: telnet XXX.XX.XXX.50 22222 from the same machine I >>> get connection refused. >>> >>> Does this sound like an ACL issue or perhaps my iptables? >>> >>> Thanks in advance, >>> Erik >> >> It could be an acl issue but I rather think it might be a network >> issue, drb is fidgety with NAT and firewalls. IS the other server >> behind NAT or a firewall? If so you may want to look at DRbFire, >> its an add on to drb to help it navigate firewalls and nat. >> >> >> Cheers- >> -Ezra > From eimorton at gmail.com Sun Sep 24 18:07:09 2006 From: eimorton at gmail.com (Erik Morton) Date: Sun, 24 Sep 2006 18:07:09 -0400 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB In-Reply-To: References: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> <17303987-9147-418D-BCBA-3AB15CF9FE18@gmail.com> Message-ID: <9A6D40C9-7DA0-43EB-8F32-F8477C78ACB6@gmail.com> I got it working with the following ACL: acl: deny: allow: localhost 127.0.0.1 XXX.XX.XXX.51 order: allow,deny I have iptables blocking port 22222 to all hosts except for the web server. Please let me know if you think this is a bad idea... Thanks again Erik On Sep 24, 2006, at 5:07 PM, Ezra Zygmuntowicz wrote: > Hey- > > Hrmm, if all your servers are in the same subnet or vlan then you > should be able to put the drb anywhere. Mongrel on the web server > is fine if the app server is loaded. ANd drb should work fine > wherever you put it. I'm not sure what the problems you are having > are all about, I have a 5 server setup with one of the servers only > running drb processes and it talks to the other 4 servers in the > subnet fine. > > It will be quicker if you put the mongrels and drb on the same > server, but I don't think it should meka to much difference on a > local network. > > -Ezra > > On Sep 24, 2006, at 1:59 PM, Erik Morton wrote: > >> Interesting. I have a three server setup at a colo and they are >> all running out of the box firewalls from RedHat EL4. Normally I >> would put my Mongrels on the "app" server with BackgroundRB. In >> this case however my app server is a bit overloaded with a HylaFAX >> installation, so I have my mongrels on the "web" server. >> >> Would you consider it a best practice to keep the mongrels and >> BackgroundRB on the same server? >> >> Erik >> On Sep 24, 2006, at 4:40 PM, Ezra Zygmuntowicz wrote: >> >>> Hey Eric- >>> >>> On Sep 24, 2006, at 8:21 AM, Erik Morton wrote: >>> >>>> I'm having trouble getting backgroundrb up and running in a >>>> production environment. >>>> >>>> Here's my backgroundrb.yml: >>>> --- >>>> port: "22222" >>>> timer_sleep: 60 >>>> load_rails: true >>>> environment: production >>>> host: XXX.XX.XXX.50 >>>> database_yml: config/database.yml >>>> acl: >>>> deny: all >>>> allow: localhost 127.0.0.1 XXX.XX.XXX.51 >>>> order: deny,allow >>>> >>>> I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ >>>> start -d >>>> I can then telnet localhost 22222 and connect to the drb server >>>> However when I: telnet XXX.XX.XXX.50 22222 from the same machine I >>>> get connection refused. >>>> >>>> Does this sound like an ACL issue or perhaps my iptables? >>>> >>>> Thanks in advance, >>>> Erik >>> >>> It could be an acl issue but I rather think it might be a >>> network issue, drb is fidgety with NAT and firewalls. IS the >>> other server behind NAT or a firewall? If so you may want to >>> look at DRbFire, its an add on to drb to help it navigate >>> firewalls and nat. >>> >>> >>> Cheers- >>> -Ezra >> > From ezmobius at gmail.com Sun Sep 24 18:12:07 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 24 Sep 2006 15:12:07 -0700 Subject: [Backgroundrb-devel] Can't Connect to BackgroundRB In-Reply-To: <9A6D40C9-7DA0-43EB-8F32-F8477C78ACB6@gmail.com> References: <9BA0E1B3-7155-4E2E-8526-1E99560036DB@gmail.com> <4E383D43-23E9-41F4-9E46-DD727C63168C@brainspl.at> <17303987-9147-418D-BCBA-3AB15CF9FE18@gmail.com> <9A6D40C9-7DA0-43EB-8F32-F8477C78ACB6@gmail.com> Message-ID: <20AB864C-1060-4041-8793-820A98089526@gmail.com> Erik- I think thats a good setup, you should be all set. Cheers- -Ezra On Sep 24, 2006, at 3:07 PM, Erik Morton wrote: > I got it working with the following ACL: > > acl: > deny: > allow: localhost 127.0.0.1 XXX.XX.XXX.51 > order: allow,deny > > I have iptables blocking port 22222 to all hosts except for the web > server. Please let me know if you think this is a bad idea... > > > Thanks again > > Erik > On Sep 24, 2006, at 5:07 PM, Ezra Zygmuntowicz wrote: > >> Hey- >> >> Hrmm, if all your servers are in the same subnet or vlan then you >> should be able to put the drb anywhere. Mongrel on the web server >> is fine if the app server is loaded. ANd drb should work fine >> wherever you put it. I'm not sure what the problems you are having >> are all about, I have a 5 server setup with one of the servers >> only running drb processes and it talks to the other 4 servers in >> the subnet fine. >> >> It will be quicker if you put the mongrels and drb on the same >> server, but I don't think it should meka to much difference on a >> local network. >> >> -Ezra >> >> On Sep 24, 2006, at 1:59 PM, Erik Morton wrote: >> >>> Interesting. I have a three server setup at a colo and they are >>> all running out of the box firewalls from RedHat EL4. Normally I >>> would put my Mongrels on the "app" server with BackgroundRB. In >>> this case however my app server is a bit overloaded with a >>> HylaFAX installation, so I have my mongrels on the "web" server. >>> >>> Would you consider it a best practice to keep the mongrels and >>> BackgroundRB on the same server? >>> >>> Erik >>> On Sep 24, 2006, at 4:40 PM, Ezra Zygmuntowicz wrote: >>> >>>> Hey Eric- >>>> >>>> On Sep 24, 2006, at 8:21 AM, Erik Morton wrote: >>>> >>>>> I'm having trouble getting backgroundrb up and running in a >>>>> production environment. >>>>> >>>>> Here's my backgroundrb.yml: >>>>> --- >>>>> port: "22222" >>>>> timer_sleep: 60 >>>>> load_rails: true >>>>> environment: production >>>>> host: XXX.XX.XXX.50 >>>>> database_yml: config/database.yml >>>>> acl: >>>>> deny: all >>>>> allow: localhost 127.0.0.1 XXX.XX.XXX.51 >>>>> order: deny,allow >>>>> >>>>> I start BackgroundRB from XXX.XX.XXX.50 with script/backgroundrb/ >>>>> start -d >>>>> I can then telnet localhost 22222 and connect to the drb server >>>>> However when I: telnet XXX.XX.XXX.50 22222 from the same machine I >>>>> get connection refused. >>>>> >>>>> Does this sound like an ACL issue or perhaps my iptables? >>>>> >>>>> Thanks in advance, >>>>> Erik >>>> >>>> It could be an acl issue but I rather think it might be a >>>> network issue, drb is fidgety with NAT and firewalls. IS the >>>> other server behind NAT or a firewall? If so you may want to >>>> look at DRbFire, its an add on to drb to help it navigate >>>> firewalls and nat. >>>> >>>> >>>> Cheers- >>>> -Ezra >>> >> > From ezmobius at gmail.com Sun Sep 24 20:30:35 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 24 Sep 2006 17:30:35 -0700 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: References: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> <1A112BA3-9A96-4280-B431-0021403DC595@gmail.com> Message-ID: <40D9C4FB-6CBF-4F19-BC68-4801DD57CCC6@gmail.com> Hey Omer- I guess I don't understand what you are trying to do? Why do you want to define your workers in the environemnt.rb? Your workers need to go into lib/workers for them to get picked up by the drb server. And I think the issue you are seeing is because you are confusing which side the workers run on. The reason the workers quit when the action is done is that raisl throws away the controller objects and makes new ones on each request. And your workers defined in env.rb are on the rails side of things and not actually running in the drb server like you think they are ;) You are going to get into trouble trying to define your workers in env.rb like that. Just put them in lib/workers. You can dynamically generate them, in there if you want and they will get loaded into the drb server. Backgroundrb kinda straddles the line between the drb server and your rails apps. SO it is easy to get confused about where things are running. -Ezra On Sep 24, 2006, at 5:00 PM, Omer Enbar wrote: > So do we have any ideas what goes wrong? > Once again, here are the details: > 1. Backgroundrb runs smoothly. > 2. I define the worker dynamically in environment.rb both in bdrb and > in the server. > 3. If I execute the worker from environment.rb it runs forever > perfectly. > 4. If I run the worker from an action, it runs perfectly as long as > the action doesn't return. > > Why would it run if I execute the worker from environment.rb forever, > but only as long as the action scope is active when running it from > the action? > > Omer. > > ---------- Forwarded message ---------- > From: Omer Enbar > Date: Sep 24, 2006 11:47 AM > Subject: Re: [Backgroundrb-devel] Connection to backgroundrb is lost > when exiting action method > To: Ezra Zygmuntowicz > Cc: backgroundrb-devel at rubyforge.org > > > Hey Ezra. > > I just wanted to use a nice way to define workers like 'run_every > 5.minutes {block}' :) > Why can't I do it that way if I define it both in backgroundrb and the > server? bdrb does load the definitions and it does run... > > Anyway, if there is no way to do it like this, I'll just write the > workers statically. > > Omer. > > On 9/24/06, Ezra Zygmuntowicz wrote: >> Hey Omer- >> >> Ok I see your problem. You don;t want the workers to be >> defined in >> environment.rb like that, its not the way bdrb works. The workers are >> required into the drb server when it starts and not on the rails side >> of things. Can you explain what it is you are trying to accomplish >> and why you need to dynamically create the worker class like that? >> I;m curious and maybe I can help you come up with a better way of >> doing this. >> >> Also the next release of backgroundrb is coming soon ? ;) >> And now it >> uses multiple processes instead of running workers in one >> multithreaded ruby process. This makes it much much more scalable and >> also keeps the same interface! >> >> >> -Ezra >> >> >> >> On Sep 24, 2006, at 12:32 AM, Omer Enbar wrote: >> >> > Hey. >> > >> > I've got this in environment.rb: >> > >> > # define a worker class on the run with 'run_every 1.second, >> method' >> > # in backgroundrb there's only definition, while in the server >> it is >> > being spawned >> > def run_every delta, meth, singleton = true >> > worker_class_name = "Worker#{Digest::SHA1.hexdigest(meth.to_s)}" >> > if not Object.constants.member? worker_class_name >> > Object.class_eval <<-METHOD_CODE >> > class #{worker_class_name} < BackgrounDRb::Rails >> > >> > repeat_every #{delta} >> > first_run Time.now >> > >> > def do_work(args) >> > log :debug, "\#{Time.now} running: \#{args[:name]}" >> > args[:method].call >> > rescue => e >> > puts "\#{e}" >> > # kill >> > end >> > end >> > METHOD_CODE >> > puts worker_class_name >> > end >> > MiddleMan.new_worker(:class => worker_class_name, >> > :singleton => singleton, >> > :args => {:method => meth, :name => >> meth.to_s}) >> > unless $0 =~ /backgroundrb/ >> > end >> > >> > def blah >> > puts "blah!" >> > end >> > >> > def run_workers >> > run_every 1.second, Object.method(:blah) >> > rescue DRb::DRbConnError >> > run_backgroundrb >> > end >> > >> > >> > Now I have an action, support /localhost/test/drb >> > >> > def drb >> > run_workers >> > sleep 5 >> > end >> > >> > When I run the action, I'll start seeing 'blah' for 5 seconds, and >> > once the execution leaves method drb, I'll have a connection error >> > druby://localhost:42531 - #> > connect(2) >> > as printed in the body of the worker. >> > >> > Thanks for the quick reply. >> > Omer. >> > >> > >> > On 9/24/06, Ezra Zygmuntowicz wrote: >> >> Hey Omer- >> >> >> >> On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: >> >> >> >> > Hey. >> >> > >> >> > I have a very annoying problem, and was wondering what is wrong. >> >> > Suppose I have backgroundrb running, and then I have an action >> >> in some >> >> > controller. In the action I define a worker. When leaving the >> >> action, >> >> > suddenly the connection to backgroundrb is lost: >> >> > DRb URI: druby://localhost:22222 >> >> > Pid: 3976 >> >> > Autostart... >> >> > done >> >> > druby://localhost:42531 - #> >> > connect(2)> >> >> > druby://localhost:42531 - #> >> > connect(2)> >> >> > ... >> >> > >> >> > When I'm still in the action, for instance using 'sleep 10'. As >> >> long >> >> > I'm inside the action, there's no problem with the worker. >> >> > >> >> > Why is that? >> >> > Why would the connection to the worker be lost? >> >> > >> >> > >> >> > Thanks, >> >> > Omer. >> >> >> >> Can you paste the code in question? I am not sure what you >> >> are >> >> trying to do that is causing you trouble. >> >> >> >> -Ezra >> >> >> >> From omer.enbar at gmail.com Sun Sep 24 03:32:19 2006 From: omer.enbar at gmail.com (Omer Enbar) Date: Sun, 24 Sep 2006 09:32:19 +0200 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> References: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> Message-ID: Hey. I've got this in environment.rb: # define a worker class on the run with 'run_every 1.second, method' # in backgroundrb there's only definition, while in the server it is being spawned def run_every delta, meth, singleton = true worker_class_name = "Worker#{Digest::SHA1.hexdigest(meth.to_s)}" if not Object.constants.member? worker_class_name Object.class_eval <<-METHOD_CODE class #{worker_class_name} < BackgrounDRb::Rails repeat_every #{delta} first_run Time.now def do_work(args) log :debug, "\#{Time.now} running: \#{args[:name]}" args[:method].call rescue => e puts "\#{e}" # kill end end METHOD_CODE puts worker_class_name end MiddleMan.new_worker(:class => worker_class_name, :singleton => singleton, :args => {:method => meth, :name => meth.to_s}) unless $0 =~ /backgroundrb/ end def blah puts "blah!" end def run_workers run_every 1.second, Object.method(:blah) rescue DRb::DRbConnError run_backgroundrb end Now I have an action, support /localhost/test/drb def drb run_workers sleep 5 end When I run the action, I'll start seeing 'blah' for 5 seconds, and once the execution leaves method drb, I'll have a connection error druby://localhost:42531 - # wrote: > Hey Omer- > > On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: > > > Hey. > > > > I have a very annoying problem, and was wondering what is wrong. > > Suppose I have backgroundrb running, and then I have an action in some > > controller. In the action I define a worker. When leaving the action, > > suddenly the connection to backgroundrb is lost: > > DRb URI: druby://localhost:22222 > > Pid: 3976 > > Autostart... > > done > > druby://localhost:42531 - # > connect(2)> > > druby://localhost:42531 - # > connect(2)> > > ... > > > > When I'm still in the action, for instance using 'sleep 10'. As long > > I'm inside the action, there's no problem with the worker. > > > > Why is that? > > Why would the connection to the worker be lost? > > > > > > Thanks, > > Omer. > > Can you paste the code in question? I am not sure what you are > trying to do that is causing you trouble. > > -Ezra > From omer.enbar at gmail.com Sun Sep 24 05:47:59 2006 From: omer.enbar at gmail.com (Omer Enbar) Date: Sun, 24 Sep 2006 11:47:59 +0200 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: <1A112BA3-9A96-4280-B431-0021403DC595@gmail.com> References: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> <1A112BA3-9A96-4280-B431-0021403DC595@gmail.com> Message-ID: Hey Ezra. I just wanted to use a nice way to define workers like 'run_every 5.minutes {block}' :) Why can't I do it that way if I define it both in backgroundrb and the server? bdrb does load the definitions and it does run... Anyway, if there is no way to do it like this, I'll just write the workers statically. Omer. On 9/24/06, Ezra Zygmuntowicz wrote: > Hey Omer- > > Ok I see your problem. You don;t want the workers to be defined in > environment.rb like that, its not the way bdrb works. The workers are > required into the drb server when it starts and not on the rails side > of things. Can you explain what it is you are trying to accomplish > and why you need to dynamically create the worker class like that? > I;m curious and maybe I can help you come up with a better way of > doing this. > > Also the next release of backgroundrb is coming soon ? ;) And now it > uses multiple processes instead of running workers in one > multithreaded ruby process. This makes it much much more scalable and > also keeps the same interface! > > > -Ezra > > > > On Sep 24, 2006, at 12:32 AM, Omer Enbar wrote: > > > Hey. > > > > I've got this in environment.rb: > > > > # define a worker class on the run with 'run_every 1.second, method' > > # in backgroundrb there's only definition, while in the server it is > > being spawned > > def run_every delta, meth, singleton = true > > worker_class_name = "Worker#{Digest::SHA1.hexdigest(meth.to_s)}" > > if not Object.constants.member? worker_class_name > > Object.class_eval <<-METHOD_CODE > > class #{worker_class_name} < BackgrounDRb::Rails > > > > repeat_every #{delta} > > first_run Time.now > > > > def do_work(args) > > log :debug, "\#{Time.now} running: \#{args[:name]}" > > args[:method].call > > rescue => e > > puts "\#{e}" > > # kill > > end > > end > > METHOD_CODE > > puts worker_class_name > > end > > MiddleMan.new_worker(:class => worker_class_name, > > :singleton => singleton, > > :args => {:method => meth, :name => meth.to_s}) > > unless $0 =~ /backgroundrb/ > > end > > > > def blah > > puts "blah!" > > end > > > > def run_workers > > run_every 1.second, Object.method(:blah) > > rescue DRb::DRbConnError > > run_backgroundrb > > end > > > > > > Now I have an action, support /localhost/test/drb > > > > def drb > > run_workers > > sleep 5 > > end > > > > When I run the action, I'll start seeing 'blah' for 5 seconds, and > > once the execution leaves method drb, I'll have a connection error > > druby://localhost:42531 - # > connect(2) > > as printed in the body of the worker. > > > > Thanks for the quick reply. > > Omer. > > > > > > On 9/24/06, Ezra Zygmuntowicz wrote: > >> Hey Omer- > >> > >> On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: > >> > >> > Hey. > >> > > >> > I have a very annoying problem, and was wondering what is wrong. > >> > Suppose I have backgroundrb running, and then I have an action > >> in some > >> > controller. In the action I define a worker. When leaving the > >> action, > >> > suddenly the connection to backgroundrb is lost: > >> > DRb URI: druby://localhost:22222 > >> > Pid: 3976 > >> > Autostart... > >> > done > >> > druby://localhost:42531 - # >> > connect(2)> > >> > druby://localhost:42531 - # >> > connect(2)> > >> > ... > >> > > >> > When I'm still in the action, for instance using 'sleep 10'. As > >> long > >> > I'm inside the action, there's no problem with the worker. > >> > > >> > Why is that? > >> > Why would the connection to the worker be lost? > >> > > >> > > >> > Thanks, > >> > Omer. > >> > >> Can you paste the code in question? I am not sure what you > >> are > >> trying to do that is causing you trouble. > >> > >> -Ezra > >> > > From omer.enbar at gmail.com Sun Sep 24 20:00:55 2006 From: omer.enbar at gmail.com (Omer Enbar) Date: Mon, 25 Sep 2006 02:00:55 +0200 Subject: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method In-Reply-To: References: <22ABD847-82C5-42C2-A49D-361376BC46AD@gmail.com> <1A112BA3-9A96-4280-B431-0021403DC595@gmail.com> Message-ID: So do we have any ideas what goes wrong? Once again, here are the details: 1. Backgroundrb runs smoothly. 2. I define the worker dynamically in environment.rb both in bdrb and in the server. 3. If I execute the worker from environment.rb it runs forever perfectly. 4. If I run the worker from an action, it runs perfectly as long as the action doesn't return. Why would it run if I execute the worker from environment.rb forever, but only as long as the action scope is active when running it from the action? Omer. ---------- Forwarded message ---------- From: Omer Enbar Date: Sep 24, 2006 11:47 AM Subject: Re: [Backgroundrb-devel] Connection to backgroundrb is lost when exiting action method To: Ezra Zygmuntowicz Cc: backgroundrb-devel at rubyforge.org Hey Ezra. I just wanted to use a nice way to define workers like 'run_every 5.minutes {block}' :) Why can't I do it that way if I define it both in backgroundrb and the server? bdrb does load the definitions and it does run... Anyway, if there is no way to do it like this, I'll just write the workers statically. Omer. On 9/24/06, Ezra Zygmuntowicz wrote: > Hey Omer- > > Ok I see your problem. You don;t want the workers to be defined in > environment.rb like that, its not the way bdrb works. The workers are > required into the drb server when it starts and not on the rails side > of things. Can you explain what it is you are trying to accomplish > and why you need to dynamically create the worker class like that? > I;m curious and maybe I can help you come up with a better way of > doing this. > > Also the next release of backgroundrb is coming soon ? ;) And now it > uses multiple processes instead of running workers in one > multithreaded ruby process. This makes it much much more scalable and > also keeps the same interface! > > > -Ezra > > > > On Sep 24, 2006, at 12:32 AM, Omer Enbar wrote: > > > Hey. > > > > I've got this in environment.rb: > > > > # define a worker class on the run with 'run_every 1.second, method' > > # in backgroundrb there's only definition, while in the server it is > > being spawned > > def run_every delta, meth, singleton = true > > worker_class_name = "Worker#{Digest::SHA1.hexdigest(meth.to_s)}" > > if not Object.constants.member? worker_class_name > > Object.class_eval <<-METHOD_CODE > > class #{worker_class_name} < BackgrounDRb::Rails > > > > repeat_every #{delta} > > first_run Time.now > > > > def do_work(args) > > log :debug, "\#{Time.now} running: \#{args[:name]}" > > args[:method].call > > rescue => e > > puts "\#{e}" > > # kill > > end > > end > > METHOD_CODE > > puts worker_class_name > > end > > MiddleMan.new_worker(:class => worker_class_name, > > :singleton => singleton, > > :args => {:method => meth, :name => meth.to_s}) > > unless $0 =~ /backgroundrb/ > > end > > > > def blah > > puts "blah!" > > end > > > > def run_workers > > run_every 1.second, Object.method(:blah) > > rescue DRb::DRbConnError > > run_backgroundrb > > end > > > > > > Now I have an action, support /localhost/test/drb > > > > def drb > > run_workers > > sleep 5 > > end > > > > When I run the action, I'll start seeing 'blah' for 5 seconds, and > > once the execution leaves method drb, I'll have a connection error > > druby://localhost:42531 - # > connect(2) > > as printed in the body of the worker. > > > > Thanks for the quick reply. > > Omer. > > > > > > On 9/24/06, Ezra Zygmuntowicz wrote: > >> Hey Omer- > >> > >> On Sep 23, 2006, at 11:03 AM, Omer Enbar wrote: > >> > >> > Hey. > >> > > >> > I have a very annoying problem, and was wondering what is wrong. > >> > Suppose I have backgroundrb running, and then I have an action > >> in some > >> > controller. In the action I define a worker. When leaving the > >> action, > >> > suddenly the connection to backgroundrb is lost: > >> > DRb URI: druby://localhost:22222 > >> > Pid: 3976 > >> > Autostart... > >> > done > >> > druby://localhost:42531 - # >> > connect(2)> > >> > druby://localhost:42531 - # >> > connect(2)> > >> > ... > >> > > >> > When I'm still in the action, for instance using 'sleep 10'. As > >> long > >> > I'm inside the action, there's no problem with the worker. > >> > > >> > Why is that? > >> > Why would the connection to the worker be lost? > >> > > >> > > >> > Thanks, > >> > Omer. > >> > >> Can you paste the code in question? I am not sure what you > >> are > >> trying to do that is causing you trouble. > >> > >> -Ezra > >> > > From michael.dauria at gmail.com Sun Sep 24 21:20:20 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Sun, 24 Sep 2006 21:20:20 -0400 Subject: [Backgroundrb-devel] Clearing ActiveRecord Sessions Message-ID: <1907e2ca0609241820hac9e7b9l2e32601cc10d0f87@mail.gmail.com> Hello all, I am interested in using BackgrounDRb to clear out ActiveRecord based sessions. I have this working, just with a small hitch, here is the code: class ClearSessionsWorker < BackgrounDRb::Rails repeat_every 7.minutes first_run Time.now def do_work(args) Session.destroy_all(["updated_at < ?", 20.minute.ago ]) end end It's not very complicated and does remove sessions, it's just that once the job is done, the thread hangs out leaving a connection to my database open. After some time, MySQL complains about having too many open connections. I was just curious what i am missing. I have tried using "kill", but that only seems to run the job once and not every 7 minutes like i so nicely ask :). I have also tried "terminate", but that did not seem to do anything for me. Any help is greatly appreciated. Thanks, .: Michael :. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060924/189c4e5c/attachment.html From jmetillon at gmail.com Mon Sep 25 07:24:18 2006 From: jmetillon at gmail.com (=?ISO-8859-1?Q?Jonathan_M=E9tillon?=) Date: Mon, 25 Sep 2006 13:24:18 +0200 Subject: [Backgroundrb-devel] Engine Yard blog Message-ID: Just received the news from Tom Mornini. Congrats Ezra for the new Engine Yard site and the blog you will be collaborating. Hope to read you there soon. http://www.engineyard.com/ Jonathan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060925/3f251fa4/attachment.html From awatermail at gmail.com Mon Sep 25 22:00:21 2006 From: awatermail at gmail.com (awatermail) Date: Tue, 26 Sep 2006 10:00:21 +0800 Subject: [Backgroundrb-devel] Is drb much slower than memcached ? In-Reply-To: Message-ID: <4518882d.128ffa5d.2598.5f7b@mx.gmail.com> I do some test, surprising to find seeming to access drb much slower than access memcached. If it's true, should we use more memcached directly than drb based things (like backgrounddrb) in scale situation ? Before the test, I can't decide whether to do(such as get/set global data operations) in drb/backgrounddrb or just access memcached in actioncontrollers. I have thought of that doing save(or some read) operations in drb/backgroundrb asynchronous will improve the performance highly. Hope for your idea, thank you. water The test: ?summary? just loop 100 times doing a simple action with drb & memcached, see how long they will take. ?background? ruby 1.8.4 (2006-04-14) [i386-mswin32] Rails 1.1.6 memcache-client-1.0.3 Distributed Ruby: _dRuby_ version 2.0.4 # from ruby/lib/rub/1.8/drb/drb.rb ?begin? # RAILS_ROOT/config/enviroments/development.rb # new drb object here to ensure just new it once at the beginning require 'drb' DRb.start_service() DRB_OBJ = DRbObject.new(nil, 'druby://localhost:9000') # In a simple actioncontroller class TestController < ApplicationController def test start_time = Time.now.to_f 100.times do DRB_OBJ.doNothingInSide() # just invoke a simple drb function, do nothing inside end end_time = Time.now.to_f flash['tm2'] = sprintf("DRB access take (%0.9f)", end_time - start_time) # log the duration to show in view start_time = Time.now.to_f 100.times do Cache::put('test', {'test' => 'just test'}) # just set the memcached through memcache-client's put method end end_time = Time.now.to_f flash['tm1'] = sprintf("Memcache access take (%0.9f)", end_time - start_time) # log the duration to show in view end end # In the simple view

tm1=#<%=flash['tm1']%>#

tm2=#<%=flash['tm2']%>#

# run & see the results tm1=#Memcache access take (0.546000004) tm2=#DRB access take (3.141000032) The running results different every time, but all show memcached much faster than drb access. From ezmobius at gmail.com Mon Sep 25 22:20:04 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 25 Sep 2006 19:20:04 -0700 Subject: [Backgroundrb-devel] [Rails] Is drb much slower than memcached ? In-Reply-To: <4518882d.128ffa5d.2598.5f7b@mx.gmail.com> References: <4518882d.128ffa5d.2598.5f7b@mx.gmail.com> Message-ID: <672B125E-D9F2-419C-8B90-6E50BD3B21D4@gmail.com> Hi- YEah memcached is pure C and is very fast. DRb is a bit slower for caching. Backgroundrb has caching features but I recommend you use memcached for scalable cache. Backgroundrb is more for the worker classes then for caching. Cheers- -Ezra On Sep 25, 2006, at 7:00 PM, awatermail wrote: > > > I do some test, surprising to find seeming to access drb much > slower than > access memcached. > > If it's true, should we use more memcached directly than drb based > things > (like backgrounddrb) in scale situation ? > > Before the test, I can't decide whether to do(such as get/set > global data > operations) in drb/backgrounddrb or just access memcached in > actioncontrollers. > > I have thought of that doing save(or some read) operations in > drb/backgroundrb asynchronous will improve the performance highly. > > Hope for your idea, thank you. > > water > > The test: > > ?summary? > just loop 100 times doing a simple action with drb & memcached, see > how long > they will take. > > ?background? > ruby 1.8.4 (2006-04-14) [i386-mswin32] > Rails 1.1.6 > memcache-client-1.0.3 > Distributed Ruby: _dRuby_ version 2.0.4 # from ruby/lib/rub/1.8/ > drb/drb.rb > > ?begin? > # RAILS_ROOT/config/enviroments/development.rb > # new drb object here to ensure just new it once at the beginning > require 'drb' > DRb.start_service() > DRB_OBJ = DRbObject.new(nil, 'druby://localhost:9000') > > # In a simple actioncontroller > class TestController < ApplicationController > def test > start_time = Time.now.to_f > 100.times do > DRB_OBJ.doNothingInSide() # just invoke a simple drb > function, do > nothing inside > end > end_time = Time.now.to_f > flash['tm2'] = sprintf("DRB access take (%0.9f)", end_time - > start_time) > # log the duration to show in view > > start_time = Time.now.to_f > 100.times do > Cache::put('test', {'test' => 'just test'}) # just set the > memcached > through memcache-client's put method > end > end_time = Time.now.to_f > flash['tm1'] = sprintf("Memcache access take (%0.9f)", end_time - > start_time) # log the duration to show in view > end > end > > # In the simple view >

tm1=#<%=flash['tm1']%>#

>

tm2=#<%=flash['tm2']%>#

> > # run & see the results > tm1=#Memcache access take (0.546000004) > tm2=#DRB access take (3.141000032) > > > The running results different every time, but all show memcached > much faster > than drb access. > > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google > Groups "Ruby on Rails: Talk" group. > To post to this group, send email to rubyonrails-talk at googlegroups.com > To unsubscribe from this group, send email to rubyonrails-talk- > unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/ > group/rubyonrails-talk > -~----------~----~----~----~------~----~------~--~--- > From gethemant at gmail.com Wed Sep 27 05:39:06 2006 From: gethemant at gmail.com (hemant) Date: Wed, 27 Sep 2006 15:09:06 +0530 Subject: [Backgroundrb-devel] [Rails] Re: Is drb much slower than memcached ? In-Reply-To: <672B125E-D9F2-419C-8B90-6E50BD3B21D4@gmail.com> References: <4518882d.128ffa5d.2598.5f7b@mx.gmail.com> <672B125E-D9F2-419C-8B90-6E50BD3B21D4@gmail.com> Message-ID: Using Eventmachie and MemCache......i am on verge of phasing out....backgroundrb. I am not sure, if it is such a nice decision.but turns out pretty fast and since i developed it....I can be little bit sure about it. The flipside is...i use TCPSocket to pass messages to the background daemons. On 9/26/06, Ezra Zygmuntowicz wrote: > > Hi- > > YEah memcached is pure C and is very fast. DRb is a bit slower for > caching. Backgroundrb has caching features but I recommend you use > memcached for scalable cache. Backgroundrb is more for the worker > classes then for caching. > > > Cheers- > -Ezra > > > On Sep 25, 2006, at 7:00 PM, awatermail wrote: > > > > > > > I do some test, surprising to find seeming to access drb much > > slower than > > access memcached. > > > > If it's true, should we use more memcached directly than drb based > > things > > (like backgrounddrb) in scale situation ? > > > > Before the test, I can't decide whether to do(such as get/set > > global data > > operations) in drb/backgrounddrb or just access memcached in > > actioncontrollers. > > > > I have thought of that doing save(or some read) operations in > > drb/backgroundrb asynchronous will improve the performance highly. > > > > Hope for your idea, thank you. > > > > water > > > > The test: > > > > ?summary? > > just loop 100 times doing a simple action with drb & memcached, see > > how long > > they will take. > > > > ?background? > > ruby 1.8.4 (2006-04-14) [i386-mswin32] > > Rails 1.1.6 > > memcache-client-1.0.3 > > Distributed Ruby: _dRuby_ version 2.0.4 # from ruby/lib/rub/1.8/ > > drb/drb.rb > > > > ?begin? > > # RAILS_ROOT/config/enviroments/development.rb > > # new drb object here to ensure just new it once at the beginning > > require 'drb' > > DRb.start_service() > > DRB_OBJ = DRbObject.new(nil, 'druby://localhost:9000') > > > > # In a simple actioncontroller > > class TestController < ApplicationController > > def test > > start_time = Time.now.to_f > > 100.times do > > DRB_OBJ.doNothingInSide() # just invoke a simple drb > > function, do > > nothing inside > > end > > end_time = Time.now.to_f > > flash['tm2'] = sprintf("DRB access take (%0.9f)", end_time - > > start_time) > > # log the duration to show in view > > > > start_time = Time.now.to_f > > 100.times do > > Cache::put('test', {'test' => 'just test'}) # just set the > > memcached > > through memcache-client's put method > > end > > end_time = Time.now.to_f > > flash['tm1'] = sprintf("Memcache access take (%0.9f)", end_time - > > start_time) # log the duration to show in view > > end > > end > > > > # In the simple view > >

tm1=#<%=flash['tm1']%>#

> >

tm2=#<%=flash['tm2']%>#

> > > > # run & see the results > > tm1=#Memcache access take (0.546000004) > > tm2=#DRB access take (3.141000032) > > > > > > The running results different every time, but all show memcached > > much faster > > than drb access. > > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google Groups "Ruby on Rails: Talk" group. > To post to this group, send email to rubyonrails-talk at googlegroups.com > To unsubscribe from this group, send email to rubyonrails-talk-unsubscribe at googlegroups.com > For more options, visit this group at http://groups.google.com/group/rubyonrails-talk > -~----------~----~----~----~------~----~------~--~--- > > -- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary. From ezmobius at gmail.com Wed Sep 27 13:54:24 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 27 Sep 2006 10:54:24 -0700 Subject: [Backgroundrb-devel] Clearing ActiveRecord Sessions In-Reply-To: <1907e2ca0609241820hac9e7b9l2e32601cc10d0f87@mail.gmail.com> References: <1907e2ca0609241820hac9e7b9l2e32601cc10d0f87@mail.gmail.com> Message-ID: <78A3C53C-6D18-4FA8-9C3B-B0A1A81BDA77@gmail.com> On Sep 24, 2006, at 6:20 PM, Michael D'Auria wrote: > Hello all, > > I am interested in using BackgrounDRb to clear out ActiveRecord > based sessions. I have this working, just with a small hitch, here > is the code: > > class ClearSessionsWorker < BackgrounDRb::Rails > > repeat_every 7.minutes > first_run Time.now > > def do_work(args) > Session.destroy_all(["updated_at < ?", 20.minute.ago ]) > end > end > > > It's not very complicated and does remove sessions, it's just that > once the job is done, the thread hangs out leaving a connection to > my database open. After some time, MySQL complains about having > too many open connections. I was just curious what i am missing. > I have tried using "kill", but that only seems to run the job once > and not every 7 minutes like i so nicely ask :). I have also tried > "terminate", but that did not seem to do anything for me. Any help > is greatly appreciated. > > Thanks, > > .: Michael :. Hi Michael- Yeah you need to call ActiveRecord::Base.connection.disconect! in your worker when it is done working. I am hard at work on a new version of the plugin that is a totaly new beast. I hope to release it soon but I am super busy these days. The new version has a much more robust timing mechanism and everything else. -Ezra From ezmobius at gmail.com Wed Sep 27 15:24:48 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 27 Sep 2006 12:24:48 -0700 Subject: [Backgroundrb-devel] BackgrounDRb Call for help In-Reply-To: <1907e2ca0609271119y191ea194kbd00f7668f13e218@mail.gmail.com> References: <1907e2ca0609271119y191ea194kbd00f7668f13e218@mail.gmail.com> Message-ID: <912A6105-B3FA-42D5-92BD-8224C5DDA6F5@gmail.com> Hey there bdrb'ers! I have been so very busy with http://engineyard.com that I have not had time to complete the new release of backgroundrb. But I have done substantial work towards it. It is basically an entirely new beast with a complete rethinking of how it works. I have put up a zip file of the new code base in the hopes that some folks will be able to start playing with it and work with me to finish it up so people can use it. So please if you feel so inclined, download this new version and have a look over the code. http://brainspl.at/backgroundrb.zip Ok here is the run down on new features and what still needs to be done to finish this up to be used. ? I have ditched the old start/stop scripts and would like to use the daemons gem to manage the daemoon as it is more robust then a hand rolled solution. So new start/stop and service scripts will need to be written using the daemons gem ? The new plugin will depend on the slave gem so make sure to install the latest gem install slave ? The new version will be multi process instead of single process. The middleman is still threaded but in a different way. It now has a ThreadPool class that you can set the size limit on. This allows for you to control how many processes get spawned at once. This makes bdrb infinitely more scalable. ? The MiddleMan class tried to do way too much in the current version. This violates a bunch of OOP rules and I decided to simplify it. So nwo the middleman just manages a thread pool of worker slaves and thats pretty much all it does. Look in the middleman_slave.rb file for the new middleman class. ? People liked using timed workers. Unfortunately the current impl of timing is sorely lacking. It works ok for very simple stuff but people have issues with it running more then one instance of the timed worker at once. This is no good. So I have created a new scheduler class. This is solely dedicated to scheduling jobs. There is a Trigger class that allows for simple repeat every style workers. But there is also a new CronTrigger class that can fire off events using cron copmpatable syntax! This means you can use cron jobs like '1 * * * * * *' or even '28/5,59 1-4,6,20 */1 * 5,0/1 * *' . this allows for a very flexible scheduler. I borrowed some of this code from the moment gem and integrated it with bdrb. Have a look at the scheduler.rb file and the scheduler test as well. ? All of the new code is in the bdrb directory in the plugin. If you want to run the new middleman_slave and see it manage multiple processes at once you can run ruby middleman_slave.rb in one terminal and then in irb here are a few commands to play with it: require 'drb' DRb.start_service('druby://localhost:0') m=DRbObject.new(nil, "druby://localhost:2000") 40.times{m.new_worker :class => :worker} m.jobs.keys m.jobs.keys.each {|o| p m.worker(o).progress} m.jobs.keys.each {|o| m[o].shutdown if m.worker(o).done} Now that we use the slave gem to manage external workers there is a slight interface change to the middleman. It works the same exact way to start workers with new_worker. But when you want to get a handle on your worker to call methods you need to use MiddleMan.worker (key).method . get_worker or MiddleMan[key] get you a handle on the slave process so you can call shutdown on it and get its PID and all that. But worker(key) gets you the handle on your actual worker object. So thats my notes for you for right now. Please feel free to write in with any questions. The main thing that would help get it back of the ground would be the daemons start stop scripts. And I need to make a new config file format and parser as well as a nice interface where you define timers. **IMPORTANT NOTE** Now that we are using multi process workers it is very important that you do not require all of rails by default. Of course some workers will need this but it would be best to spawn one of these up and re use it somehow. So we need to keep the drb server clean slim as just a manager. So each worker class will need to require what it needs for db connection and other stuff. This needs some more thought given to it. But the current single threaded bdrb is a mess, it opens way too many db handles and threading is not enough to handle a ton of workers. This new way of doing things will require a bit more work when building your worker classes, but the payoff will be huge and it will allow you to scale to real high traffic sites now. So anyone with ideas about this please pipe up. I think I will make a script you can require in your workers that will setup Active record for you. But I think that we should not be requiring all of rails it is so inefficient. I know this last point may make it harder for some folks to make workers easily because all of rails won't be available. But this was necessary in order to make bdrb truly scalable. Running an entire rails process just to run a worker class is a huge waste. So this will need to be worked out as we start to use the new plugin. I want to thank everyone for their patience, I have been so busy I haven't had time to finish this up. Any help from you guys would be very appreciated. Thoughts? -Ezra From ubermunkey at gmail.com Thu Sep 28 12:10:05 2006 From: ubermunkey at gmail.com (Rowan Hick) Date: Thu, 28 Sep 2006 12:10:05 -0400 Subject: [Backgroundrb-devel] Duplicate record weirdness ? Message-ID: First off BackgroundRb rocks!!!!. Top marks Ezra, it's gone a long way to stopping us from doing a nasty curl/cron hack :) Okay so here's what we're using it for. We're synchronising data from a Filemaker database, via web service equest, into a mysql database via a rails model called 'Syncer' [How's that for an oddball use case!]. Now this model exposes a static method which goes away and does the business. When it starts synchronising it creates a log record for it, and when it finishes it updates that log record. Fair enough. All is working beautfiully and this is the perfect candidate for using with backgroundRb. First couple of runs through it worked perfectly. Then things started going wonky. My worker is as simple as this.. class FilemakerSyncWorker < BackgrounDRb::Rails repeat_every 20.minutes first_run Time.now def do_work(args) Syncer.new.synchronise end end What's really weird, and maybe is my complete lack of understanding of threads and such like is this, randomly, throughout the day the log records created in .synchronise are getting *almost* duplicated, it's like the method is getting invoked twice at the exact same time, and finishing just slightly off the previous time. Example from logs today.. | ID | Status | Start Time | End Time | Status | 165 | Failed | 2006-09-28 10:28:23 | 2006-09-28 10:28:27 | Expected | 164 | Failed | 2006-09-28 10:08:23 | 2006-09-28 10:08:24 | Expected | 163 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:26 | Almost Duplicate (note end time) | 162 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:25 | Expected | 161 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:12 | Almost Duplicate | 160 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:13 | Expected | 159 | Complete | 2006-09-28 09:08:23 | 2006-09-28 09:09:55 | Expected I've checked the running process, and there's just the one ruby process, so it's not like it's spawning any different process. Given the end times are fractionally different of the log records it's not like the record is getting duplicated. This is on a Dual G5 box, I haven't tested any other box yet. It's not killing us as the same records are getting updated when it duplicates so we don't really care that it is duplicating, it's just odd and I don't think should be happening... Can anyone shed any light on it ? Thanks again Ezra !! Rowan --- www.rowanhick.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060928/cf951b5d/attachment.html From me at seebq.com Thu Sep 28 19:20:09 2006 From: me at seebq.com (Charles Brian Quinn) Date: Thu, 28 Sep 2006 19:20:09 -0400 Subject: [Backgroundrb-devel] Duplicate record weirdness ? In-Reply-To: References: Message-ID: <3a2de0cd0609281620w4bbc6c74gabc02478609cb97a@mail.gmail.com> Not to detract from backgroundRB because I love it myself, and am excited about the rewrite, BTW, but we have a situation where we're performing a "sync" operation with a salesforce database, and sometimes it is quite slow (and has a lot to sync) -- as such, we had to stop using backgroundrb and went to a regular cron job (ugh) using script/runner and a custom class that just uses AR. In doing so, we wrote a quick wrapper using ruby lockfile library to make sure that when a task is running, that it creates and pings a lockfile -- this way when a secondary cron set to update every 1 or 5 minutes runs, it waits a certain amount of time, and then just doesn't run if one's already running. Simliarly, if a job has been "running" for over an hour, it kills the lock and just steals it and runs. Is a nice library and the wrapper was simple enough. I have seen issues (on this list) with the timer doing strange things (duplicate calls). In case you didn't know, there's a rewrite that Ezra is squeezing in between his day job, writing a deployment book, and launching a hosting company (!) -- it is fixing part of the problems with the timer class, and he's outlined (search the archive) the steps it needs to fix. hope this helps, cheers, -- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com On 9/28/06, Rowan Hick wrote: > First off BackgroundRb rocks!!!!. Top marks Ezra, it's gone a long way to > stopping us from doing a nasty curl/cron hack :) > > Okay so here's what we're using it for. > > We're synchronising data from a Filemaker database, via web service equest, > into a mysql database via a rails model called 'Syncer' [How's that for an > oddball use case!]. Now this model exposes a static method which goes away > and does the business. When it starts synchronising it creates a log record > for it, and when it finishes it updates that log record. Fair enough. All is > working beautfiully and this is the perfect candidate for using with > backgroundRb. First couple of runs through it worked perfectly. Then things > started going wonky. > > My worker is as simple as this.. > > class FilemakerSyncWorker < BackgrounDRb::Rails > repeat_every 20.minutes > first_run Time.now > def do_work(args) > Syncer.new.synchronise > end > end > > What's really weird, and maybe is my complete lack of understanding of > threads and such like is this, randomly, throughout the day the log records > created in .synchronise are getting *almost* duplicated, it's like the > method is getting invoked twice at the exact same time, and finishing just > slightly off the previous time. Example from logs today.. > > | ID | Status | Start Time | End Time | Status > | 165 | Failed | 2006-09-28 10:28:23 | 2006-09-28 10:28:27 | Expected > | 164 | Failed | 2006-09-28 10:08:23 | 2006-09-28 10:08:24 | Expected > | 163 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:26 | Almost > Duplicate (note end time) > | 162 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:25 | Expected > | 161 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:12 | Almost > Duplicate > | 160 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:13 | Expected > | 159 | Complete | 2006-09-28 09:08:23 | 2006-09-28 09:09:55 | Expected > > I've checked the running process, and there's just the one ruby process, so > it's not like it's spawning any different process. Given the end times are > fractionally different of the log records it's not like the record is > getting duplicated. This is on a Dual G5 box, I haven't tested any other box > yet. It's not killing us as the same records are getting updated when it > duplicates so we don't really care that it is duplicating, it's just odd and > I don't think should be happening... > > Can anyone shed any light on it ? > > Thanks again Ezra !! > > > Rowan > --- > www.rowanhick.com > > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > From ubermunkey at gmail.com Fri Sep 29 10:14:52 2006 From: ubermunkey at gmail.com (Rowan Hick) Date: Fri, 29 Sep 2006 10:14:52 -0400 Subject: [Backgroundrb-devel] Duplicate record weirdness ? In-Reply-To: <3a2de0cd0609281620w4bbc6c74gabc02478609cb97a@mail.gmail.com> References: <3a2de0cd0609281620w4bbc6c74gabc02478609cb97a@mail.gmail.com> Message-ID: Hey Charles, thanks for the reply. Interesting approach, I may just introduce a quick hack to keep things ticking along for the moment. Just read the call for help email and will try out the new version. Cheers R On 9/28/06, Charles Brian Quinn wrote: > > Not to detract from backgroundRB because I love it myself, and am > excited about the rewrite, BTW, but we have a situation where we're > performing a "sync" operation with a salesforce database, and > sometimes it is quite slow (and has a lot to sync) -- as such, we had > to stop using backgroundrb and went to a regular cron job (ugh) using > script/runner and a custom class that just uses AR. > > In doing so, we wrote a quick wrapper using ruby lockfile library to > make sure that when a task is running, that it creates and pings a > lockfile -- this way when a secondary cron set to update every 1 or 5 > minutes runs, it waits a certain amount of time, and then just doesn't > run if one's already running. Simliarly, if a job has been "running" > for over an hour, it kills the lock and just steals it and runs. Is a > nice library and the wrapper was simple enough. > > I have seen issues (on this list) with the timer doing strange things > (duplicate calls). In case you didn't know, there's a rewrite that > Ezra is squeezing in between his day job, writing a deployment book, > and launching a hosting company (!) -- it is fixing part of the > problems with the timer class, and he's outlined (search the archive) > the steps it needs to fix. > > hope this helps, cheers, > -- > Charles Brian Quinn > self-promotion: www.seebq.com > highgroove studios: www.highgroove.com > slingshot hosting: www.slingshothosting.com > > > On 9/28/06, Rowan Hick wrote: > > First off BackgroundRb rocks!!!!. Top marks Ezra, it's gone a long way > to > > stopping us from doing a nasty curl/cron hack :) > > > > Okay so here's what we're using it for. > > > > We're synchronising data from a Filemaker database, via web service > equest, > > into a mysql database via a rails model called 'Syncer' [How's that for > an > > oddball use case!]. Now this model exposes a static method which goes > away > > and does the business. When it starts synchronising it creates a log > record > > for it, and when it finishes it updates that log record. Fair enough. > All is > > working beautfiully and this is the perfect candidate for using with > > backgroundRb. First couple of runs through it worked perfectly. Then > things > > started going wonky. > > > > My worker is as simple as this.. > > > > class FilemakerSyncWorker < BackgrounDRb::Rails > > repeat_every 20.minutes > > first_run Time.now > > def do_work(args) > > Syncer.new.synchronise > > end > > end > > > > What's really weird, and maybe is my complete lack of understanding of > > threads and such like is this, randomly, throughout the day the log > records > > created in .synchronise are getting *almost* duplicated, it's like the > > method is getting invoked twice at the exact same time, and finishing > just > > slightly off the previous time. Example from logs today.. > > > > | ID | Status | Start Time | End Time | Status > > | 165 | Failed | 2006-09-28 10:28:23 | 2006-09-28 10:28:27 | Expected > > | 164 | Failed | 2006-09-28 10:08:23 | 2006-09-28 10:08:24 | > Expected > > | 163 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:26 | Almost > > Duplicate (note end time) > > | 162 | Complete | 2006-09-28 09:48:23 | 2006-09-28 09:49:25 | Expected > > | 161 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:12 | Almost > > Duplicate > > | 160 | Complete | 2006-09-28 09:28:23 | 2006-09-28 09:29:13 | Expected > > | 159 | Complete | 2006-09-28 09:08:23 | 2006-09-28 09:09:55 | Expected > > > > I've checked the running process, and there's just the one ruby process, > so > > it's not like it's spawning any different process. Given the end times > are > > fractionally different of the log records it's not like the record is > > getting duplicated. This is on a Dual G5 box, I haven't tested any other > box > > yet. It's not killing us as the same records are getting updated when it > > duplicates so we don't really care that it is duplicating, it's just odd > and > > I don't think should be happening... > > > > Can anyone shed any light on it ? > > > > Thanks again Ezra !! > > > > > > Rowan > > --- > > www.rowanhick.com > > > > > > > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20060929/63ca2ba7/attachment-0001.html From david at lemstra.ca Fri Sep 29 14:32:52 2006 From: david at lemstra.ca (David Lemstra) Date: Fri, 29 Sep 2006 14:32:52 -0400 Subject: [Backgroundrb-devel] BackgrounDRb Call for help In-Reply-To: <912A6105-B3FA-42D5-92BD-8224C5DDA6F5@gmail.com> References: <1907e2ca0609271119y191ea194kbd00f7668f13e218@mail.gmail.com> <912A6105-B3FA-42D5-92BD-8224C5DDA6F5@gmail.com> Message-ID: <451D66D4.1080500@lemstra.ca> Ezra, I agree w/ you on pretty much all points. The current version is doing what I need it to do, but all the thread/mutex juggling is making it much slower/ more complex/ more flaky than it needs to be (especially w/ my queue_worker). Plus doing compute intensive tasks don't (to my knowledge) take advantage of SMP setups. I've come to realize that ruby doesn't really cut it for my compute tasks, so I'm biting the bullet and migrating that part to pure C. DRb2 sounds like a perfect fit. I'm looking forward to checking out your new code at that point for the process based system. I learned most of what I know about threading starting from your original code, so hey, why not for process control too? :) It'll be at least a week before I get to it though. I'm in the same boat as you. Have to prioritize getting the $$ rolling in before making things perfect. Sigh... Yeah, loading all of rails is a huge waste of memory. I'm thinking along the lines of a checklist of rails pieces for each worker class that can be individually enabled/disabled. ActiveRecord, if enabled, would be fed a sublist of models to be loaded. Anyways, it'll get there... Thanks a bunch for all the hard work Ezra, David Lemstra Ezra Zygmuntowicz wrote: > Hey there bdrb'ers! > > I have been so very busy with http://engineyard.com that I have not > had time to complete the new release of backgroundrb. But I have done > substantial work towards it. It is basically an entirely new beast > with a complete rethinking of how it works. I have put up a zip file > of the new code base in the hopes that some folks will be able to > start playing with it and work with me to finish it up so people can > use it. So please if you feel so inclined, download this new version > and have a look over the code. > > http://brainspl.at/backgroundrb.zip > > Ok here is the run down on new features and what still needs to be > done to finish this up to be used. > > ? I have ditched the old start/stop scripts and would like to use the > daemons gem to manage the daemoon as it is more robust then a hand > rolled solution. So new start/stop and service scripts will need to > be written using the daemons gem > > ? The new plugin will depend on the slave gem so make sure to install > the latest gem install slave > > ? The new version will be multi process instead of single process. > The middleman is still threaded but in a different way. It now has a > ThreadPool class that you can set the size limit on. This allows for > you to control how many processes get spawned at once. This makes > bdrb infinitely more scalable. > > ? The MiddleMan class tried to do way too much in the current > version. This violates a bunch of OOP rules and I decided to simplify > it. So nwo the middleman just manages a thread pool of worker slaves > and thats pretty much all it does. Look in the middleman_slave.rb > file for the new middleman class. > > ? People liked using timed workers. Unfortunately the current impl of > timing is sorely lacking. It works ok for very simple stuff but > people have issues with it running more then one instance of the > timed worker at once. This is no good. So I have created a new > scheduler class. This is solely dedicated to scheduling jobs. There > is a Trigger class that allows for simple repeat every style workers. > But there is also a new CronTrigger class that can fire off events > using cron copmpatable syntax! This means you can use cron jobs like > '1 * * * * * *' or even '28/5,59 1-4,6,20 */1 * 5,0/1 * *' . this > allows for a very flexible scheduler. I borrowed some of this code > from the moment gem and integrated it with bdrb. Have a look at the > scheduler.rb file and the scheduler test as well. > > ? All of the new code is in the bdrb directory in the plugin. If you > want to run the new middleman_slave and see it manage multiple > processes at once you can run ruby middleman_slave.rb in one terminal > and then in irb here are a few commands to play with it: > > require 'drb' > DRb.start_service('druby://localhost:0') > m=DRbObject.new(nil, "druby://localhost:2000") > 40.times{m.new_worker :class => :worker} > m.jobs.keys > m.jobs.keys.each {|o| p m.worker(o).progress} > m.jobs.keys.each {|o| m[o].shutdown if m.worker(o).done} > > > Now that we use the slave gem to manage external workers there is a > slight interface change to the middleman. It works the same exact way > to start workers with new_worker. But when you want to get a handle > on your worker to call methods you need to use MiddleMan.worker > (key).method . get_worker or MiddleMan[key] get you a handle on the > slave process so you can call shutdown on it and get its PID and all > that. But worker(key) gets you the handle on your actual worker object. > > > So thats my notes for you for right now. Please feel free to write > in with any questions. The main thing that would help get it back of > the ground would be the daemons start stop scripts. And I need to > make a new config file format and parser as well as a nice interface > where you define timers. > > **IMPORTANT NOTE** > > Now that we are using multi process workers it is very important > that you do not require all of rails by default. Of course some > workers will need this but it would be best to spawn one of these up > and re use it somehow. > > So we need to keep the drb server clean slim as just a manager. So > each worker class will need to require what it needs for db > connection and other stuff. This needs some more thought given to it. > But the current single threaded bdrb is a mess, it opens way too many > db handles and threading is not enough to handle a ton of workers. > This new way of doing things will require a bit more work when > building your worker classes, but the payoff will be huge and it will > allow you to scale to real high traffic sites now. So anyone with > ideas about this please pipe up. I think I will make a script you can > require in your workers that will setup Active record for you. But I > think that we should not be requiring all of rails it is so inefficient. > > > I know this last point may make it harder for some folks to make > workers easily because all of rails won't be available. But this was > necessary in order to make bdrb truly scalable. Running an entire > rails process just to run a worker class is a huge waste. So this > will need to be worked out as we start to use the new plugin. > > I want to thank everyone for their patience, I have been so busy I > haven't had time to finish this up. Any help from you guys would be > very appreciated. > > > Thoughts? > > -Ezra