From eden.li at gmail.com Wed Oct 4 08:46:17 2006 From: eden.li at gmail.com (Eden Li) Date: Wed, 4 Oct 2006 20:46:17 +0800 Subject: [Backgroundrb-devel] first_run timezone weirdness Message-ID: <565dbdd40610040546ue8e51bdw52013dc256f756ab@mail.gmail.com> If you're in a timezone with an abbreviation that map to more than one timezone (eg, CST), first_run may not work as you expect it to. Witness: $ irb -r time irb(main):001:0> Time.now => Wed Oct 04 20:37:24 CST 2006 irb(main):002:0> Time.parse(Time.now.to_s) => Thu Oct 05 10:37:30 CST 2006 Notice the 14 extra hours... Time.parse converted the time thinking CST meant -06:00 when my system time is actually on the +08:00 flavor of CST. So, BackgroundDRb::Rails#first_run creates a method that turns it from a Time.now into a (practical) Time.parse(Time.now.to_s) which will cause the first job to run 14 hours later than it should. There a few ways to fix this, right now the hack I've come up with is to pass in Time.now without the timezone letters so Time.parse will convert the time using the local timezone: def MyWorker < BackgroundDRb::Rails first_run Time.now.to_s.sub(/[A-Z]{3}/, '') ... end The other option is to patch backgroundrb to use UTC/GMT solely for comparison -- convert the passed in time to GMT then stringify it for the class eval'ed method -- then compare it against Time.now.gmtime in start_process. I would gladly do this if only I could check out the branch from rubyforge (is this down?) From jcfischer.lists at gmail.com Wed Oct 4 10:49:04 2006 From: jcfischer.lists at gmail.com (Jens-Christian Fischer) Date: Wed, 4 Oct 2006 16:49:04 +0200 Subject: [Backgroundrb-devel] do_work and long running rails tasks? Message-ID: Hi there I have some long running queries against a nice litte database (20 million new records per day) that I have offloaded to backgroundRB. The trouble is, that backgroundrb seems to be blockin on Model.find calls. Here's the code --- worker.rb -- def do_work(args) @rows = [] @percent = 0 @host = Host.find( args[:id]) ips = @host.ipaddresses step = 100.0/ips.size ips.each do |ip| @percent += step new_rows = Sysloglog.find(:all, :limit => 20, :offset => 0, :order => "eventtime DESC", :conditions => [ "assets_ipaddr_id = ?", ip.id ] ) @rows << new_rows.collect { |s| s.id } end self.kill rescue Exception => e @logger.error "Exception: #{e}" @percent = 100 self.kill end -- snip --- --- controller.rb -- def syslog_progress if request.xhr? worker = MiddleMan.get_worker(session[:host_info]) if worker progress_percent = worker.progress render :update do |page| page.call('progressPercent', 'progressbar', progress_percent) @rows = Sysloglog.find( worker.rows) rescue [] page.replace_html 'syslog_rows', :partial => "analysis/ syslog", :object => @rows if progress_percent >= 100 page.assign 'stop_polling', true #<< here is the magick! page.hide 'progress' MiddleMan.delete_worker(session[:host_info]) end end else render :update do |page| end end else redirect_to :action => 'list' end end -- snip -- An AJAX call to syslog_progress blocks for the time it takes to do the find( :all .... ) (which sort of renders the whole point of the sending this task to the background moot) Is this the way it is? Any other ways of doing this? Thanks Jens-Christian From michael.dauria at gmail.com Fri Oct 6 13:13:55 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Fri, 6 Oct 2006 13:13:55 -0400 Subject: [Backgroundrb-devel] Ezra's Call For Help... Message-ID: <1907e2ca0610061013n1b657d2p172ba207d425b054@mail.gmail.com> So has anyone thought about tackling his ToDo list? I would like to contribute if i can, but i don't want people making changes and then find out someone has already done so. I guess what i am saying is that i can step up and help organize the changes that people would like to modify and aid in testing them. What does everyone think? Does anyone have any items they would like to tackle? Let's get BackgrounDRb to where Ezra would like, it's the least we can do for having him give this out for free in the first place :) .: Michael :. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061006/360d9a77/attachment.html From ctmailinglists at googlemail.com Thu Oct 12 11:18:59 2006 From: ctmailinglists at googlemail.com (Chris T) Date: Thu, 12 Oct 2006 16:18:59 +0100 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed Message-ID: <452E5CE3.9070406@gmail.com> Can't believe I can't see the problem here, but after a few hours bashing my head on a brick wall, going to risk looking stupid by seeing if anyone can point out my idiocy (that's the worst thing about coding on your own). Trying to get BackgrounDRb to take over the scraping task which is currently being done in a Rails controller. That's not the problem at the moment, as I can't even get past first base. For some reason, it doesn't seem to be setting up my worker class properly. Here's my worker class def: |class ScrapeWorker < BackgrounDRb::Rails attr_accessor :progress, :message, :ids_retrieved def do_work(args) @progress = 0 @message = args[:message] || "" @job_completed = false @hse_notices_retrieved = 0 puts args[:message] get_new_ids(args) end def get_new_ids(args) .... end end| However, I get NoMethodError (undefined method `ids_retrieved' for #) Set up a quick unit test, and not surprisingly fails on first respond_to | def test_should_respond_to_progress_message_and_hse_notices_retrieved MiddleMan.new_worker(:class => :scrape_worker, :args => {:message => "Uh -Oh"}, :job_key => :test_scraper, :ttl => 5) s = MiddleMan[:test_scraper] assert s.respond_to?(:progress) assert s.respond_to?(:message) assert s.respond_to?(:hse_notices_retrieved) assert_equal "Uh -Oh", s.message end| I'm sure it must be something really obviously but can't for the life of me work out what. Any ideas? Thanks in advance Chris p.s. I'm on OS X, though not sure that's relevant here... From me at seebq.com Thu Oct 12 11:49:59 2006 From: me at seebq.com (Charles Brian Quinn) Date: Thu, 12 Oct 2006 11:49:59 -0400 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <452E5CE3.9070406@gmail.com> References: <452E5CE3.9070406@gmail.com> Message-ID: <3a2de0cd0610120849s66d9a887nfcfe2d7095302a17@mail.gmail.com> should it be: s = MiddleMan.get_worker(:test_scraper) ? On 10/12/06, Chris T wrote: > Can't believe I can't see the problem here, but after a few hours > bashing my head on a brick wall, going to risk looking stupid by seeing > if anyone can point out my idiocy (that's the worst thing about coding > on your own). > > Trying to get BackgrounDRb to take over the scraping task which is > currently being done in a Rails controller. That's not the problem at > the moment, as I can't even get past first base. For some reason, it > doesn't seem to be setting up my worker class properly. > > Here's my worker class def: > > |class ScrapeWorker < BackgrounDRb::Rails > attr_accessor :progress, :message, :ids_retrieved > > def do_work(args) > @progress = 0 > @message = args[:message] || "" > @job_completed = false > @hse_notices_retrieved = 0 > puts args[:message] > get_new_ids(args) > end > > def get_new_ids(args) > .... > end > end| > > However, I get NoMethodError (undefined method `ids_retrieved' for > #) > > Set up a quick unit test, and not surprisingly fails on first respond_to > > | def test_should_respond_to_progress_message_and_hse_notices_retrieved > MiddleMan.new_worker(:class => :scrape_worker, :args => > {:message => "Uh -Oh"}, :job_key => :test_scraper, :ttl => 5) > s = MiddleMan[:test_scraper] > assert s.respond_to?(:progress) > assert s.respond_to?(:message) > assert s.respond_to?(:hse_notices_retrieved) > assert_equal "Uh -Oh", s.message > end| > > I'm sure it must be something really obviously but can't for the life of > me work out what. Any ideas? > > Thanks in advance > Chris > > p.s. I'm on OS X, though not sure that's relevant here... > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com From fred at 82ask.com Thu Oct 12 12:37:38 2006 From: fred at 82ask.com (Frederick Cheung) Date: Thu, 12 Oct 2006 17:37:38 +0100 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <452E5CE3.9070406@gmail.com> References: <452E5CE3.9070406@gmail.com> Message-ID: <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> On 12 Oct 2006, at 16:18, Chris T wrote: > Can't believe I can't see the problem here, but after a few hours > bashing my head on a brick wall, going to risk looking stupid by > seeing > if anyone can point out my idiocy (that's the worst thing about coding > on your own). > Random guess: you haven't restarted the backgroundrb process since you've added the progress method ? Fred From ezmobius at gmail.com Thu Oct 12 14:32:27 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 12 Oct 2006 11:32:27 -0700 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> References: <452E5CE3.9070406@gmail.com> <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> Message-ID: <93A29A52-7834-40F3-A362-9C81FA0AA334@brainspl.at> On Oct 12, 2006, at 9:37 AM, Frederick Cheung wrote: > > On 12 Oct 2006, at 16:18, Chris T wrote: > >> Can't believe I can't see the problem here, but after a few hours >> bashing my head on a brick wall, going to risk looking stupid by >> seeing >> if anyone can point out my idiocy (that's the worst thing about >> coding >> on your own). >> > Random guess: you haven't restarted the backgroundrb process since > you've added the progress method ? > > Fred Also I am not entirely sure that you can use respond_to? across a drb connection like that. Because drb uses method missing internally to dispatch across the wire it won't be able to respond_to? what you are trying to do. Is it just the test case that is failing? -EZra From ezmobius at gmail.com Thu Oct 12 15:33:18 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 12 Oct 2006 12:33:18 -0700 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <452E9752.7030405@gmail.com> References: <452E5CE3.9070406@gmail.com> <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> <93A29A52-7834-40F3-A362-9C81FA0AA334@brainspl.at> <452E9752.7030405@gmail.com> Message-ID: <9DBFFE9E-3C86-425B-9CD5-E446BB6C9EE5@gmail.com> On Oct 12, 2006, at 12:28 PM, Chris T wrote: > Ezra Zygmuntowicz wrote: >> >> On Oct 12, 2006, at 9:37 AM, Frederick Cheung wrote: >> >>> >>> On 12 Oct 2006, at 16:18, Chris T wrote: >>> >>>> Can't believe I can't see the problem here, but after a few hours >>>> bashing my head on a brick wall, going to risk looking stupid by >>>> seeing >>>> if anyone can point out my idiocy (that's the worst thing about >>>> coding >>>> on your own). >>>> >>> Random guess: you haven't restarted the backgroundrb process since >>> you've added the progress method ? >>> >>> Fred >> >> >> Also I am not entirely sure that you can use respond_to? >> across a drb connection like that. Because drb uses method missing >> internally to dispatch across the wire it won't be able to >> respond_to? what you are trying to do. Is it just the test case >> that is failing? >> >> -EZra >> > Ezra > I've tried doing this through the console, just to see where the > prob is, but keep having difficulties with all the requires. Do I > need to do all the requires in the test case, including requiring > the worker? Keep getting uninitialized constant... Could that be a > clue, or won't it work via the console. > Cheers for your help > Chris Chris- It will work fine from the console as long as the backgroundrb server is running before you start script/console then you can just access the MidleMan directly. Make sure you have entirely killed the drb server and then restart it and drop into script/console and try it from there. Does it work if you try to use one of the example workers at all? Have you ever had the plugin working yet and just now its broken or what? Also I see in your unit test that you are testing for hse_notices_retrieved but you didnt make an attr_accessor for it. From what you have showed me I don't see any reason why it shouldn't be working. Make sure that you have really killed the server and then restart it and try again. -Ezra From ctmailinglists at googlemail.com Fri Oct 13 05:13:28 2006 From: ctmailinglists at googlemail.com (Chris T) Date: Fri, 13 Oct 2006 10:13:28 +0100 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <9DBFFE9E-3C86-425B-9CD5-E446BB6C9EE5@gmail.com> References: <452E5CE3.9070406@gmail.com> <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> <93A29A52-7834-40F3-A362-9C81FA0AA334@brainspl.at> <452E9752.7030405@gmail.com> <9DBFFE9E-3C86-425B-9CD5-E446BB6C9EE5@gmail.com> Message-ID: <452F58B8.3010503@gmail.com> Ezra Zygmuntowicz wrote: > > On Oct 12, 2006, at 12:28 PM, Chris T wrote: > >> Ezra Zygmuntowicz wrote: >>> >>> On Oct 12, 2006, at 9:37 AM, Frederick Cheung wrote: >>> >>>> >>>> On 12 Oct 2006, at 16:18, Chris T wrote: >>>> >>>>> Can't believe I can't see the problem here, but after a few hours >>>>> bashing my head on a brick wall, going to risk looking stupid by >>>>> seeing >>>>> if anyone can point out my idiocy (that's the worst thing about >>>>> coding >>>>> on your own). >>>>> >>>> Random guess: you haven't restarted the backgroundrb process since >>>> you've added the progress method ? >>>> >>>> Fred >>> >>> >>> Also I am not entirely sure that you can use respond_to? across >>> a drb connection like that. Because drb uses method missing >>> internally to dispatch across the wire it won't be able to >>> respond_to? what you are trying to do. Is it just the test case that >>> is failing? >>> >>> -EZra >>> >> Ezra >> I've tried doing this through the console, just to see where the prob >> is, but keep having difficulties with all the requires. Do I need to >> do all the requires in the test case, including requiring the worker? >> Keep getting uninitialized constant... Could that be a clue, or won't >> it work via the console. >> Cheers for your help >> Chris > > Chris- > > It will work fine from the console as long as the backgroundrb > server is running before you start script/console then you can just > access the MidleMan directly. Make sure you have entirely killed the > drb server and then restart it and drop into script/console and try it > from there. Does it work if you try to use one of the example workers > at all? Have you ever had the plugin working yet and just now its > broken or what? Also I see in your unit test that you are testing for > hse_notices_retrieved but you didnt make an attr_accessor for it. From > what you have showed me I don't see any reason why it shouldn't be > working. Make sure that you have really killed the server and then > restart it and try again. > > -Ezra > OK. Sorted this now. It was one those those multiple bugs -- where a whole lot of things were not working quite right, and so making it difficult to zero in on the problem. First, the problems were hidden by two things -- I was running autotest in the background, which was of course rerunning the test every time I changed anything. Second, I had got the sleep set to the default of 60, so the worker wasn't always dead by the time the test was re-run [Note to self: not much point in having TTL less than the sleep time]. Sometime it was, sometime it wasn't. Result: inconsistency in error messages. Third, when trying different combinations of things, I tried not submitting any args. Result: more apparent (though not actual) weirdness: of course the line @message = args[:message] || "" fails because args is then nil, not an empty Hash, so the setup fails at that point. Hence, why it wasn't responding to the accessor methods. Finally, I hadn't really appreciated (i.e. it hadn't properly sunk in) that you had to restart backgroundrb every time you made a change in the worker class. I thought I was doing it, but wit all the inconsistency, I'm now not 100% sure. Anyway, all working now. For future ref the respond_to? works fine as a way of testing the accessor methods have been loaded OK, and I'd advise anyone who is running autotest to turn it off while debugging worker classes -- the combination of TTL, sleep, quickly running tests, and the need to restart backgroundrb makes life way to messy. Thanks for your help, and thanks for the plugin. Chris p.s. Is it possible to have different configs for development, test and production? From ctmailinglists at googlemail.com Fri Oct 13 05:55:07 2006 From: ctmailinglists at googlemail.com (Chris T) Date: Fri, 13 Oct 2006 10:55:07 +0100 Subject: [Backgroundrb-devel] BackgrounDRb newbie stuck at first base. Fresh pair of eyes needed In-Reply-To: <9DBFFE9E-3C86-425B-9CD5-E446BB6C9EE5@gmail.com> References: <452E5CE3.9070406@gmail.com> <7B70084E-B6FC-434A-B522-E29AD7B250E3@82ask.com> <93A29A52-7834-40F3-A362-9C81FA0AA334@brainspl.at> <452E9752.7030405@gmail.com> <9DBFFE9E-3C86-425B-9CD5-E446BB6C9EE5@gmail.com> Message-ID: <452F627B.707@gmail.com> Ezra Zygmuntowicz wrote: > > On Oct 12, 2006, at 12:28 PM, Chris T wrote: > >> Ezra Zygmuntowicz wrote: >>> >>> On Oct 12, 2006, at 9:37 AM, Frederick Cheung wrote: >>> >>>> >>>> On 12 Oct 2006, at 16:18, Chris T wrote: >>>> >>>>> Can't believe I can't see the problem here, but after a few hours >>>>> bashing my head on a brick wall, going to risk looking stupid by >>>>> seeing >>>>> if anyone can point out my idiocy (that's the worst thing about >>>>> coding >>>>> on your own). >>>>> >>>> Random guess: you haven't restarted the backgroundrb process since >>>> you've added the progress method ? >>>> >>>> Fred >>> >>> >>> Also I am not entirely sure that you can use respond_to? across >>> a drb connection like that. Because drb uses method missing >>> internally to dispatch across the wire it won't be able to >>> respond_to? what you are trying to do. Is it just the test case that >>> is failing? >>> >>> -EZra >>> >> Ezra >> I've tried doing this through the console, just to see where the prob >> is, but keep having difficulties with all the requires. Do I need to >> do all the requires in the test case, including requiring the worker? >> Keep getting uninitialized constant... Could that be a clue, or won't >> it work via the console. >> Cheers for your help >> Chris > > Chris- > > It will work fine from the console as long as the backgroundrb > server is running before you start script/console then you can just > access the MidleMan directly. Make sure you have entirely killed the > drb server and then restart it and drop into script/console and try it > from there. Does it work if you try to use one of the example workers > at all? Have you ever had the plugin working yet and just now its > broken or what? Also I see in your unit test that you are testing for > hse_notices_retrieved but you didnt make an attr_accessor for it. From > what you have showed me I don't see any reason why it shouldn't be > working. Make sure that you have really killed the server and then > restart it and try again. > > -Ezra > One other thing, if backgroundrb crashes or goes off on one (easy to do when you're doing screen scraping, even if you are using mocks), rake backgroundrb:stop doesn't stop the process (although it throws no error or warning). I had to go into OS X's activity monitor to kill it. Think that was possibly causing some of the erratic behaviour. Too new to OS X to know whether this is a Mac thing, and haven't had a chance to look into the rake tasks yet, but just thought it was worth reporting. Cheers CT From ctmailinglists at googlemail.com Sat Oct 14 05:35:30 2006 From: ctmailinglists at googlemail.com (Chris T) Date: Sat, 14 Oct 2006 10:35:30 +0100 Subject: [Backgroundrb-devel] Setting environment Message-ID: <4530AF62.2020204@gmail.com> Not sure this is known behaviour but it seems that if want to set the environment to, for example, test, and you want to use Active Record you have to both explicitly set the RAILS_ENV and the BackgrounDRb environment. So, if you have a config file backgroundrb_test.yml (as well as the default) and set the environment to test in that, this is what *seems* to happen: $ script/backgroundrb/start # starts backgroundrb in development environment and uses AR in development environment $ script/backgroundrb/start -c backgroundrb_test # starts backgroundrb in test environment but uses AR in development environment, leading to weird shit with fixtures and so on. $ script/backgroundrb/start -c backgroundrb_test # starts backgroundrb in test environment but uses AR in development environment, leading to weird shit with fixtures and so on. $ RAILS_ENV=test script/backgroundrb/start # starts backgroundrb in development environment but I think uses AR in test environment -- not sure what end result is here. $ RAILS_ENV=test script/backgroundrb/start -c backgroundrb_test # starts backgroundrb in test environment and AR in test environment. All tickety-boo and working as expected. As I said, this may be expected behaviour, but it confused me for a while and meant my tests (which use mocks) seemed to behaving very oddly. Hope this helps someone else in the future. Cheers CT From steve at finagle.org Sun Oct 15 15:27:54 2006 From: steve at finagle.org (Steve Sloan) Date: Sun, 15 Oct 2006 12:27:54 -0700 Subject: [Backgroundrb-devel] Setting environment In-Reply-To: <4530AF62.2020204@gmail.com> References: <4530AF62.2020204@gmail.com> Message-ID: <45328BBA.6090009@finagle.org> Chris T wrote: > Not sure this is known behaviour but it seems that if want to set the > environment to, for example, test, and you want to use Active Record you > have to both explicitly set the RAILS_ENV and the BackgrounDRb environment. I ran into essentially the same problem (needing to run BRB in the test environment so AR would use the right DB). My "solution" was to add entries in backgroundrb.yml for environment, log_file and pid_file to allow for multiple instances of BRB to run simultaneously in different environments. My config now looks like: environment: <%= ENV['RAILS_ENV'] %> port: <%= 22222 + ['production', 'development', 'test'].index(ENV['RAILS_ENV']) %> log_file: <%= "log/brb-#{ENV['RAILS_ENV']}.log" %> pid_file: <%= "log/brb-#{ENV['RAILS_ENV']}.pid" %> ... Obviously, I had to hack BRB to use these settings, modify the start/stop scripts to take an --env argument, and add Rake tasks to (re)start the test instance. Add that restart task as a preqreq of 'test', and the BRB test instance will restart (and reload) before each test run. If interested, I can provide diffs ... -- Steve From xmlblog at gmail.com Mon Oct 16 11:06:39 2006 From: xmlblog at gmail.com (Christian Romney) Date: Mon, 16 Oct 2006 11:06:39 -0400 Subject: [Backgroundrb-devel] MiddleMan singleton issue in start script Message-ID: Hi all, I found this thread in the August backgroundrb-devel archives. >This means that in order to reference the new_worker method *in the >instance that is running in the drb server* you can't do that >directly. In order to spawn other workers from within a worker I >would have to change the MiddleMan class to be a singleton. This way >you could grab the one instance of it and call new_worker on that and >it would work as expected. But the way it is currently setup, you >don't have a reference to the Main MiddleMan instance from within >your workers. > I don't think making the MiddleMan class be a singleton would affect >any current code but I will have to test this. If it doesn't interfere then I think it makes sense to turn the MiddleMan into a >singleton. that way you cold gain access to it from within worker >classes. I didn't see anything after that date complaining about the issue I'm seeing, so here it is: The start script has two lines of code that call MiddleMan.new. This results in a NoMethodError, because when you include Singleton it makes this method private. Attached is my patch, which I've also submitted on Rubyforge. It calls MiddleMan.instance, but the parameters to new get nuked. This got my start script working again, but I read that the whole start script will change soon when the new version using slave comes out, so I'm not sure it'll do anyone much good. Regardless, passing it on for good karma. This patch was created against the current rev (as of this writing), which is 47. -- Best Regards, Christian Romney http://www.xml-blog.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/a7cda523/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: fixed_singleton_in_start-rev47.patch Type: application/octet-stream Size: 477 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/a7cda523/attachment.obj From xmlblog at gmail.com Mon Oct 16 11:32:47 2006 From: xmlblog at gmail.com (Christian Romney) Date: Mon, 16 Oct 2006 11:32:47 -0400 Subject: [Backgroundrb-devel] Start script is perfectly fine or How to look like an idiot Message-ID: Ok let this be a lesson to the general public. If you want to look as idiotic as I do right now do the following: Stay up all night and attempt to debug code the next day without coffee Patch the wrong file RAILS_ROOT/scripts/backgroundrb/start instead of one in plugin dir. Create a patch in the right location containing nothing but an extra space Send it to the whole Ruby community AND file a bug. Only afterward, look at any of the damn files to discover that you've made an ass of yourself. Got it? Good! Apologies to everyone. Apparently when I updated the plugin and re-ran rake backgroundrb:setup it did not overwrite my old version of the start script with the new, correct one already in trunk. -- Best Regards, Christian Romney -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/3923eb11/attachment-0001.html From bill.walton at charter.net Mon Oct 16 13:18:21 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 16 Oct 2006 12:18:21 -0500 Subject: [Backgroundrb-devel] backgroundrb.yml not generated Message-ID: <015d01c6f147$15ee0410$6401a8c0@Presario> I'm trying to do the tutorial at http://www.infoq.com/articles/BackgrounDRb. A little more than half-way through it says "Upon instalation, the plugin writes a config file into RAILS_ROOT/config/backgroundrb.yml." It didn't happen for me. I installed the plugin with: ruby script\plugin install svn:\\rubyforge.org//var/svn/backgroundrb It added a bunch of files, but not backgroundrb.yml. Any idea what I might have done wrong? Thanks, Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/a51bad8d/attachment.html From eimorton at gmail.com Mon Oct 16 13:56:33 2006 From: eimorton at gmail.com (Erik Morton) Date: Mon, 16 Oct 2006 13:56:33 -0400 Subject: [Backgroundrb-devel] backgroundrb.yml not generated In-Reply-To: <015d01c6f147$15ee0410$6401a8c0@Presario> References: <015d01c6f147$15ee0410$6401a8c0@Presario> Message-ID: I think you need to run rake backgroundrb:setup On Oct 16, 2006, at 1:18 PM, Bill Walton wrote: > I'm trying to do the tutorial at http://www.infoq.com/articles/ > BackgrounDRb. A little more than half-way through it says "Upon > instalation, the plugin writes a config file into RAILS_ROOT/config/ > backgroundrb.yml." It didn't happen for me. > > I installed the plugin with: > ruby script\plugin install svn:\\rubyforge.org//var/svn/backgroundrb > > It added a bunch of files, but not backgroundrb.yml. > > Any idea what I might have done wrong? > > Thanks, > Bill > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From michael.dauria at gmail.com Mon Oct 16 13:58:28 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Mon, 16 Oct 2006 13:58:28 -0400 Subject: [Backgroundrb-devel] backgroundrb.yml not generated In-Reply-To: <015d01c6f147$15ee0410$6401a8c0@Presario> References: <015d01c6f147$15ee0410$6401a8c0@Presario> Message-ID: <1907e2ca0610161058x340adff5ua4f3d98021337fa3@mail.gmail.com> Not sure what happened, but here is a basic one: --- port: "22222" timer_sleep: 60 load_rails: true environment: production host: localhost database_yml: config/database.yml acl: deny: all allow: localhost 127.0.0.1 order: deny,allow On 10/16/06, Bill Walton wrote: > > I'm trying to do the tutorial at > http://www.infoq.com/articles/BackgrounDRb. A little more than half-way > through it says "Upon instalation, the plugin writes a config file into > RAILS_ROOT/config/backgroundrb.yml." It didn't happen for me. > > I installed the plugin with: > ruby script\plugin install svn:\\rubyforge.org//var/svn/backgroundrb > > It added a bunch of files, but not backgroundrb.yml. > > Any idea what I might have done wrong? > > Thanks, > Bill > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/85795921/attachment.html From bill.walton at charter.net Mon Oct 16 14:06:48 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 16 Oct 2006 13:06:48 -0500 Subject: [Backgroundrb-devel] backgroundrb.yml not generated References: <015d01c6f147$15ee0410$6401a8c0@Presario> Message-ID: <019501c6f14d$ddb5c1d0$6401a8c0@Presario> Hi Erik, Erik Morton wrote: >I think you need to run rake backgroundrb:setup Thank you. That did it. Best regards, Bill From ezmobius at gmail.com Mon Oct 16 14:41:55 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 16 Oct 2006 11:41:55 -0700 Subject: [Backgroundrb-devel] Start script is perfectly fine or How to look like an idiot In-Reply-To: References: Message-ID: <5F270FA1-0EC7-41F2-9A68-961D2589211C@gmail.com> On Oct 16, 2006, at 8:32 AM, Christian Romney wrote: > Ok let this be a lesson to the general public. If you want to look > as idiotic as I do right now do the following: > Stay up all night and attempt to debug code the next day without > coffee > Patch the wrong file RAILS_ROOT/scripts/backgroundrb/start instead > of one in plugin dir. > Create a patch in the right location containing nothing but an > extra space > Send it to the whole Ruby community AND file a bug. > Only afterward, look at any of the damn files to discover that > you've made an ass of yourself. > > Got it? Good! > > Apologies to everyone. Apparently when I updated the plugin and re- > ran rake backgroundrb:setup it > did not overwrite my old version of the start script with the new, > correct one already in trunk. > > -- > Best Regards, > Christian Romney > \ Haha- no worries Chris. I still appreciate the willingness to make a patch and send it out. bravo ;) -- Ezra Zygmuntowicz -- Lead Rails Architect -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- Reliability, Ease of Use, Scalability -- (866) 518-YARD (9273) From bill.walton at charter.net Mon Oct 16 17:55:39 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 16 Oct 2006 16:55:39 -0500 Subject: [Backgroundrb-devel] accessing session data in worker Message-ID: <020901c6f16d$d215a140$6401a8c0@Presario> Is there something special I need to do to access session data in a worker? I'm using an AR session store. I've been working with Ezra's tutorial, modifying it a little here and there to figure it out. So I changed the progress bar to a simple count-down in the worker which sends back the count to display in the view. No big deal. But if I try to access session data in the worker method, I get no count displayed in the view. Any ideas? Thanks, Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/f5a1aa1e/attachment.html From michael.dauria at gmail.com Mon Oct 16 18:21:21 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Mon, 16 Oct 2006 18:21:21 -0400 Subject: [Backgroundrb-devel] accessing session data in worker In-Reply-To: <020901c6f16d$d215a140$6401a8c0@Presario> References: <020901c6f16d$d215a140$6401a8c0@Presario> Message-ID: <1907e2ca0610161521w1f1da92fr1c31422fee980d0c@mail.gmail.com> Do you have a code snippet to show? The Session model should be availible if you are loading rails. On 10/16/06, Bill Walton wrote: > > Is there something special I need to do to access session data in a > worker? > > I'm using an AR session store. I've been working with Ezra's tutorial, > modifying it a little here and there to figure it out. So I changed the > progress bar to a simple count-down in the worker which sends back the count > to display in the view. No big deal. But if I try to access session data > in the worker method, I get no count displayed in the view. > > Any ideas? > > Thanks, > Bill > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061016/7a150482/attachment.html From ezmobius at gmail.com Mon Oct 16 20:22:53 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 16 Oct 2006 17:22:53 -0700 Subject: [Backgroundrb-devel] accessing session data in worker In-Reply-To: <020901c6f16d$d215a140$6401a8c0@Presario> References: <020901c6f16d$d215a140$6401a8c0@Presario> Message-ID: <826FA8A2-34FD-4DB1-AC75-A9D0FBAE2E94@brainspl.at> On Oct 16, 2006, at 2:55 PM, Bill Walton wrote: > Is there something special I need to do to access session data in a > worker? > > I'm using an AR session store. I've been working with Ezra's > tutorial, modifying it a little here and there to figure it out. > So I changed the progress bar to a simple count-down in the worker > which sends back the count to display in the view. No big deal. > But if I try to access session data in the worker method, I get no > count displayed in the view. > > Any ideas? > > Thanks, > Bill > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel The session is not available in the workers. If you need a value from the session then you must pass it into the new_worker call ass :args Cheers- -- Ezra Zygmuntowicz -- Lead Rails Architect -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- Reliability, Ease of Use, Scalability -- (866) 518-YARD (9273) From bill.walton at charter.net Tue Oct 17 09:46:41 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 08:46:41 -0500 Subject: [Backgroundrb-devel] accessing session data in worker References: <020901c6f16d$d215a140$6401a8c0@Presario> <826FA8A2-34FD-4DB1-AC75-A9D0FBAE2E94@brainspl.at> Message-ID: <00c701c6f1f2$b2b00ea0$6401a8c0@Presario> Hi Ezra, Ezra Zygmuntowicz wrote: > The session is not available in the workers. Thanks for clearing that up for me! So the data 'inside' the session record is not available, but the session record IS? I'm using BackgroundRB to do session cleanup for abandoned sessions. I'm thinking that I can pass the session id to the worker and then, inside the worker, retrieve and delete that record with something like session_record = Session.find(:first, :conditions => ["sessid = ?", passed_in_session_id]) session_record.destroy Is that right? Or do I need to do something special to access that record? Thanks, Bill From michael.dauria at gmail.com Tue Oct 17 10:42:05 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Tue, 17 Oct 2006 10:42:05 -0400 Subject: [Backgroundrb-devel] accessing session data in worker In-Reply-To: <00c701c6f1f2$b2b00ea0$6401a8c0@Presario> References: <020901c6f16d$d215a140$6401a8c0@Presario> <826FA8A2-34FD-4DB1-AC75-A9D0FBAE2E94@brainspl.at> <00c701c6f1f2$b2b00ea0$6401a8c0@Presario> Message-ID: <1907e2ca0610170742r535633a5h3993953de1f73271@mail.gmail.com> I was accessing the Session model directly from within a worker and it was working: class ClearSessionsWorker < BackgrounDRb::Rails repeat_every 7.minutes first_run Time.now def do_work(args) Session.destroy_all(["updated_at < ?", 20.minute.ago ]) ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key end end I just wasn't cleaning the workers up properly, but it was cleaning out sessions as it should... On 10/17/06, Bill Walton wrote: > > Hi Ezra, > > Ezra Zygmuntowicz wrote: > > > The session is not available in the workers. > > Thanks for clearing that up for me! So the data 'inside' the session > record > is not available, but the session record IS? I'm using BackgroundRB to do > session cleanup for abandoned sessions. I'm thinking that I can pass the > session id to the worker and then, inside the worker, retrieve and delete > that record with something like > > session_record = Session.find(:first, :conditions => ["sessid = ?", > passed_in_session_id]) > session_record.destroy > > Is that right? Or do I need to do something special to access that > record? > > Thanks, > Bill > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061017/d7e9d4a2/attachment.html From bill.walton at charter.net Tue Oct 17 12:26:39 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 11:26:39 -0500 Subject: [Backgroundrb-devel] accessing session data in worker References: <020901c6f16d$d215a140$6401a8c0@Presario> <826FA8A2-34FD-4DB1-AC75-A9D0FBAE2E94@brainspl.at> <00c701c6f1f2$b2b00ea0$6401a8c0@Presario> <1907e2ca0610170742r535633a5h3993953de1f73271@mail.gmail.com> Message-ID: <00e801c6f209$07e678d0$6401a8c0@Presario> Hi Michael, Michael D'Auria wrote: > I was accessing the Session model directly > from within a worker and it was working: I don't seem to be able to access my models at all. I've got two AR models in this little sandbox app: Session and Emrec. I thought I'd start with the Emrec model (haven't tried to access the Session model yet). In the controller, before starting up the worker, I create an Emrec. In the worker, I want to delete it. I was trying to pass in the id of the record I want but couldn't get that to work. I figured it was a problem with how I was trying to access the record id, so I tried to hardcode the delete. My worker looks like... ----------- worker --------- class FooWorker < BackgrounDRb::Rails attr_reader :time_remaining def do_work(args) @time_remaining = 7 calculate_the_meaning_of_life(args) end def calculate_the_meaning_of_life(args) while @time_remaining > 0 # calculations here @time_remaining -= 1 sleep(1) end Emrec.delete(1) end end --------- end of worker --------- The emrec is not getting deleted. Any idea what I'm doing wrong here? Also, assuming I get the hardcoded version working, what's the syntax for accessing the values passed in the args hash? Thanks, Bill From ezmobius at gmail.com Tue Oct 17 13:12:15 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 17 Oct 2006 10:12:15 -0700 Subject: [Backgroundrb-devel] accessing session data in worker In-Reply-To: <1907e2ca0610170742r535633a5h3993953de1f73271@mail.gmail.com> References: <020901c6f16d$d215a140$6401a8c0@Presario> <826FA8A2-34FD-4DB1-AC75-A9D0FBAE2E94@brainspl.at> <00c701c6f1f2$b2b00ea0$6401a8c0@Presario> <1907e2ca0610170742r535633a5h3993953de1f73271@mail.gmail.com> Message-ID: <952F7351-B91D-49C6-B56E-E2068633FC82@brainspl.at> Ahh ok I misunderstood you then. I thought you were trying to use session[:foo] from within a worker my bad. -Ezra On Oct 17, 2006, at 7:42 AM, Michael D'Auria wrote: > I was accessing the Session model directly from within a worker and > it was working: > > class ClearSessionsWorker < BackgrounDRb::Rails > > repeat_every 7.minutes > first_run Time.now > > def do_work(args) > Session.destroy_all(["updated_at < ?", 20.minute.ago ]) > > ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key > end > end > > I just wasn't cleaning the workers up properly, but it was cleaning > out sessions as it should... > > > On 10/17/06, Bill Walton wrote: Hi Ezra, > > Ezra Zygmuntowicz wrote: > > > The session is not available in the workers. > > Thanks for clearing that up for me! So the data 'inside' the > session record > is not available, but the session record IS? I'm using > BackgroundRB to do > session cleanup for abandoned sessions. I'm thinking that I can > pass the > session id to the worker and then, inside the worker, retrieve and > delete > that record with something like > > session_record = Session.find(:first, :conditions => ["sessid = ?", > passed_in_session_id]) > session_record.destroy > > Is that right? Or do I need to do something special to access that > record? > > Thanks, > Bill > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From bill.walton at charter.net Tue Oct 17 14:15:21 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 13:15:21 -0500 Subject: [Backgroundrb-devel] accessing args in worker Message-ID: <012b01c6f218$35823360$6401a8c0@Presario> If, in the controller, I start a worker and pass in an argument like this: session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => {:emrec_id => @emrec.id}) How do I access the value in the worker? I can't seem to figure it out ;-p Thanks! Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061017/48549ab5/attachment.html From bill.walton at charter.net Tue Oct 17 14:44:51 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 13:44:51 -0500 Subject: [Backgroundrb-devel] accessing args in worker References: <012b01c6f218$35823360$6401a8c0@Presario> Message-ID: <013401c6f21c$55b3f160$6401a8c0@Presario> Thank you, Rob! I really appreciate it. Best regards, Bill ----- Original Message ----- From: Rob Biedenharn To: Bill Walton Sent: Tuesday, October 17, 2006 1:38 PM Subject: Re: [Backgroundrb-devel] accessing args in worker On Oct 17, 2006, at 2:15 PM, Bill Walton wrote: If, in the controller, I start a worker and pass in an argument like this: session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => {:emrec_id => @emrec.id}) How do I access the value in the worker? I can't seem to figure it out ;-p Thanks! Bill def do_work(args) @emred_id = args[:emrec_id] # ... do more stuff ... end -Rob Rob Biedenharn http://agileconsultingllc.com Rob at AgileConsultingLLC.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061017/843a0bcd/attachment.html From ezmobius at gmail.com Tue Oct 17 14:51:45 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 17 Oct 2006 11:51:45 -0700 Subject: [Backgroundrb-devel] accessing args in worker In-Reply-To: <012b01c6f218$35823360$6401a8c0@Presario> References: <012b01c6f218$35823360$6401a8c0@Presario> Message-ID: <016A0B69-8AAA-415F-B2DE-48DD33DB2670@brainspl.at> On Oct 17, 2006, at 11:15 AM, Bill Walton wrote: > If, in the controller, I start a worker and pass in an argument > like this: > > session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, > :args => {:emrec_id => > @emrec.id}) > > How do I access the value in the worker? I can't seem to figure it > out ;-p > > Thanks! > Bill Hey Bill- You will be able to access that hash in your do_work method as args [:emrec_id] Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From bill.walton at charter.net Tue Oct 17 16:21:34 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 15:21:34 -0500 Subject: [Backgroundrb-devel] Session access interfers with other model access Message-ID: <01a701c6f229$db1cb370$6401a8c0@Presario> Sorry to be such a bother but I'm not getting this. I have two models: Emrec and Session (I'm using AR for session mgmt.) In my worker I can access the Emrec model and delete a record, AS LONG AS I don't try to access the Session model. With the Session model access commented out as below, the Emrec record gets deleted. If I uncomment those lines, the Emrec record is _not_ deleted. Any idea what I'm doing wrong? Thanks, Bill In my controller... @session_id = session.session_id session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => {:emrec_id => @emrec.id, :session_id => @session_id}) In my worker... def do_work(args) @time_remaining = 7 @emrec_id = args[:emrec_id] #@session_rec = Session.find(:first, # :conditions => ["sessid = ?", args[:session_id]]) calculate_the_meaning_of_life(args) Emrec.delete(@emrec_id) end From bill.walton at charter.net Tue Oct 17 16:52:20 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 17 Oct 2006 15:52:20 -0500 Subject: [Backgroundrb-devel] Session access interfers with other modelaccess References: <01a701c6f229$db1cb370$6401a8c0@Presario> Message-ID: <01dc01c6f22e$266d42f0$6401a8c0@Presario> Update... I explicitly generated a Session model and that made things a little better. I can uncomment the lines below and now the Emrec DOES get deleted. However, if I try to also delete the Session record, neither gets deleted. Any ideas what gives? Thanks, Bill ----- Original Message ----- From: "Bill Walton" To: "BackgroundRb" Sent: Tuesday, October 17, 2006 3:21 PM Subject: [Backgroundrb-devel] Session access interfers with other modelaccess > Sorry to be such a bother but I'm not getting this. > > I have two models: Emrec and Session (I'm using AR for session mgmt.) In > my > worker I can access the Emrec model and delete a record, AS LONG AS I > don't > try to access the Session model. With the Session model access commented > out as below, the Emrec record gets deleted. If I uncomment those lines, > the Emrec record is _not_ deleted. Any idea what I'm doing wrong? > > Thanks, > Bill > > > In my controller... > > @session_id = session.session_id > session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, > :args => {:emrec_id => > @emrec.id, > :session_id => > @session_id}) > > > In my worker... > > def do_work(args) > @time_remaining = 7 > @emrec_id = args[:emrec_id] > > #@session_rec = Session.find(:first, > # :conditions => ["sessid = ?", > args[:session_id]]) > > calculate_the_meaning_of_life(args) > Emrec.delete(@emrec_id) > end > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From michael.dauria at gmail.com Tue Oct 17 17:13:29 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Tue, 17 Oct 2006 17:13:29 -0400 Subject: [Backgroundrb-devel] Session access interfers with other modelaccess In-Reply-To: <01dc01c6f22e$266d42f0$6401a8c0@Presario> References: <01a701c6f229$db1cb370$6401a8c0@Presario> <01dc01c6f22e$266d42f0$6401a8c0@Presario> Message-ID: <1907e2ca0610171413r2044e43bhdfba366a122f3fa5@mail.gmail.com> Are there any logs we can look at? In dev mode we can see MySQL logs On 10/17/06, Bill Walton wrote: > > Update... > > I explicitly generated a Session model and that made things a little > better. > I can uncomment the lines below and now the Emrec DOES get deleted. > However, if I try to also delete the Session record, neither gets deleted. > Any ideas what gives? > > Thanks, > Bill > > ----- Original Message ----- > From: "Bill Walton" > To: "BackgroundRb" > Sent: Tuesday, October 17, 2006 3:21 PM > Subject: [Backgroundrb-devel] Session access interfers with other > modelaccess > > > > Sorry to be such a bother but I'm not getting this. > > > > I have two models: Emrec and Session (I'm using AR for session > mgmt.) In > > my > > worker I can access the Emrec model and delete a record, AS LONG AS I > > don't > > try to access the Session model. With the Session model access > commented > > out as below, the Emrec record gets deleted. If I uncomment those > lines, > > the Emrec record is _not_ deleted. Any idea what I'm doing wrong? > > > > Thanks, > > Bill > > > > > > In my controller... > > > > @session_id = session.session_id > > session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, > > :args => {:emrec_id => > > @emrec.id, > > :session_id => > > @session_id}) > > > > > > In my worker... > > > > def do_work(args) > > @time_remaining = 7 > > @emrec_id = args[:emrec_id] > > > > #@session_rec = Session.find(:first, > > # :conditions => ["sessid = ?", > > args[:session_id]]) > > > > calculate_the_meaning_of_life(args) > > Emrec.delete(@emrec_id) > > end > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061017/9ab43663/attachment.html From twr at frii.com Wed Oct 18 01:43:26 2006 From: twr at frii.com (todd Richmond) Date: Tue, 17 Oct 2006 23:43:26 -0600 Subject: [Backgroundrb-devel] Please help with 2 (related?) problems Message-ID: <1161150206.9024.21.camel@localhost.localdomain> Hi All, I was able to get the BackgrounDrb plugin installed, and even got the example progress bar working. So far so good. But now that I'm trying to do something a little more real, I'm getting stuck. The first problem I'm getting is that the background task seems to be getting terminated by Rails. In any case, I am getting the following error as the first thing that happens in my background logger: Connection reset by peer - (DRb::DRbConnError) What might be the common causes of this error? The second problem that I'm having, which I suspect may be related, is that my view tries to call the get_progress action, but doesn't get a response. In the view file I have: <%= periodically_call_remote(:url => {:controller => 'parser', :action => 'get_progress'}, :frequency => 1) %> And in the parser controller, I have (basically verbatim from the example): def get_progress if request.xhr? progress_percent = MiddleMan.get_worker(session[:job_key]).progress render :update do |page| page.call('progressPercent', 'progressbar', progress_percent) page.redirect_to( :action => 'done') if progress_percent >= 100 end else redirect_to :action => 'index' end end When I run the controller, I get the following error in the log file: Processing ParserController#get_progress (for 127.0.0.1 at 2006-10-17 23:23:23) [POST] Session ID: 7425fe7cfe8db2c0211d5046b5b8e166 Parameters: {"action"=>"get_progress", "controller"=>"parser"} ActionController::UnknownAction (No action responded to get_progress): Anyone have any ideas what might be the problem? I'm fairly new to Ruby, Rails, and BackgrounDrb, so please be gentle. ;) Thanks a bunch, Todd Richmond -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061017/ea83e59b/attachment.html From bill.walton at charter.net Wed Oct 18 15:03:22 2006 From: bill.walton at charter.net (Bill Walton) Date: Wed, 18 Oct 2006 14:03:22 -0500 Subject: [Backgroundrb-devel] Session access interfers with other modelaccess References: <01a701c6f229$db1cb370$6401a8c0@Presario> <01dc01c6f22e$266d42f0$6401a8c0@Presario> <1907e2ca0610171413r2044e43bhdfba366a122f3fa5@mail.gmail.com> Message-ID: <000e01c6f2e8$1812fd70$6401a8c0@Presario> Sorry for the fat-fingered send earlier. Further investigation reveals concrete evidence. Just not sure what it means. Sure would appreciate any insight / suggestions. Code below. I found that the backgroundrb.log was telling me it was encountering an error in the worker, trying to do a delete on a nil id. What I've found out is this. If I start a new browser (i.e., a new session) and point it to the app, the find in the worker fails and so the delete of the session record fails. Problem it, I can see the record (via MySQL-Front) in the table prior to the worker attempt to find it. Why does the find fail? If I restart the app (i.e., still using the same session id), the find in the worker SUCCEEDS, the session record is deleted, and a new (empty) session is created. Why does the find succeed this time? Is there something different about the session record at this point (like something I could _make_ happen)? There's no _visible_ difference. Any ideas at all what might be going on here? I'd be *real* happy to send along the view/layout and model if anybody's willing to give it a quick look. TIA, Bill --------- Controller ------------ class CreateController < ApplicationController def index @emrec = Emrec.new @emrec.last_updated_at = Time.now @emrec.save @session_id = session.session_id session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => {:emrec_id => @emrec.id, :session_id => @session_id}) @session_rec_id = @session_id end def get_time_remaining if request.xhr? progress_percent = MiddleMan.get_worker(session[:job_key]).time_remaining render :update do |page| page.call('progressPercent', 'clock', progress_percent) # commented-out to eliminate race condition on WinXP from debugging # page.redirect_to(:action => 'done') if progress_percent <= 0 end else redirect_to :action => 'not_xhr' end end def done render :text => "Your FooWorker task has completed" MiddleMan.delete_worker(session[:job_key]) end end ----------- end Controller ------------ ---------- worker ------------- class FooWorker < BackgrounDRb::Rails attr_reader :time_remaining def do_work(args) @time_remaining = 15 @emrec_id = args[:emrec_id] @session_id = args[:session_id] @session_rec = Session.find(:first, :conditions => ["sessid = ?", @session_id]) calculate_the_meaning_of_life(args) if !@session_rec.nil? Session.delete(@session_rec.id) end Emrec.delete(@emrec_id) end def calculate_the_meaning_of_life(args) while @time_remaining > 0 @time_remaining -= 1 sleep(1) end end end ----- end worker ------ From cameron at theworkinggroup.ca Wed Oct 18 17:29:12 2006 From: cameron at theworkinggroup.ca (Cameron Booth) Date: Wed, 18 Oct 2006 16:29:12 -0500 Subject: [Backgroundrb-devel] Memory staying high even after calling MiddleMan.gc!(Time.now) Message-ID: <8BAC1630-91B1-4472-A10B-F4A4B14C3819@theworkinggroup.ca> Hi there, I'm just implementing backgroundrb to see if I can offload some "processor intensive" areas of my rails app (generating PDFs with lots of imagery in them, using rFPDF and rMagick as well). So far I've been able to figure it out fairly quickly....it's a pretty cool tool! Some questions though: First: I'm running the basic cron-job script to call GC every few minutes via cron, and generally while memory usage shoots up when I'm processing, shortly after it drops back down (base level is about 20Mb). However, in some cases during my testing (trying to hit it many times, etc), it seems like it gets stuck at higher memory usage (like 120Mb!!). The cron job seems to do nothing to help Second: What are people's experiences running 2 instances on the same box? I have a development site, and the live site running on the same box. Do I have to have them run drb on separate ports (by editing the port # on deploy via Capistrano?), or can it run in both places on the same port, but with a different environment? Thanks in advance!! Cameron From ezmobius at gmail.com Wed Oct 18 20:15:05 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Wed, 18 Oct 2006 17:15:05 -0700 Subject: [Backgroundrb-devel] Memory staying high even after calling MiddleMan.gc!(Time.now) In-Reply-To: <8BAC1630-91B1-4472-A10B-F4A4B14C3819@theworkinggroup.ca> References: <8BAC1630-91B1-4472-A10B-F4A4B14C3819@theworkinggroup.ca> Message-ID: Hi~ On Oct 18, 2006, at 2:29 PM, Cameron Booth wrote: > Hi there, > > I'm just implementing backgroundrb to see if I can offload some > "processor intensive" areas of my rails app (generating PDFs with > lots of imagery in them, using rFPDF and rMagick as well). > > So far I've been able to figure it out fairly quickly....it's a > pretty cool tool! > > Some questions though: > > First: > > I'm running the basic cron-job script to call GC every few minutes > via cron, and generally while memory usage shoots up when I'm > processing, shortly after it drops back down (base level is about > 20Mb). However, in some cases during my testing (trying to hit it > many times, etc), it seems like it gets stuck at higher memory usage > (like 120Mb!!). The cron job seems to do nothing to help Without seeing what you are doing in the worker its hard to say what the problem is. My drb servers stay pretty stable/ But the new release will fix a bunch of things and make the entire system much more robust. skaar has stepped up to the plate and is working hard on the new release so we hope to have it out soon. > > Second: > > What are people's experiences running 2 instances on the same box? I > have a development site, and the live site running on the same box. > Do I have to have them run drb on separate ports (by editing the port > # on deploy via Capistrano?), or can it run in both places on the > same port, but with a different environment?\ You will need a separate port for each drb server and environment. So one for dev and one for prod on diff ports. > > > Thanks in advance!! > > Cameron Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From bill.walton at charter.net Thu Oct 19 09:19:37 2006 From: bill.walton at charter.net (Bill Walton) Date: Thu, 19 Oct 2006 08:19:37 -0500 Subject: [Backgroundrb-devel] RESOLVED: Session access interfers with other model access References: <01a701c6f229$db1cb370$6401a8c0@Presario><01dc01c6f22e$266d42f0$6401a8c0@Presario><1907e2ca0610171413r2044e43bhdfba366a122f3fa5@mail.gmail.com> <000e01c6f2e8$1812fd70$6401a8c0@Presario> Message-ID: <015b01c6f381$3f18d2c0$6401a8c0@Presario> Bill Walton wrote: > > If I start a new browser (i.e., a new session) and > point it to the app, the find in the worker fails and > so the delete of the session record fails. Problem is, > I can see the record (via MySQL-Front) in the table > prior to the worker attempt to find it. Why does the > find fail? I still do not understand why I can see the record in the table via MySQL-Front but Rails / BackgrounDRb cannot find it, but I figured out how to fix the problem. Doing a session.update in the controller prior to the Middleman call somehow puts the record in a state where the find in the worker on the sessid field succeeds. Hope this helps someone in the future. If anybody can explain what's going on with the session I'd be very interested to learn. Best regards, Bill From ezmobius at gmail.com Thu Oct 19 11:51:54 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Thu, 19 Oct 2006 08:51:54 -0700 Subject: [Backgroundrb-devel] RESOLVED: Session access interfers with other model access In-Reply-To: <015b01c6f381$3f18d2c0$6401a8c0@Presario> References: <01a701c6f229$db1cb370$6401a8c0@Presario><01dc01c6f22e$266d42f0$6401a8c0@Presario><1907e2ca0610171413r2044e43bhdfba366a122f3fa5@mail.gmail.com> <000e01c6f2e8$1812fd70$6401a8c0@Presario> <015b01c6f381$3f18d2c0$6401a8c0@Presario> Message-ID: On Oct 19, 2006, at 6:19 AM, Bill Walton wrote: > > Bill Walton wrote: >> >> If I start a new browser (i.e., a new session) and >> point it to the app, the find in the worker fails and >> so the delete of the session record fails. Problem is, >> I can see the record (via MySQL-Front) in the table >> prior to the worker attempt to find it. Why does the >> find fail? > > I still do not understand why I can see the record in the table via > MySQL-Front but Rails / BackgrounDRb cannot find it, but I figured > out how > to fix the problem. > > Doing a session.update in the controller prior to the Middleman > call somehow > puts the record in a state where the find in the worker on the > sessid field > succeeds. > > Hope this helps someone in the future. If anybody can explain > what's going > on with the session I'd be very interested to learn. > > Best regards, > Bill Ahh good catch Bill. That makes total sense though. When you call the middleman from a rails action without doing the session.update, the session record is still in memory and not written to the db yet. update makes it write out the session into the db. Glad you got it working. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From bill.walton at charter.net Thu Oct 19 13:09:02 2006 From: bill.walton at charter.net (Bill Walton) Date: Thu, 19 Oct 2006 12:09:02 -0500 Subject: [Backgroundrb-devel] RESOLVED: Session access interfers with other model access References: <01a701c6f229$db1cb370$6401a8c0@Presario><01dc01c6f22e$266d42f0$6401a8c0@Presario><1907e2ca0610171413r2044e43bhdfba366a122f3fa5@mail.gmail.com> <000e01c6f2e8$1812fd70$6401a8c0@Presario> <015b01c6f381$3f18d2c0$6401a8c0@Presario> Message-ID: <01bd01c6f3a1$483fc1e0$6401a8c0@Presario> Hi Ezra, Ezra Zygmuntowicz wrote: > When you call the middleman from a rails action without doing the > session.update, the session record is still in memory and not written to > the db yet. update makes it write out the session into the db. That's not what it looks like. What I mean is that I can _see_ the record in the database via MySQL-Front before the find is attempted. Unless I very much misunderstand the way MySQL-Front works, the record has, in fact, already been written to the db. Rails (or BackgrounDRb) just can't retrieve it for some reason. > Glad you got it working. Thanks! I just wish I understood _why_ it's working. Best regards, Bill From bill.walton at charter.net Mon Oct 23 15:24:17 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 23 Oct 2006 14:24:17 -0500 Subject: [Backgroundrb-devel] can a worker commit suicide? Message-ID: <067e01c6f6d8$d7332560$6401a8c0@Presario> Can a worker kill themselves when they're 'done'? Or do I have to do that either from the controller or the worker manager? Thanks, Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061023/11dd429c/attachment.html From eden.li at gmail.com Mon Oct 23 15:48:18 2006 From: eden.li at gmail.com (Eden Li) Date: Tue, 24 Oct 2006 03:48:18 +0800 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: <067e01c6f6d8$d7332560$6401a8c0@Presario> References: <067e01c6f6d8$d7332560$6401a8c0@Presario> Message-ID: <565dbdd40610231248k5c451496x4d0ca782db4a96ac@mail.gmail.com> You can specify a time to live in the args to MiddleMan.new_worker. >From the README http://backgroundrb.rubyforge.org/: ... you can specify a :ttl(time to live) in seconds and your worker will get killed after those seconds run out. The following will start a FooWorker class with a text argument of "Bar" and a time to live of 5 minutes session[:job_key] = MiddleMan.new_worker(:class => :foo_worker, :args => "Bar", :ttl => 300) On 10/24/06, Bill Walton wrote: > Can a worker kill themselves when they're 'done'? Or do I have to do that > either from the controller or the worker manager? From bill.walton at charter.net Mon Oct 23 16:19:47 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 23 Oct 2006 15:19:47 -0500 Subject: [Backgroundrb-devel] can a worker commit suicide? References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> Message-ID: <06a901c6f6e0$cc1ae660$6401a8c0@Presario> Hi Eden, Eden Li wrote: > ... you can specify a :ttl(time to live) in seconds and your worker will > get killed after those seconds run out. I'm using BackgrounDRb to monitor session activity and deleting user entered data in the database when their session times out. Every time they do something, their time-til-timeout gets reset. There's no notion of a max time. As long as they keep using the app, their session stays alive. I was wondering if there's a way, inside the worker, to do the equivalent of: MiddleMan.delete_worker(session[:job_key]) Thanks, Bill From michael.dauria at gmail.com Mon Oct 23 16:28:54 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Mon, 23 Oct 2006 16:28:54 -0400 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: <06a901c6f6e0$cc1ae660$6401a8c0@Presario> References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> Message-ID: <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> You can have the worker delete itself, just make sure you call ActiveRecord::Base.connection.disconect! first: ActiveRecord::Base.connection.disconect! ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key That shoudl ensure the db connection has been closed and delete the instance of the current worker On 10/23/06, Bill Walton wrote: > > Hi Eden, > > Eden Li wrote: > > > ... you can specify a :ttl(time to live) in seconds and your worker will > > get killed after those seconds run out. > > I'm using BackgrounDRb to monitor session activity and deleting user > entered > data in the database when their session times out. Every time they do > something, their time-til-timeout gets reset. There's no notion of a max > time. As long as they keep using the app, their session stays alive. > > I was wondering if there's a way, inside the worker, to do the equivalent > of: > MiddleMan.delete_worker(session[:job_key]) > > Thanks, > Bill > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061023/4d850c67/attachment.html From bill.walton at charter.net Mon Oct 23 18:20:16 2006 From: bill.walton at charter.net (Bill Walton) Date: Mon, 23 Oct 2006 17:20:16 -0500 Subject: [Backgroundrb-devel] can a worker commit suicide? References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> Message-ID: <001001c6f6f1$6bb58fd0$6401a8c0@Presario> Thank you, Michael. I'm only disconnecting that particular worker, right? It looks like it, since I can restart the app (browse to it again) and everything looks to work ok. But I thought I'd ask. Also, is there any way to tell how many workers are running? I'm on Windows XP and all I can see with Task Mgr. is a ruby.exe process for BackgrounDRb. Thanks again, Bill ----- Original Message ----- From: Michael D'Auria To: Bill Walton Cc: Eden Li ; BackgroundRb Sent: Monday, October 23, 2006 3:28 PM Subject: Re: [Backgroundrb-devel] can a worker commit suicide? You can have the worker delete itself, just make sure you call ActiveRecord::Base.connection.disconect! first: ActiveRecord::Base.connection.disconect! ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key That shoudl ensure the db connection has been closed and delete the instance of the current worker On 10/23/06, Bill Walton wrote: Hi Eden, Eden Li wrote: > ... you can specify a :ttl(time to live) in seconds and your worker will > get killed after those seconds run out. I'm using BackgrounDRb to monitor session activity and deleting user entered data in the database when their session times out. Every time they do something, their time-til-timeout gets reset. There's no notion of a max time. As long as they keep using the app, their session stays alive. I was wondering if there's a way, inside the worker, to do the equivalent of: MiddleMan.delete_worker(session[:job_key]) Thanks, Bill _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061023/7fda71ff/attachment.html From eimorton at gmail.com Mon Oct 23 20:26:10 2006 From: eimorton at gmail.com (Erik Morton) Date: Mon, 23 Oct 2006 20:26:10 -0400 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: <001001c6f6f1$6bb58fd0$6401a8c0@Presario> References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> <001001c6f6f1$6bb58fd0$6401a8c0@Presario> Message-ID: <3D38D663-8953-4D5F-8B67-F9C65D966108@gmail.com> I recommend closing the connection on the ActiveRecord instance in the worker. In past versions of BackgroundRB those connections were never closed. To check on open connections execute show processlist; On Oct 23, 2006, at 6:20 PM, Bill Walton wrote: > Thank you, Michael. I'm only disconnecting that particular worker, > right? It looks like it, since I can restart the app (browse to it > again) and everything looks to work ok. But I thought I'd ask. > Also, is there any way to tell how many workers are running? I'm > on Windows XP and all I can see with Task Mgr. is a ruby.exe > process for BackgrounDRb. > > Thanks again, > Bill > ----- Original Message ----- > From: Michael D'Auria > To: Bill Walton > Cc: Eden Li ; BackgroundRb > Sent: Monday, October 23, 2006 3:28 PM > Subject: Re: [Backgroundrb-devel] can a worker commit suicide? > > You can have the worker delete itself, just make sure you call > ActiveRecord::Base.connection.disconect! first: > > ActiveRecord::Base.connection.disconect! > ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key > > That shoudl ensure the db connection has been closed and delete the > instance of the current worker > > > On 10/23/06, Bill Walton wrote: Hi Eden, > > Eden Li wrote: > > > ... you can specify a :ttl(time to live) in seconds and your > worker will > > get killed after those seconds run out. > > I'm using BackgrounDRb to monitor session activity and deleting > user entered > data in the database when their session times out. Every time they do > something, their time-til-timeout gets reset. There's no notion of > a max > time. As long as they keep using the app, their session stays alive. > > I was wondering if there's a way, inside the worker, to do the > equivalent > of: > MiddleMan.delete_worker(session[:job_key]) > > Thanks, > Bill > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From ezmobius at gmail.com Mon Oct 23 23:10:28 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 23 Oct 2006 20:10:28 -0700 Subject: [Backgroundrb-devel] Status Update Message-ID: Heya folks- I just wanted to give a little status update on the new version of BackgrounDRb. Skaar has stepped up big time and done a ton of work getting the new architecture going so big thanks to him. I will be doing some documentation and a few more tweaks and we should have a new release shortly. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From michael.dauria at gmail.com Mon Oct 23 23:45:54 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Mon, 23 Oct 2006 23:45:54 -0400 Subject: [Backgroundrb-devel] Status Update In-Reply-To: References: Message-ID: <1907e2ca0610232045ma2cd7dla653afb8e3345017@mail.gmail.com> suhweet! :D On 10/23/06, Ezra Zygmuntowicz wrote: > > Heya folks- > > > I just wanted to give a little status update on the new version of > BackgrounDRb. Skaar has stepped up big time and done a ton of work > getting the new architecture going so big thanks to him. I will be > doing some documentation and a few more tweaks and we should have a > new release shortly. > > Cheers- > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061023/b1dad5ec/attachment.html From bill.walton at charter.net Tue Oct 24 08:58:48 2006 From: bill.walton at charter.net (Bill Walton) Date: Tue, 24 Oct 2006 07:58:48 -0500 Subject: [Backgroundrb-devel] deployment issues? Message-ID: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> Thanks to many of you, I've got my session management working with BackGrounDRb. Very cool. So now I need to deploy it. I'm on a shared hosting plan (at a2hosting in case that matters). Are there any issues I should expect to have to deal with in getting them to deploy the plugin? Anything special I need to do? I'm wondering in particular about the 'start up' steps. Thanks! Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061024/39117c7f/attachment.html From michael.dauria at gmail.com Tue Oct 24 10:39:17 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Tue, 24 Oct 2006 10:39:17 -0400 Subject: [Backgroundrb-devel] deployment issues? In-Reply-To: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> References: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> Message-ID: <1907e2ca0610240739w19204755kfd71744ee0dd6c83@mail.gmail.com> I am pretty sure that when you deploy, you want to make sure that BackgrounDRb is running BEFORE your webserver/fcgi processes. On 10/24/06, Bill Walton wrote: > > Thanks to many of you, I've got my session management working with > BackGrounDRb. Very cool. So now I need to deploy it. I'm on a shared > hosting plan (at a2hosting in case that matters). Are there any issues I > should expect to have to deal with in getting them to deploy the plugin? > Anything special I need to do? I'm wondering in particular about the 'start > up' steps. > > Thanks! > Bill > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061024/a22625e6/attachment.html From ragarver at gmail.com Tue Oct 24 13:33:59 2006 From: ragarver at gmail.com (Ryan Garver) Date: Tue, 24 Oct 2006 10:33:59 -0700 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: <3D38D663-8953-4D5F-8B67-F9C65D966108@gmail.com> References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> <001001c6f6f1$6bb58fd0$6401a8c0@Presario> <3D38D663-8953-4D5F-8B67-F9C65D966108@gmail.com> Message-ID: <64D75629-E689-4E29-8D5C-B3DA55287516@gmail.com> It appears that this method of committing suicide, if called from within the do_work method appears to result in dead lock. Which makes sense since most of the operations on workers block on a mutex including delete_worker. I am curious about this because I am using backgroundrb for periodic scheduling where most of the time (in my case all of the time) they will not have an external monitor to clean them up. I'm hesitant to use ttls because some of the operations may take a long time. Any suggestions - Ryan On Oct 23, 2006, at 5:26 PM, Erik Morton wrote: > I recommend closing the connection on the ActiveRecord instance in > the worker. In past versions of BackgroundRB those connections were > never closed. To check on open connections execute show processlist; > > > On Oct 23, 2006, at 6:20 PM, Bill Walton wrote: > >> Thank you, Michael. I'm only disconnecting that particular worker, >> right? It looks like it, since I can restart the app (browse to it >> again) and everything looks to work ok. But I thought I'd ask. >> Also, is there any way to tell how many workers are running? I'm >> on Windows XP and all I can see with Task Mgr. is a ruby.exe >> process for BackgrounDRb. >> >> Thanks again, >> Bill >> ----- Original Message ----- >> From: Michael D'Auria >> To: Bill Walton >> Cc: Eden Li ; BackgroundRb >> Sent: Monday, October 23, 2006 3:28 PM >> Subject: Re: [Backgroundrb-devel] can a worker commit suicide? >> >> You can have the worker delete itself, just make sure you call >> ActiveRecord::Base.connection.disconect! first: >> >> ActiveRecord::Base.connection.disconect! >> ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key >> >> That shoudl ensure the db connection has been closed and delete the >> instance of the current worker >> >> >> On 10/23/06, Bill Walton wrote: Hi Eden, >> >> Eden Li wrote: >> >>> ... you can specify a :ttl(time to live) in seconds and your >> worker will >>> get killed after those seconds run out. >> >> I'm using BackgrounDRb to monitor session activity and deleting >> user entered >> data in the database when their session times out. Every time >> they do >> something, their time-til-timeout gets reset. There's no notion of >> a max >> time. As long as they keep using the app, their session stays alive. >> >> I was wondering if there's a way, inside the worker, to do the >> equivalent >> of: >> MiddleMan.delete_worker(session[:job_key]) >> >> Thanks, >> Bill >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From skaar at waste.org Tue Oct 24 16:00:00 2006 From: skaar at waste.org (skaar) Date: Tue, 24 Oct 2006 15:00:00 -0500 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: <64D75629-E689-4E29-8D5C-B3DA55287516@gmail.com> References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> <001001c6f6f1$6bb58fd0$6401a8c0@Presario> <3D38D663-8953-4D5F-8B67-F9C65D966108@gmail.com> <64D75629-E689-4E29-8D5C-B3DA55287516@gmail.com> Message-ID: <20061024195959.GB24734@waste.org> Not that it will help you immediately, but the new BackgrounDRb should not have this deadlock issue. Workers are separ?te processes and the do_work thread is local to this process. The process object itself (using the slave library) is available to the worker and through that you can call delete_worker on the MiddleMan, which has it's own thread pool. You don't have to do this yourself. The worker superclasses already have a delete method which does this: class SelfDestructingWorker < BackgrounDRb::Worker::Base def do_work(args) sleep 20 self.delete end end called through the MiddleMan, it looks something like: MiddleMan.new_worker :class => :self_destructing_worker, :job_key => :self_destruct Another behavior that should be noted, is that workers have a singleton behavior in that: 2000.times do MiddleMan.new_worker :class => :self_destructing_worker, :job_key => :self_destruct end would return the same worker reference, until delete is called, when a new worker would be created and the cycle started over again. /skaar * Ryan Garver (ragarver at gmail.com) [061024 13:00]: > It appears that this method of committing suicide, if called from > within the do_work method appears to result in dead lock. Which > makes sense since most of the operations on workers block on a mutex > including delete_worker. I am curious about this because I am using > backgroundrb for periodic scheduling where most of the time (in my > case all of the time) they will not have an external monitor to clean > them up. I'm hesitant to use ttls because some of the operations may > take a long time. Any suggestions > > - Ryan > > On Oct 23, 2006, at 5:26 PM, Erik Morton wrote: > > > I recommend closing the connection on the ActiveRecord instance in > > the worker. In past versions of BackgroundRB those connections were > > never closed. To check on open connections execute show processlist; > > > > > > On Oct 23, 2006, at 6:20 PM, Bill Walton wrote: > > > >> Thank you, Michael. I'm only disconnecting that particular worker, > >> right? It looks like it, since I can restart the app (browse to it > >> again) and everything looks to work ok. But I thought I'd ask. > >> Also, is there any way to tell how many workers are running? I'm > >> on Windows XP and all I can see with Task Mgr. is a ruby.exe > >> process for BackgrounDRb. > >> > >> Thanks again, > >> Bill > >> ----- Original Message ----- > >> From: Michael D'Auria > >> To: Bill Walton > >> Cc: Eden Li ; BackgroundRb > >> Sent: Monday, October 23, 2006 3:28 PM > >> Subject: Re: [Backgroundrb-devel] can a worker commit suicide? > >> > >> You can have the worker delete itself, just make sure you call > >> ActiveRecord::Base.connection.disconect! first: > >> > >> ActiveRecord::Base.connection.disconect! > >> ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key > >> > >> That shoudl ensure the db connection has been closed and delete the > >> instance of the current worker > >> > >> > >> On 10/23/06, Bill Walton wrote: Hi Eden, > >> > >> Eden Li wrote: > >> > >>> ... you can specify a :ttl(time to live) in seconds and your > >> worker will > >>> get killed after those seconds run out. > >> > >> I'm using BackgrounDRb to monitor session activity and deleting > >> user entered > >> data in the database when their session times out. Every time > >> they do > >> something, their time-til-timeout gets reset. There's no notion of > >> a max > >> time. As long as they keep using the app, their session stays alive. > >> > >> I was wondering if there's a way, inside the worker, to do the > >> equivalent > >> of: > >> MiddleMan.delete_worker(session[:job_key]) > >> > >> Thanks, > >> Bill > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > >> > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From renaud.morvan at feedback20.com Tue Oct 24 18:58:09 2006 From: renaud.morvan at feedback20.com (Renaud Morvan) Date: Wed, 25 Oct 2006 00:58:09 +0200 Subject: [Backgroundrb-devel] Broken thread Safe connection Management on Mysql (Mysql too many connections errors) Message-ID: <453E9A81.4060009@feedback20.com> Hi, There is an issue on socket connection with backgroundrb (rev 47) and mysql backend: after a certain number of job you reach the mysql connection limit as socket are never closed and nobody can connect to mysql anymore till you stop the backgroundrb daemon. Already reported on: http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000066.html, http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/000169.html It is quite easy to reproduce: - just have a simple worker with a simple command using AR (Model.find I.E.) - open a console and launch the work 100+ times, an exception will be raised 'too many connection'. The whole system won't be able to use mysql anymore as mysql socket are not closed In my humble opinion there are two reasons for it: - ActiveRecord::Base.allow_concurrency = true That will make every thread open it's own socket to mysql - Mysql adapter implementations (RAILS or gems) that close connection on Garbage connexion (using ObjectSpace.define_finalizer ) There is probably a memory leak on connections as the Proc finalizer is not be executed when worker thread is killed (= connections are not GC). In this case the only way to release the sockets is to close the backgroundrb daemon. Those two phenomenons lead to an accumulation of opened socket till you reach the system limit, and when you reach it, nothing no new connections can be done on mysql anymore on the whole system. (this is observable with a netstat |grep mysql) This can be workaround by setting ActiveRecord::Base.allow_concurrency to false (do not see the risks but there are some probably) or by manually closing the connections in worker before termination the tread (in this case you have to use job_ctrl to avoid thread to be killed before having closed the connections) Cronned restart of backgroundrb also fix this :) Anyway maybe the problem is in the design of Backgroundrb itself, that should use a pool of thread and not create as many thread as there are works, in this case the pool of thread will use a finite number of socket, whose connections will be reused. Otherwise it would be good to find the memory leak (= why killing thread don't make connections to be GC, or more precisely why ObjectSpace.define_finalizer does not work in this case) but it seems to be a bit tricky.( some rubyist seems to have the same kind of problem with thread and ObjectSpace.define_finalizer http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/291 or http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/145913) Hope this help, if someone has better solutions please talk :) Renaud Morvan http://exalead.feedback20.com From michael.dauria at gmail.com Tue Oct 24 22:07:54 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Tue, 24 Oct 2006 22:07:54 -0400 Subject: [Backgroundrb-devel] Broken thread Safe connection Management on Mysql (Mysql too many connections errors) In-Reply-To: <453E9A81.4060009@feedback20.com> References: <453E9A81.4060009@feedback20.com> Message-ID: <1907e2ca0610241907ldba9343oa8e8caa1e0aeeb@mail.gmail.com> If you call this within the worker: ActiveRecord::Base.connection.disconect! It will close it's connection to the DB. Just make sure you call it after it's done :) .: Michael :. On 10/24/06, Renaud Morvan wrote: > > Hi, > > There is an issue on socket connection with backgroundrb (rev 47) and > mysql backend: after a certain number of job you reach the mysql > connection limit as socket are never closed and nobody can connect to > mysql anymore till you stop the backgroundrb daemon. > > Already reported on: > http://rubyforge.org/pipermail/backgroundrb-devel/2006-July/000066.html, > http://rubyforge.org/pipermail/backgroundrb-devel/2006-August/000169.html > > It is quite easy to reproduce: > - just have a simple worker with a simple command using AR (Model.find I.E > .) > - open a console and launch the work 100+ times, an exception will be > raised 'too many connection'. The whole system won't be able to use > mysql anymore as mysql socket are not closed > > In my humble opinion there are two reasons for it: > - ActiveRecord::Base.allow_concurrency = true > That will make every thread open it's own socket to mysql > - Mysql adapter implementations (RAILS or gems) that close connection on > Garbage connexion (using ObjectSpace.define_finalizer ) > There is probably a memory leak on connections as the Proc finalizer is > not be executed when worker thread is killed (= connections are not > GC). In this case the only way to release the sockets is to close the > backgroundrb daemon. > > Those two phenomenons lead to an accumulation of opened socket till you > reach the system limit, and when you reach it, nothing no new > connections can be done on mysql anymore on the whole system. (this is > observable with a netstat |grep mysql) > > This can be workaround by setting ActiveRecord::Base.allow_concurrency > to false (do not see the risks but there are some probably) or by > manually closing the connections in worker before termination the tread > (in this case you have to use job_ctrl to avoid thread to be killed > before having closed the connections) > > Cronned restart of backgroundrb also fix this :) > > Anyway maybe the problem is in the design of Backgroundrb itself, that > should use a pool of thread and not create as many thread as there are > works, in this case the pool of thread will use a finite number of > socket, whose connections will be reused. > > Otherwise it would be good to find the memory leak (= why killing thread > don't make connections to be GC, or more precisely why > ObjectSpace.define_finalizer does not work in this case) but it seems to > be a bit tricky.( some rubyist seems to have the same kind of problem > with thread and ObjectSpace.define_finalizer > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/291 or > http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/145913) > > Hope this help, if someone has better solutions please talk :) > > Renaud Morvan > http://exalead.feedback20.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061024/3792aa5d/attachment-0001.html From ragarver at gmail.com Tue Oct 24 23:14:56 2006 From: ragarver at gmail.com (Ryan Garver) Date: Tue, 24 Oct 2006 20:14:56 -0700 Subject: [Backgroundrb-devel] can a worker commit suicide? In-Reply-To: References: <067e01c6f6d8$d7332560$6401a8c0@Presario> <565dbdd40610231246y68e52fdfu4732d8004f51592a@mail.gmail.com> <06a901c6f6e0$cc1ae660$6401a8c0@Presario> <1907e2ca0610231328p30cbf2d1l444b626057aa1abd@mail.gmail.com> <001001c6f6f1$6bb58fd0$6401a8c0@Presario> <3D38D663-8953-4D5F-8B67-F9C65D966108@gmail.com> <64D75629-E689-4E29-8D5C-B3DA55287516@gmail.com> Message-ID: <3305B2C6-32B7-4447-ABC1-0395DCB7B010@gmail.com> Erik, I'm running with svn:externals off of rubyforge. The disconnecting of the db actually works well. But as soon as I called the self.kill or terminate it froze. This may not be deadlock, but I was finding the server unreachable with very little information about why on stdout (this is _not_ in detached mode). As it turns out this may bee a moot issue since the worker is acting as a singleton (apparently) as a periodic job. This may be wrong, but thats what I've pieced together between the source and the mailing list. On Oct 24, 2006, at 5:50 PM, Erik Morton wrote: > In which version of BackgroundRB did you get this result? I threw > together some test code during the summer and had over 100 threads > going at once, which resulted in 100 connections. Each thread then > killed its connection and I didn't see any deadlock issues. > > Regards, > Erik > On Oct 24, 2006, at 1:33 PM, Ryan Garver wrote: > >> It appears that this method of committing suicide, if called from >> within the do_work method appears to result in dead lock. Which >> makes sense since most of the operations on workers block on a mutex >> including delete_worker. I am curious about this because I am using >> backgroundrb for periodic scheduling where most of the time (in my >> case all of the time) they will not have an external monitor to clean >> them up. I'm hesitant to use ttls because some of the operations may >> take a long time. Any suggestions >> >> - Ryan >> >> On Oct 23, 2006, at 5:26 PM, Erik Morton wrote: >> >>> I recommend closing the connection on the ActiveRecord instance in >>> the worker. In past versions of BackgroundRB those connections were >>> never closed. To check on open connections execute show processlist; >>> >>> >>> On Oct 23, 2006, at 6:20 PM, Bill Walton wrote: >>> >>>> Thank you, Michael. I'm only disconnecting that particular worker, >>>> right? It looks like it, since I can restart the app (browse to it >>>> again) and everything looks to work ok. But I thought I'd ask. >>>> Also, is there any way to tell how many workers are running? I'm >>>> on Windows XP and all I can see with Task Mgr. is a ruby.exe >>>> process for BackgrounDRb. >>>> >>>> Thanks again, >>>> Bill >>>> ----- Original Message ----- >>>> From: Michael D'Auria >>>> To: Bill Walton >>>> Cc: Eden Li ; BackgroundRb >>>> Sent: Monday, October 23, 2006 3:28 PM >>>> Subject: Re: [Backgroundrb-devel] can a worker commit suicide? >>>> >>>> You can have the worker delete itself, just make sure you call >>>> ActiveRecord::Base.connection.disconect! first: >>>> >>>> ActiveRecord::Base.connection.disconect! >>>> ::BackgrounDRb::MiddleMan.instance.delete_worker @_job_key >>>> >>>> That shoudl ensure the db connection has been closed and delete the >>>> instance of the current worker >>>> >>>> >>>> On 10/23/06, Bill Walton wrote: Hi Eden, >>>> >>>> Eden Li wrote: >>>> >>>>> ... you can specify a :ttl(time to live) in seconds and your >>>> worker will >>>>> get killed after those seconds run out. >>>> >>>> I'm using BackgrounDRb to monitor session activity and deleting >>>> user entered >>>> data in the database when their session times out. Every time >>>> they do >>>> something, their time-til-timeout gets reset. There's no notion of >>>> a max >>>> time. As long as they keep using the app, their session stays >>>> alive. >>>> >>>> I was wondering if there's a way, inside the worker, to do the >>>> equivalent >>>> of: >>>> MiddleMan.delete_worker(session[:job_key]) >>>> >>>> Thanks, >>>> Bill >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From bill.walton at charter.net Wed Oct 25 07:09:17 2006 From: bill.walton at charter.net (Bill Walton) Date: Wed, 25 Oct 2006 06:09:17 -0500 Subject: [Backgroundrb-devel] deployment issues? References: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> Message-ID: <025601c6f826$048182c0$6401a8c0@Presario> I sent the tech support folks at my host the following question: I need to get the BackgrounDRb plugin ( http://rubyforge.org/projects/backgroundrb/ ) installed for my app. Is that something I get you to do? Or is it something I need to do myself? To which they replied: If this is a simple gem install we can do that. Please provide the exact name of the gem used in the 'gem install' command. Thanks. How should I respond? Is there a difference between a gem and a plugin? Is this something I can / should do myself on a shared hosting platform? Thanks, Bill ----- Original Message ----- From: Bill Walton To: BackgroundRb Sent: Tuesday, October 24, 2006 7:58 AM Subject: [Backgroundrb-devel] deployment issues? Thanks to many of you, I've got my session management working with BackGrounDRb. Very cool. So now I need to deploy it. I'm on a shared hosting plan (at a2hosting in case that matters). Are there any issues I should expect to have to deal with in getting them to deploy the plugin? Anything special I need to do? I'm wondering in particular about the 'start up' steps. Thanks! Bill ------------------------------------------------------------------------------ _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061025/860800f6/attachment.html From gethemant at gmail.com Wed Oct 25 10:33:14 2006 From: gethemant at gmail.com (hemant) Date: Wed, 25 Oct 2006 20:03:14 +0530 Subject: [Backgroundrb-devel] Really long running worker threads and monitoring them Message-ID: Hi, I have been using Backgroundrb for quite sometime...and now...as i guess its a known problem that...worker threads throw up..."Mysql Connection has Gone away" error..when they run for really long time. Is there any conclusive workaround or solution. If not..and let's say...my worker threads are not critically important for my app....then which program you guys are using to restart the backgroundrb threads. Monit? anyone... -- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary. From michael.dauria at gmail.com Wed Oct 25 10:54:52 2006 From: michael.dauria at gmail.com (Michael D'Auria) Date: Wed, 25 Oct 2006 10:54:52 -0400 Subject: [Backgroundrb-devel] deployment issues? In-Reply-To: <025601c6f826$048182c0$6401a8c0@Presario> References: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> <025601c6f826$048182c0$6401a8c0@Presario> Message-ID: <1907e2ca0610250754ya457ffh555985c024b09712@mail.gmail.com> BackgrounDRb is a plugin, which means it is specific to your app whereas a gem is a globally available piece of software. The best gem example i can give is mongrel, you install it once and use it with all your apps. This means that you will have to install the plugin yourself, what is your dev environment? If you develop on your local machine, which you should be, then you can just issue "script/plugin install svn://rubyforge.org//var/svn/backgroundrb" All on one line of course. You can follow the rest of the installation directions here: http://backgroundrb.rubyforge.org/. Once it is installed locally and working locally, you can just simply upload your project to your hosting provider. .: Michael :. On 10/25/06, Bill Walton wrote: > > I sent the tech support folks at my host the following question: > > I need to get the BackgrounDRb plugin ( > http://rubyforge.org/projects/backgroundrb/ ) installed for my app. Is > that something I get you to do? Or is it something I need to do myself? > > To which they replied: > > If this is a simple gem install we can do that. Please provide the exact > name of the gem used in the 'gem install' command. Thanks. > > How should I respond? Is there a difference between a gem and a plugin? > Is this something I can / should do myself on a shared hosting platform? > > Thanks, > Bill > > ----- Original Message ----- > > *From:* Bill Walton > *To:* BackgroundRb > *Sent:* Tuesday, October 24, 2006 7:58 AM > *Subject:* [Backgroundrb-devel] deployment issues? > > > Thanks to many of you, I've got my session management working with > BackGrounDRb. Very cool. So now I need to deploy it. I'm on a shared > hosting plan (at a2hosting in case that matters). Are there any issues I > should expect to have to deal with in getting them to deploy the plugin? > Anything special I need to do? I'm wondering in particular about the 'start > up' steps. > > Thanks! > Bill > > ------------------------------ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061025/21097062/attachment.html From Rob at AgileConsultingLLC.com Wed Oct 25 15:23:11 2006 From: Rob at AgileConsultingLLC.com (Rob Biedenharn) Date: Wed, 25 Oct 2006 15:23:11 -0400 Subject: [Backgroundrb-devel] deployment issues? In-Reply-To: <1907e2ca0610250754ya457ffh555985c024b09712@mail.gmail.com> References: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> <025601c6f826$048182c0$6401a8c0@Presario> <1907e2ca0610250754ya457ffh555985c024b09712@mail.gmail.com> Message-ID: <50092774-57F9-4E4E-AE85-27AC7D49220C@AgileConsultingLLC.com> > ----- Original Message ----- > From:Bill Walton > To: BackgroundRb > Sent: Tuesday, October 24, 2006 7:58 AM > Subject: [Backgroundrb-devel] deployment issues? > > > Thanks to many of you, I've got my session management working with > BackGrounDRb. Very cool. So now I need to deploy it. I'm on a > shared hosting plan (at a2hosting in case that matters). Are there > any issues I should expect to have to deal with in getting them to > deploy the plugin? Anything special I need to do? I'm wondering > in particular about the 'start up' steps. > > Thanks! > Bill > On 10/25/06, Bill Walton wrote: > I sent the tech support folks at my host the following question: > > I need to get the BackgrounDRb plugin ( http://rubyforge.org/ > projects/backgroundrb/ ) installed for my app. Is that something I > get you to do? Or is it something I need to do myself? > > To which they replied: > > If this is a simple gem install we can do that. Please provide the > exact name of the gem used in the 'gem install' command. Thanks. > > How should I respond? Is there a difference between a gem and a > plugin? Is this something I can / should do myself on a shared > hosting platform? > > Thanks, > Bill > On Oct 25, 2006, at 10:54 AM, Michael D'Auria wrote: > BackgrounDRb is a plugin, which means it is specific to your app > whereas a gem is a globally available piece of software. The best > gem example i can give is mongrel, you install it once and use it > with all your apps. > > This means that you will have to install the plugin yourself, what > is your dev environment? If you develop on your local machine, > which you should be, then you can just issue "script/plugin install > svn://rubyforge.org//var/svn/backgroundrb" All on one line of > course. You can follow the rest of the installation directions > here: http://backgroundrb.rubyforge.org/. Once it is installed > locally and working locally, you can just simply upload your > project to your hosting provider. > > .: Michael :. And the startup script for BackgrounDRb specifically expects: require RAILS_ROOT + '/vendor/plugins/backgroundrb/backgroundrb.rb' so you really don't have a choice but to install it as a plugin. I just got a deployed BackgrounDRb working on a PowerVPS host and needed to make RAILS_ENV=production explicit as well as adding a 'nohup' to keep the detached process running when deployed (or restarted) through capistrano. I just received a response from TextDrive (i.e., another shared host environment) about my question regarding a port number to use. The suggestion was to use a Unix-domain socket rather than a TCP socket (port). I need to pull the latest code from the svn and see about what kind of patch(es) I might be able to give back to Ezra and one of them may be to permit use of a Unix rather than TCP socket. I'll post my solution back to the list. -Rob Rob Biedenharn http://agileconsultingllc.com Rob at AgileConsultingLLC.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061025/acb79fe1/attachment-0001.html From bill.walton at charter.net Wed Oct 25 16:20:10 2006 From: bill.walton at charter.net (Bill Walton) Date: Wed, 25 Oct 2006 15:20:10 -0500 Subject: [Backgroundrb-devel] deployment issues? References: <00ae01c6f76c$2874a6e0$6401a8c0@Presario> <025601c6f826$048182c0$6401a8c0@Presario> <1907e2ca0610250754ya457ffh555985c024b09712@mail.gmail.com> <50092774-57F9-4E4E-AE85-27AC7D49220C@AgileConsultingLLC.com> Message-ID: <039501c6f872$fb878280$6401a8c0@Presario> Rob Biedenharn wrote: I hadn't even thought about the port issue yet. Thanks. I'll get on it. > I need to pull the latest code from the svn I've just been trying to do the install and keep getting a timeout and "can't connect to rubyforge" message. Let me know if you're able to get in, would you? I always wonder if it's just me ;-) Thanks, Bill From jerrod at indierockmedia.com Fri Oct 27 10:02:05 2006 From: jerrod at indierockmedia.com (Jerrod Blavos) Date: Fri, 27 Oct 2006 10:02:05 -0400 Subject: [Backgroundrb-devel] Error after moving to production server Message-ID: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> so i have an after_create method on my track model that kicks off a background process that converts an mp3 to a swf. this is working fine locally on my machine, but upon moving it to my production server, i am getting the following error. I've only been using backgroundRB for like 48 hours, so i'm not really sure what this means or where to go. any ideas? DRb::DRbConnError (too large packet 1008813135): /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' /app/models/track.rb:55:in `save_swf' /app/models/track.rb:40:in `process' From skaar at waste.org Fri Oct 27 12:15:44 2006 From: skaar at waste.org (skaar) Date: Fri, 27 Oct 2006 11:15:44 -0500 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> Message-ID: <20061027161544.GA32075@waste.org> * Jerrod Blavos (jerrod at indierockmedia.com) [061027 11:05]: > so i have an after_create method on my track model that kicks off a > background process that converts an mp3 to a swf. > this is working fine locally on my machine, but upon moving it to my > production server, i am getting the following error. I've only been > using backgroundRB for like 48 hours, so i'm not really sure what > this means or where to go. any ideas? the stack strace tells that you are attempting to send a 960MB or so payload, and load_limit is 25MB by default. Are you sure that you are sending what you intend to send? /skaar > > DRb::DRbConnError (too large packet 1008813135): > /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' > /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' > /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' > /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' > /app/models/track.rb:55:in `save_swf' > /app/models/track.rb:40:in `process' > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From eimorton at gmail.com Fri Oct 27 12:17:47 2006 From: eimorton at gmail.com (Erik Morton) Date: Fri, 27 Oct 2006 12:17:47 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> Message-ID: <60B16EDF-4028-4BFD-8C0C-DF1A01D4E780@gmail.com> There must be a limit on the size of the packet that DRB will accept. Can you have your worker read the file from a database or the file system, rather than over the DRB socket? Erik On Oct 27, 2006, at 10:02 AM, Jerrod Blavos wrote: > so i have an after_create method on my track model that kicks off a > background process that converts an mp3 to a swf. > this is working fine locally on my machine, but upon moving it to my > production server, i am getting the following error. I've only been > using backgroundRB for like 48 hours, so i'm not really sure what > this means or where to go. any ideas? > > DRb::DRbConnError (too large packet 1008813135): > /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' > /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' > /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' > /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' > /app/models/track.rb:55:in `save_swf' > /app/models/track.rb:40:in `process' > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From ezmobius at gmail.com Fri Oct 27 12:24:56 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Fri, 27 Oct 2006 09:24:56 -0700 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> Message-ID: <2B0FA5D7-291E-47AA-9373-BACB34796D6E@brainspl.at> On Oct 27, 2006, at 7:02 AM, Jerrod Blavos wrote: > so i have an after_create method on my track model that kicks off a > background process that converts an mp3 to a swf. > this is working fine locally on my machine, but upon moving it to my > production server, i am getting the following error. I've only been > using backgroundRB for like 48 hours, so i'm not really sure what > this means or where to go. any ideas? > > DRb::DRbConnError (too large packet 1008813135): > /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' > /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' > /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' > /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' > /app/models/track.rb:55:in `save_swf' > /app/models/track.rb:40:in `process' Hey Jerrod- What are you sending across the wire? This error means you are sending too much data for the drb server to handle with its current settings. Are you sending something really big as :args to your workers ? -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From me at seebq.com Fri Oct 27 12:25:07 2006 From: me at seebq.com (Charles Brian Quinn) Date: Fri, 27 Oct 2006 12:25:07 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> Message-ID: <3a2de0cd0610270925x543fc0e1xdd9232827fec9491@mail.gmail.com> Are you passing the file? Better to pass an id or name of the file and let backgrounDRb load it up. If not, disregard this message, as it appears the actual DRB server got something too big sent to it.... On 10/27/06, Jerrod Blavos wrote: > so i have an after_create method on my track model that kicks off a > background process that converts an mp3 to a swf. > this is working fine locally on my machine, but upon moving it to my > production server, i am getting the following error. I've only been > using backgroundRB for like 48 hours, so i'm not really sure what > this means or where to go. any ideas? > > DRb::DRbConnError (too large packet 1008813135): > /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' > /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' > /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' > /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' > /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' > /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' > /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' > /app/models/track.rb:55:in `save_swf' > /app/models/track.rb:40:in `process' > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Charles Brian Quinn self-promotion: www.seebq.com highgroove studios: www.highgroove.com slingshot hosting: www.slingshothosting.com From jerrod at indierockmedia.com Fri Oct 27 14:51:01 2006 From: jerrod at indierockmedia.com (Jerrod Blavos) Date: Fri, 27 Oct 2006 14:51:01 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <20061027161544.GA32075@waste.org> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> <20061027161544.GA32075@waste.org> Message-ID: <4BAF601E-5098-4D13-97A9-F8A207724A49@indierockmedia.com> interesting. Ok here is what is happening. im asking backgrounDrb to call an ffmpeg command. the command is to convert a 1.9mb mp3 into what end up being a .9mb swf file. sounds like perhaps the ffmpeg command is not right? here is the method on the model thatgets called after_create def save_swf command_ffmpeg = "ffmpeg -i '#{path}' -ab 96 -f swf '# {swf_path}' 2>&1" self.job_key = MiddleMan.new_worker :class=>:audio_worker, :args=>command_ffmpeg end and here is the worker method >>>>>>>>> def do_work(args) @progress = nil @results = nil IO.popen(args) do |pipe| pipe.each do |line| if line =~ /Overwrite ?/ @results = "Problem Encountered. File Already exists." end if line =~ /Duration: (\d\d):(\d\d):(\d\d).(\d)/ @duration = (($1.to_i * 60 + $2.to_i) * 60 + $3.to_i) * 10 + $4.to_i end time_exp = /time=(\d+).(\d+)/ match = time_exp.match(line) if !match == nil? p = ($1.to_i * 10 + $2.to_i) * 100 / @duration p = 100 if p > 98 if progress != p @progress = p if p == 100 @results = "SWF Successfully Created!" end end end end end end >>>>> On Oct 27, 2006, at 12:15 PM, skaar wrote: > * Jerrod Blavos (jerrod at indierockmedia.com) [061027 11:05]: >> so i have an after_create method on my track model that kicks off a >> background process that converts an mp3 to a swf. >> this is working fine locally on my machine, but upon moving it to my >> production server, i am getting the following error. I've only been >> using backgroundRB for like 48 hours, so i'm not really sure what >> this means or where to go. any ideas? > > the stack strace tells that you are attempting to send a 960MB or so > payload, and load_limit is 25MB by default. Are you sure that you are > sending what you intend to send? > > /skaar > >> >> DRb::DRbConnError (too large packet 1008813135): >> /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' >> /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' >> /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' >> /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' >> /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' >> /app/models/track.rb:55:in `save_swf' >> /app/models/track.rb:40:in `process' >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- > ---------------------------------------------------------------------- > |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n > | | >=========== W.A.S.T.E. | genarratologies > |/|/ (_) is the wisdom | skaar at waste.org > ---------------------------------------------------------------------- From jerrod at indierockmedia.com Fri Oct 27 14:53:56 2006 From: jerrod at indierockmedia.com (Jerrod Blavos) Date: Fri, 27 Oct 2006 14:53:56 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <60B16EDF-4028-4BFD-8C0C-DF1A01D4E780@gmail.com> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> <60B16EDF-4028-4BFD-8C0C-DF1A01D4E780@gmail.com> Message-ID: <153C6B12-85BE-4BDF-9B19-AE89F8DD517D@indierockmedia.com> im pretty sure it should be reading from the file system. the worker should just be calling the ffmpeg command using IO.popen (args) do |pipe| On Oct 27, 2006, at 12:17 PM, Erik Morton wrote: > There must be a limit on the size of the packet that DRB will > accept. Can you have your worker read the file from a database or > the file system, rather than over the DRB socket? > > Erik > On Oct 27, 2006, at 10:02 AM, Jerrod Blavos wrote: > >> so i have an after_create method on my track model that kicks off a >> background process that converts an mp3 to a swf. >> this is working fine locally on my machine, but upon moving it to my >> production server, i am getting the following error. I've only been >> using backgroundRB for like 48 hours, so i'm not really sure what >> this means or where to go. any ideas? >> >> DRb::DRbConnError (too large packet 1008813135): >> /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' >> /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' >> /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' >> /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' >> /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' >> /app/models/track.rb:55:in `save_swf' >> /app/models/track.rb:40:in `process' >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From jerrod at indierockmedia.com Fri Oct 27 14:55:13 2006 From: jerrod at indierockmedia.com (Jerrod Blavos) Date: Fri, 27 Oct 2006 14:55:13 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <2B0FA5D7-291E-47AA-9373-BACB34796D6E@brainspl.at> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> <2B0FA5D7-291E-47AA-9373-BACB34796D6E@brainspl.at> Message-ID: <635BF701-D66F-41CA-AB95-9082D31633BE@indierockmedia.com> all im sending through the :args is the text ffmpeg command that im calling through IO.popen(args) hmm. perhaps my command isnt right. On Oct 27, 2006, at 12:24 PM, Ezra Zygmuntowicz wrote: > > On Oct 27, 2006, at 7:02 AM, Jerrod Blavos wrote: > >> so i have an after_create method on my track model that kicks off a >> background process that converts an mp3 to a swf. >> this is working fine locally on my machine, but upon moving it to my >> production server, i am getting the following error. I've only been >> using backgroundRB for like 48 hours, so i'm not really sure what >> this means or where to go. any ideas? >> >> DRb::DRbConnError (too large packet 1008813135): >> /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' >> /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' >> /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' >> /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' >> /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' >> /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' >> /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' >> /app/models/track.rb:55:in `save_swf' >> /app/models/track.rb:40:in `process' > > > Hey Jerrod- > > > What are you sending across the wire? This error means you are > sending too much data for the drb server to handle with its current > settings. Are you sending something really big as :args to your > workers ? > > > > -- Ezra Zygmuntowicz-- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > From jerrod at indierockmedia.com Fri Oct 27 15:39:07 2006 From: jerrod at indierockmedia.com (Jerrod Blavos) Date: Fri, 27 Oct 2006 15:39:07 -0400 Subject: [Backgroundrb-devel] Error after moving to production server In-Reply-To: <20061027191335.GB32075@waste.org> References: <86BFB691-2E5C-4652-9502-50786B51A662@indierockmedia.com> <20061027161544.GA32075@waste.org> <4BAF601E-5098-4D13-97A9-F8A207724A49@indierockmedia.com> <20061027191335.GB32075@waste.org> Message-ID: <878B1007-D47A-4EFA-A2A2-AB15708509FB@indierockmedia.com> Ok that seemed to help. that and i hadnt added the job_key field to the db. oops. either way it is now working beautifully. what will the update/rewrite change? -J On Oct 27, 2006, at 3:13 PM, skaar wrote: > just to check, can you make sure that do_work(args), doesn't return > the > data, can you in the worker do > > def do_work(args) > ... your code ... > nil > end > > We're very close to a new release (total re-write) of backgroundrb, > which might be a lot better for you. Stay tuned :) > > /skaar > > * Jerrod Blavos (jerrod at indierockmedia.com) [061027 14:08]: >> interesting. Ok here is what is happening. im asking backgrounDrb to >> call an ffmpeg command. the command is to convert a 1.9mb mp3 into >> what end up being a .9mb swf file. sounds like perhaps the ffmpeg >> command is not right? >> >> here is the method on the model thatgets called after_create >> >> def save_swf >> command_ffmpeg = "ffmpeg -i '#{path}' -ab 96 -f swf '# >> {swf_path}' 2>&1" >> self.job_key = >> MiddleMan.new_worker :class=>:audio_worker, :args=>command_ffmpeg >> end >> >> and here is the worker method >> >>>>>>>>>>> >> def do_work(args) >> @progress = nil >> @results = nil >> IO.popen(args) do |pipe| >> pipe.each do |line| >> if line =~ /Overwrite ?/ >> @results = "Problem Encountered. File Already exists." >> end >> if line =~ /Duration: (\d\d):(\d\d):(\d\d).(\d)/ >> @duration = (($1.to_i * 60 + $2.to_i) * 60 + $3.to_i) * >> 10 + $4.to_i >> end >> time_exp = /time=(\d+).(\d+)/ >> match = time_exp.match(line) >> if !match == nil? >> p = ($1.to_i * 10 + $2.to_i) * 100 / @duration >> p = 100 if p > 98 >> if progress != p >> @progress = p >> if p == 100 >> @results = "SWF Successfully Created!" >> end >> end >> end >> end >> end >> end >>>>>>> >> >> >> >> >> >> On Oct 27, 2006, at 12:15 PM, skaar wrote: >> >>> * Jerrod Blavos (jerrod at indierockmedia.com) [061027 11:05]: >>>> so i have an after_create method on my track model that kicks off a >>>> background process that converts an mp3 to a swf. >>>> this is working fine locally on my machine, but upon moving it >>>> to my >>>> production server, i am getting the following error. I've only >>>> been >>>> using backgroundRB for like 48 hours, so i'm not really sure what >>>> this means or where to go. any ideas? >>> >>> the stack strace tells that you are attempting to send a 960MB or so >>> payload, and load_limit is 25MB by default. Are you sure that you >>> are >>> sending what you intend to send? >>> >>> /skaar >>> >>>> >>>> DRb::DRbConnError (too large packet 1008813135): >>>> /usr/lib/ruby/1.8/drb/drb.rb:570:in `load' >>>> /usr/lib/ruby/1.8/drb/drb.rb:629:in `recv_reply' >>>> /usr/lib/ruby/1.8/drb/drb.rb:918:in `recv_reply' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1192:in `send_message' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1083:in `method_missing' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1167:in `open' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1082:in `method_missing' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1100:in `with_friend' >>>> /usr/lib/ruby/1.8/drb/drb.rb:1081:in `method_missing' >>>> /app/models/track.rb:55:in `save_swf' >>>> /app/models/track.rb:40:in `process' >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >>> -- >>> -------------------------------------------------------------------- >>> -- >>> |\|\ where in the | >>> s_u_b_s_t_r_u_c_t_i_o_n >>> | | >=========== W.A.S.T.E. | >>> genarratologies >>> |/|/ (_) is the wisdom | >>> skaar at waste.org >>> -------------------------------------------------------------------- >>> -- >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > -- > ---------------------------------------------------------------------- > |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n > | | >=========== W.A.S.T.E. | genarratologies > |/|/ (_) is the wisdom | skaar at waste.org > ---------------------------------------------------------------------- From joshua at zaadz.com Fri Oct 27 16:15:15 2006 From: joshua at zaadz.com (Joshua Warchol) Date: Fri, 27 Oct 2006 16:15:15 -0400 Subject: [Backgroundrb-devel] double runs from repeat_every Message-ID: <454268D3.6090708@zaadz.com> I'm just starting out with BackgrounDRB and I'm pretty excited. I know there's some development going on, but would patches against the released code be welcome? Nothing profound, just some documentation errors. Now for the real reason I'm writing. I'm testing the new(ish) repeat_every code and it appears to be having a problem. I've got a very, very simple worker. The do_work method just logs the current time, job key and process ID. It's started from an autostart configuration in backgroundrb.yml. Most of the time it seems to work just fine. If I say repeat_every 1.minute, it logs every minute. But somewhat often it double logs, indicating the do_work call was hit twice. I've tried this with a variety of repeat_every & timer_sleep values. I imagine the start_timer loop is somehow calling start_process on this same job more than once. Not sure how that could happen, from my read of the code. Anyone else seen this before? What evidence can I provide to help figure this out? Josh -- Joshua Warchol Code Poet, Zaadz Inc. http://joshua.zaadz.com/ From cssheong at pacific.net.sg Fri Oct 27 20:05:43 2006 From: cssheong at pacific.net.sg (Chang Sau Sheong) Date: Sat, 28 Oct 2006 08:05:43 +0800 Subject: [Backgroundrb-devel] Problem starting up the drb server Message-ID: <45429ED7.5020602@pacific.net.sg> Hi, I'm rather new to backgrounDRb and DRb itself. I tried to run backgrounDRb on my shared hosting site at Railsplayground.com using rake backgroundrb:start and got this error: /usr/local/lib/ruby/1.8/drb/drb.rb:862:in `initialize': getaddrinfo: Name or service not known (SocketError) from /usr/local/lib/ruby/1.8/drb/drb.rb:862:in `open_server' from /usr/local/lib/ruby/1.8/drb/drb.rb:756:in `open_server' from /usr/local/lib/ruby/1.8/drb/drb.rb:754:in `open_server' from /usr/local/lib/ruby/1.8/drb/drb.rb:1336:in `initialize' from /usr/local/lib/ruby/1.8/drb/drb.rb:1624:in `start_service' from /home/saush/reachible/config/../script/backgroundrb/../../config/../vendor/plugins/backgroundrb/lib/backgroundrb.rb:11 from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:21:in `require' from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:147:in `require' ... 8 levels... from /home/saush/reachible/config/../script/backgroundrb/../../config/environment.rb:13 from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:21:in `require' from /usr/local/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:147:in `require' from /home/saush/reachible/config/../script/backgroundrb/start:68 Can anyone from this list help? Thks! -- CHANG Sau Sheong http://blog.saush.com - brain dump http://jaccal.sourceforge.net - smart card toolkit http://screensvr.rubyforge.org - Flickr screensaver http://ced.dev.java.net - Chinese-English Dictionary http://www.projectible.com - online project publishing Fellow, Singapore Institute of Arbitrators -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20061028/0367fe3d/attachment.html From ezmobius at gmail.com Sun Oct 29 20:40:46 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Sun, 29 Oct 2006 17:40:46 -0800 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. Message-ID: Hello Folks- I am really happy to announce the release of BackgrounDRb 0.2.0 . This is a complete rewrite and re-architecture of this codebase. I want to thank skaar for all of his hard work. I put out a call for help with the new architecture and skaar stepped up bigtime. I had already written the new scheduler and cron compatible syntax and moved to multi process architecture. But he took what I had that wasn't very whole and did a whopping 100 svn commits to the project! Big ups to skaar for this release! = BackgrounDRb 0.2.0 !! Note that this is in an entirely different svn repo then the old plugin. Make sure you remove the old before you install the new. http://svn.devjavu.com/backgroundrb/tags/release-0.2.0/README http://svn.devjavu.com/backgroundrb/tags/release-0.2.0 (latest release) http://svn.devjavu.com/backgroundrb/trunk (svn trunk) http://backgroundrb.devjavu.com (trac) http://backgroundrb.rubyforge.org (rdoc) Copyright (c) 2006 Ezra Zygmuntowicz and skaar[at]waste[dot]org == DISCLAIMER The 0.2.0 release of BackgrounDRb is a complete re-write of previous releases and is to be considered experimental, in-complete and in many respect untested. Our goal is to reach a release recommended for production use by 0.3.x. This said, this release is a more robust solution that the previous release. Also note that support for Windows is deprecated with this release of BackgrounDRb, sorry : ( . It may work under cygwin but we have not tried it. == Introduction BackgrounDRb is a ruby job server and scheduler. It main intent is to be used with Ruby on Rails applications for offloading long running tasks. Since a rails application blocks while servicing a request it is best to move long running tasks off into a background process that is divorced from the http request/response cycle. This new release of BackgrounDRb is also modular and can be used without Rails. So any ruby program or framework can use it. == Technology Overview This 0.2.x branch of BackgrounDRb introduces a completely new architecture then the previous versions. Instead of a single process, multi threaded environment, the new system uses multi process with IPC to manage workers. So each of your workers will run in their own ruby interpreter. The interface that you use within rails remains mostly unchanged. The new architecture allows for a much more robust system. Each of your worker classes get run in their own ruby interpreter. The main server process holds a collection of references to your running worker objects. This acts as the interface or MiddleMan that you use to interact with your remote jobs from rails or other ruby client code. When workers are spawned an interprocess communication channel is opened between the MiddleMan and the worker. There is a heartbeat setup between parent and child processes so that if you kill the parent, all children processes are killed off the next time they try to hearbeat. All of this happens transparently to you as a user. The new release comes with a brand new job scheduler. You can use simple triggers like repeat_every or you can get more complex and use full cron compatible syntax. Workers can be scheduled by two built in 'trigger' types. A simple 'trigger' is specified with start, stop and interval: require 'active_support' MiddleMan.schedule_worker( :class => :example_worker, :job_key => :schedule_test, :worker_method => :other_method, :trigger_args => { :start => Time.now + 5.seconds, :end => Time.now + 10.minutes, :repeat_interval => 30.seconds } The cron trigger uses a similar syntax to cron found on UNIX systems: MiddleMan.schedule_worker( :class => simple_class, :job_key => :schedule_test, :worker_method => :arg_method, :worker_method_args => "my argument to arg_method", :trigger_args => "0 15 10 * * * *" ) Also note that when the server starts up, you will see 3 processes running. One of the is the MiddleMan server, one is the results worker and one is the logger worker. When you do a logger.info("foo log!") in your workers, you are actually logging to the Logger worker. As you might imagine, this new way of managing multiple processes will scale a lot better then the multi threaded single process version ;) But also be aware that there is still a thread pool in the middleman that you can control the size of. All this does is keep the plugin from spawning too many processes. It will allow however many workers you specify to run at once and any more then that will just queue up and wit for their turn to spawn. If you are already a user of the old BackgrounDRb please give the new version a tryout. It should not require very much work to port your worker classes to the new architecture. This is alpha software folks. It works for me? but we will not be held responsible if it ruins your day ;) -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From info at siebert-wd.de Mon Oct 30 11:01:37 2006 From: info at siebert-wd.de (Siebert Michael) Date: Mon, 30 Oct 2006 17:01:37 +0100 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: <941256A3-0A7A-49C2-918F-2DD359F1D1B4@siebert-wd.de> First of all: Thanks you two guys out there! It looks very nice, although it seems i wont use the new version since the one you gave me (pre-pre-pre version with worker pool and threads) works perfectly for my current project. but i will surely try it out. great work! PS: wheres the restart rake task gone? :) Am 30.10.2006 um 02:40 schrieb Ezra Zygmuntowicz: > Hello Folks- > > I am really happy to announce the release of BackgrounDRb 0.2.0 . > This is a complete rewrite and re-architecture of this codebase. I > want to > thank skaar for all of his hard work. I put out a call for help with > the new > architecture and skaar stepped up bigtime. I had already written > the new > scheduler and cron compatible syntax and moved to multi process > architecture. But he took what I had that wasn't very whole and did a > whopping 100 svn commits to the project! Big ups to skaar for this > release! > > > = BackgrounDRb 0.2.0 > > > !! Note that this is in an entirely different svn repo then the old > plugin. > Make sure you remove the old before you install the new. > > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0/README > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0 (latest > release) > http://svn.devjavu.com/backgroundrb/trunk (svn trunk) > http://backgroundrb.devjavu.com (trac) > http://backgroundrb.rubyforge.org (rdoc) > > Copyright (c) 2006 Ezra Zygmuntowicz and skaar[at]waste[dot]org > > == DISCLAIMER > > The 0.2.0 release of BackgrounDRb is a complete re-write of previous > releases and is to be considered experimental, in-complete and in many > respect untested. Our goal is to reach a release recommended for > production use by 0.3.x. This said, this release is a more robust > solution that the previous release. > > Also note that support for Windows is deprecated with this release of > BackgrounDRb, sorry : ( . It may work under cygwin but we have not > tried it. > > == Introduction > > BackgrounDRb is a ruby job server and scheduler. It main intent is > to be > used with Ruby on Rails applications for offloading long running > tasks. > Since a rails application blocks while servicing a request it is best > to move long running tasks off into a background process that is > divorced > from the http request/response cycle. > > This new release of BackgrounDRb is also modular and can be used > without > Rails. So any ruby program or framework can use it. > > == Technology Overview > > This 0.2.x branch of BackgrounDRb introduces a completely new > architecture > then the previous versions. Instead of a single process, multi > threaded environment, > the new system uses multi process with IPC to manage workers. So each > of your > workers will run in their own ruby interpreter. The interface that you > use within rails remains mostly unchanged. > > The new architecture allows for a much more robust system. Each of > your > worker classes get run in their own ruby interpreter. The main server > process > holds a collection of references to your running worker objects. This > acts as > the interface or MiddleMan that you use to interact with your remote > jobs from > rails or other ruby client code. When workers are spawned an > interprocess > communication channel is opened between the MiddleMan and the worker. > There is a heartbeat setup between parent and child processes so > that if > you kill the parent, all children processes are killed off the next > time they try to > hearbeat. All of this happens transparently to you as a user. > > The new release comes with a brand new job scheduler. You can use > simple triggers like repeat_every or you can get more complex and use > full cron compatible syntax. > > Workers can be scheduled by two built in 'trigger' types. A simple > 'trigger' is specified with start, stop and interval: > > require 'active_support' > MiddleMan.schedule_worker( > :class => :example_worker, > :job_key => :schedule_test, > :worker_method => :other_method, > :trigger_args => { > :start => Time.now + 5.seconds, > :end => Time.now + 10.minutes, > :repeat_interval => 30.seconds > } > > The cron trigger uses a similar syntax to cron found on UNIX systems: > > MiddleMan.schedule_worker( > :class => simple_class, > :job_key => :schedule_test, > :worker_method => :arg_method, > :worker_method_args => "my argument to arg_method", > :trigger_args => "0 15 10 * * * *" > ) > > > Also note that when the server starts up, you will see 3 processes > running. > One of the is the MiddleMan server, one is the results worker and > one is > the logger worker. When you do a logger.info("foo log!") in your > workers, > you are actually logging to the Logger worker. > > As you might imagine, this new way of managing multiple processes will > scale a lot better then the multi threaded single process > version ;) But > also be aware that there is still a thread pool in the middleman > that you > can control the size of. All this does is keep the plugin from > spawning too > many processes. It will allow however many workers you specify to > run at > once and any more then that will just queue up and wit for their > turn to > spawn. > > If you are already a user of the old BackgrounDRb please give the new > version a tryout. It should not require very much work to port your > worker > classes to the new architecture. > > This is alpha software folks. It works for me? but we will not be > held responsible > if it ruins your day ;) > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel ----------------------------- Siebert Michael info at siebert-wd.de From eimorton at gmail.com Mon Oct 30 11:19:35 2006 From: eimorton at gmail.com (Erik Morton) Date: Mon, 30 Oct 2006 11:19:35 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: <467B5138-6A19-4F58-B661-5D7DE693017A@gmail.com> Fantastic news! And perfect timing too, as I am beginning a very large project that will rely heavily on Backgroundrb and will gladly help root out bugs. I did hit one snag in getting it up and running. I removed the current version of BackgroundRB and installed the trunk. I ran setup and then backgroundrb start and received the following error: $ script/backgroundrb start ERROR: there is already one or more instance(s) of the program running However there was no BackgroundRB running. I'm on OSX using daemons (1.0.1) and slave (1.0.0). I couldn't find that the error message in the backgroundrb source. Any thoughts? Erik On Oct 29, 2006, at 8:40 PM, Ezra Zygmuntowicz wrote: > Hello Folks- > > I am really happy to announce the release of BackgrounDRb 0.2.0 . > This is a complete rewrite and re-architecture of this codebase. I > want to > thank skaar for all of his hard work. I put out a call for help with > the new > architecture and skaar stepped up bigtime. I had already written > the new > scheduler and cron compatible syntax and moved to multi process > architecture. But he took what I had that wasn't very whole and did a > whopping 100 svn commits to the project! Big ups to skaar for this > release! > > > = BackgrounDRb 0.2.0 > > > !! Note that this is in an entirely different svn repo then the old > plugin. > Make sure you remove the old before you install the new. > > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0/README > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0 (latest > release) > http://svn.devjavu.com/backgroundrb/trunk (svn trunk) > http://backgroundrb.devjavu.com (trac) > http://backgroundrb.rubyforge.org (rdoc) > > Copyright (c) 2006 Ezra Zygmuntowicz and skaar[at]waste[dot]org > > == DISCLAIMER > > The 0.2.0 release of BackgrounDRb is a complete re-write of previous > releases and is to be considered experimental, in-complete and in many > respect untested. Our goal is to reach a release recommended for > production use by 0.3.x. This said, this release is a more robust > solution that the previous release. > > Also note that support for Windows is deprecated with this release of > BackgrounDRb, sorry : ( . It may work under cygwin but we have not > tried it. > > == Introduction > > BackgrounDRb is a ruby job server and scheduler. It main intent is > to be > used with Ruby on Rails applications for offloading long running > tasks. > Since a rails application blocks while servicing a request it is best > to move long running tasks off into a background process that is > divorced > from the http request/response cycle. > > This new release of BackgrounDRb is also modular and can be used > without > Rails. So any ruby program or framework can use it. > > == Technology Overview > > This 0.2.x branch of BackgrounDRb introduces a completely new > architecture > then the previous versions. Instead of a single process, multi > threaded environment, > the new system uses multi process with IPC to manage workers. So each > of your > workers will run in their own ruby interpreter. The interface that you > use within rails remains mostly unchanged. > > The new architecture allows for a much more robust system. Each of > your > worker classes get run in their own ruby interpreter. The main server > process > holds a collection of references to your running worker objects. This > acts as > the interface or MiddleMan that you use to interact with your remote > jobs from > rails or other ruby client code. When workers are spawned an > interprocess > communication channel is opened between the MiddleMan and the worker. > There is a heartbeat setup between parent and child processes so > that if > you kill the parent, all children processes are killed off the next > time they try to > hearbeat. All of this happens transparently to you as a user. > > The new release comes with a brand new job scheduler. You can use > simple triggers like repeat_every or you can get more complex and use > full cron compatible syntax. > > Workers can be scheduled by two built in 'trigger' types. A simple > 'trigger' is specified with start, stop and interval: > > require 'active_support' > MiddleMan.schedule_worker( > :class => :example_worker, > :job_key => :schedule_test, > :worker_method => :other_method, > :trigger_args => { > :start => Time.now + 5.seconds, > :end => Time.now + 10.minutes, > :repeat_interval => 30.seconds > } > > The cron trigger uses a similar syntax to cron found on UNIX systems: > > MiddleMan.schedule_worker( > :class => simple_class, > :job_key => :schedule_test, > :worker_method => :arg_method, > :worker_method_args => "my argument to arg_method", > :trigger_args => "0 15 10 * * * *" > ) > > > Also note that when the server starts up, you will see 3 processes > running. > One of the is the MiddleMan server, one is the results worker and > one is > the logger worker. When you do a logger.info("foo log!") in your > workers, > you are actually logging to the Logger worker. > > As you might imagine, this new way of managing multiple processes will > scale a lot better then the multi threaded single process > version ;) But > also be aware that there is still a thread pool in the middleman > that you > can control the size of. All this does is keep the plugin from > spawning too > many processes. It will allow however many workers you specify to > run at > once and any more then that will just queue up and wit for their > turn to > spawn. > > If you are already a user of the old BackgrounDRb please give the new > version a tryout. It should not require very much work to port your > worker > classes to the new architecture. > > This is alpha software folks. It works for me? but we will not be > held responsible > if it ruins your day ;) > > -- Ezra Zygmuntowicz > -- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From skaar at waste.org Mon Oct 30 11:27:06 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 10:27:06 -0600 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: <941256A3-0A7A-49C2-918F-2DD359F1D1B4@siebert-wd.de> References: <941256A3-0A7A-49C2-918F-2DD359F1D1B4@siebert-wd.de> Message-ID: <20061030162705.GA26594@waste.org> > PS: wheres the restart rake task gone? :) it's just one of several things that was in the old release that we didn't get around to re-add - please report these in trac at: http://backgroundrb.devjavu.com/ (you will have to register) and we'll get to them as soon as we can - there will be at least one more 0.2.x release before we reach 0.3.x which would be the release we expect to be "production ready" ( whatever that means :) ) All feedback and testing is much appreciated. /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From skaar at waste.org Mon Oct 30 13:46:12 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 12:46:12 -0600 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: <467B5138-6A19-4F58-B661-5D7DE693017A@gmail.com> References: <467B5138-6A19-4F58-B661-5D7DE693017A@gmail.com> Message-ID: <20061030184612.GB26594@waste.org> > However there was no BackgroundRB running. I'm on OSX using daemons > (1.0.1) and slave (1.0.0). I couldn't find that the error message in > the backgroundrb source. Ezra had issues with daemons pre 1.0.2 on MacOS X - which is pretty much why we said 1.0.2 as a dependency in the readme - give it a shot. /skaar From ezmobius at gmail.com Mon Oct 30 13:56:33 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 10:56:33 -0800 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: <467B5138-6A19-4F58-B661-5D7DE693017A@gmail.com> References: <467B5138-6A19-4F58-B661-5D7DE693017A@gmail.com> Message-ID: <08A3BFEB-58AF-4FCF-BD06-4F2CE16474B5@gmail.com> Erik- That error means that there is an old log/backgroundrb.pid file. If you remove it then that error should go away. -Ezra On Oct 30, 2006, at 8:19 AM, Erik Morton wrote: > Fantastic news! And perfect timing too, as I am beginning a very > large project that will rely heavily on Backgroundrb and will gladly > help root out bugs. > > I did hit one snag in getting it up and running. I removed the > current version of BackgroundRB and installed the trunk. > > I ran setup and then backgroundrb start and received the following > error: > > $ script/backgroundrb start > ERROR: there is already one or more instance(s) of the program running > > However there was no BackgroundRB running. I'm on OSX using daemons > (1.0.1) and slave (1.0.0). I couldn't find that the error message in > the backgroundrb source. > > Any thoughts? > > Erik > On Oct 29, 2006, at 8:40 PM, Ezra Zygmuntowicz wrote: > >> Hello Folks- >> >> I am really happy to announce the release of BackgrounDRb 0.2.0 . >> This is a complete rewrite and re-architecture of this codebase. I >> want to >> thank skaar for all of his hard work. I put out a call for help with >> the new >> architecture and skaar stepped up bigtime. I had already written >> the new >> scheduler and cron compatible syntax and moved to multi process >> architecture. But he took what I had that wasn't very whole and did a >> whopping 100 svn commits to the project! Big ups to skaar for this >> release! >> >> >> = BackgrounDRb 0.2.0 >> >> >> !! Note that this is in an entirely different svn repo then the old >> plugin. >> Make sure you remove the old before you install the new. >> >> http://svn.devjavu.com/backgroundrb/tags/release-0.2.0/README >> http://svn.devjavu.com/backgroundrb/tags/release-0.2.0 (latest >> release) >> http://svn.devjavu.com/backgroundrb/trunk (svn trunk) >> http://backgroundrb.devjavu.com (trac) >> http://backgroundrb.rubyforge.org (rdoc) >> >> Copyright (c) 2006 Ezra Zygmuntowicz and skaar[at]waste[dot]org >> >> == DISCLAIMER >> >> The 0.2.0 release of BackgrounDRb is a complete re-write of previous >> releases and is to be considered experimental, in-complete and in >> many >> respect untested. Our goal is to reach a release recommended for >> production use by 0.3.x. This said, this release is a more robust >> solution that the previous release. >> >> Also note that support for Windows is deprecated with this release of >> BackgrounDRb, sorry : ( . It may work under cygwin but we have not >> tried it. >> >> == Introduction >> >> BackgrounDRb is a ruby job server and scheduler. It main intent is >> to be >> used with Ruby on Rails applications for offloading long running >> tasks. >> Since a rails application blocks while servicing a request it is best >> to move long running tasks off into a background process that is >> divorced >> from the http request/response cycle. >> >> This new release of BackgrounDRb is also modular and can be used >> without >> Rails. So any ruby program or framework can use it. >> >> == Technology Overview >> >> This 0.2.x branch of BackgrounDRb introduces a completely new >> architecture >> then the previous versions. Instead of a single process, multi >> threaded environment, >> the new system uses multi process with IPC to manage workers. So each >> of your >> workers will run in their own ruby interpreter. The interface that >> you >> use within rails remains mostly unchanged. >> >> The new architecture allows for a much more robust system. Each of >> your >> worker classes get run in their own ruby interpreter. The main server >> process >> holds a collection of references to your running worker objects. This >> acts as >> the interface or MiddleMan that you use to interact with your remote >> jobs from >> rails or other ruby client code. When workers are spawned an >> interprocess >> communication channel is opened between the MiddleMan and the worker. >> There is a heartbeat setup between parent and child processes so >> that if >> you kill the parent, all children processes are killed off the next >> time they try to >> hearbeat. All of this happens transparently to you as a user. >> >> The new release comes with a brand new job scheduler. You can use >> simple triggers like repeat_every or you can get more complex and use >> full cron compatible syntax. >> >> Workers can be scheduled by two built in 'trigger' types. A simple >> 'trigger' is specified with start, stop and interval: >> >> require 'active_support' >> MiddleMan.schedule_worker( >> :class => :example_worker, >> :job_key => :schedule_test, >> :worker_method => :other_method, >> :trigger_args => { >> :start => Time.now + 5.seconds, >> :end => Time.now + 10.minutes, >> :repeat_interval => 30.seconds >> } >> >> The cron trigger uses a similar syntax to cron found on UNIX systems: >> >> MiddleMan.schedule_worker( >> :class => simple_class, >> :job_key => :schedule_test, >> :worker_method => :arg_method, >> :worker_method_args => "my argument to arg_method", >> :trigger_args => "0 15 10 * * * *" >> ) >> >> >> Also note that when the server starts up, you will see 3 processes >> running. >> One of the is the MiddleMan server, one is the results worker and >> one is >> the logger worker. When you do a logger.info("foo log!") in your >> workers, >> you are actually logging to the Logger worker. >> >> As you might imagine, this new way of managing multiple processes >> will >> scale a lot better then the multi threaded single process >> version ;) But >> also be aware that there is still a thread pool in the middleman >> that you >> can control the size of. All this does is keep the plugin from >> spawning too >> many processes. It will allow however many workers you specify to >> run at >> once and any more then that will just queue up and wit for their >> turn to >> spawn. >> >> If you are already a user of the old BackgrounDRb please give the new >> version a tryout. It should not require very much work to port your >> worker >> classes to the new architecture. >> >> This is alpha software folks. It works for me? but we will not be >> held responsible >> if it ruins your day ;) >> >> -- Ezra Zygmuntowicz >> -- Lead Rails Evangelist >> -- ez at engineyard.com >> -- Engine Yard, Serious Rails Hosting >> -- (866) 518-YARD (9273) >> >> >> >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From joshua at zaadz.com Mon Oct 30 14:23:20 2006 From: joshua at zaadz.com (Joshua Warchol) Date: Mon, 30 Oct 2006 14:23:20 -0500 Subject: [Backgroundrb-devel] Question about new results hash Message-ID: <45465128.1040503@zaadz.com> Is it really a hash? As in, can I pass it into a method that a worker is calling so that, for example, a method in my ActiveRecord that's actually doing the work? Really liking the new code. Does rubyforge or devjavu have the rdocs anywhere? -- Joshua Warchol Code Poet, Zaadz Inc. http://joshua.zaadz.com/ From skaar at waste.org Mon Oct 30 14:38:28 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 13:38:28 -0600 Subject: [Backgroundrb-devel] Question about new results hash In-Reply-To: <45465128.1040503@zaadz.com> References: <45465128.1040503@zaadz.com> Message-ID: <20061030193828.GC26594@waste.org> * Joshua Warchol (joshua at zaadz.com) [061030 13:34]: > Is it really a hash? As in, can I pass it into a method that a worker is > calling so that, for example, a method in my ActiveRecord that's > actually doing the work? if you want to be on the safe side, you might call results.to_hash, which should be ok. I've not tried to pass results itself to AR - please try it and report back how it behaves. > Really liking the new code. Does rubyforge or devjavu have the rdocs > anywhere? rdoc: http://backgroundrb.rubyforge.org /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From hutch at recursive.ca Mon Oct 30 14:46:43 2006 From: hutch at recursive.ca (Bob Hutchison) Date: Mon, 30 Oct 2006 14:46:43 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: Hi, On 29-Oct-06, at 8:40 PM, Ezra Zygmuntowicz wrote: > I am really happy to announce the release of BackgrounDRb 0.2.0 . This is good news. Say you were running a few background processes and someone or something killed your main process. Is there any way of knowing which processes died because of heart failure when the server is restarted? Cheers, Bob ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby -- From skaar at waste.org Mon Oct 30 14:59:54 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 13:59:54 -0600 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: <20061030195953.GD26594@waste.org> > Say you were running a few background processes and someone or > something killed your main process. Is there any way of knowing which > processes died because of heart failure when the server is restarted? The child processes should died when the main process dies. The workers are created using the Slave library, which sets up a heartbeat in the child. We don't persist which workers was running. What is it that you are trying to do? /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From ezmobius at gmail.com Mon Oct 30 15:01:17 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 12:01:17 -0800 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: On Oct 30, 2006, at 11:46 AM, Bob Hutchison wrote: > Hi, > > On 29-Oct-06, at 8:40 PM, Ezra Zygmuntowicz wrote: > >> I am really happy to announce the release of BackgrounDRb 0.2.0 . > > > This is good news. > > Say you were running a few background processes and someone or > something killed your main process. Is there any way of knowing > which processes died because of heart failure when the server is > restarted? > > Cheers, > Bob > \ Hey bob- Thats a good question. The current answer is no, if the middleman server gets killed, all workers will get killed as well. But looking to the future we will try to have a way to have dual middlemen so they can watch each other and bring the other one up if it dies. And maybe a way to serialize the current jobs to disk like journalling so if things crash they could be restored easily. Please do give me your thoughts on this. We are going to be working on this thing to make it awesome by 0.3.0 so if there are things you want to see then speak up. Thanks -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From joshua at zaadz.com Mon Oct 30 15:17:09 2006 From: joshua at zaadz.com (Joshua Warchol) Date: Mon, 30 Oct 2006 15:17:09 -0500 Subject: [Backgroundrb-devel] SVN requiring authentication Message-ID: <45465DC5.1040906@zaadz.com> I can't get to the code anymore. Devjavu's svn is asking for a username and password. http://svn.devjavu.com/backgroundrb/ Authorization Required This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required. ------------------------------------------------------------------------ Apache/2.0.59 (FreeBSD) mod_fastcgi/2.4.2 DAV/2 PHP/5.1.4 SVN/1.4.0 Server at svn.devjavu.com Port 80 -- Joshua Warchol Code Poet, Zaadz Inc. http://joshua.zaadz.com/ From joshua at zaadz.com Mon Oct 30 15:21:43 2006 From: joshua at zaadz.com (Joshua Warchol) Date: Mon, 30 Oct 2006 15:21:43 -0500 Subject: [Backgroundrb-devel] Testing Message-ID: <45465ED7.8000208@zaadz.com> With the 0.2.0 code, unit testing of workers isn't working out of the box. I created a bare rails project (dubbed "ninja") and generated one worker "FirstWorker" and did rake test rake test (in /home/joshua/tmp/20061030-bdbtest/ninja) /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" "test/unit/first_worker_test.rb" /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:123:in `const_missing': uninitialized constant Worker (NameError) from /home/joshua/tmp/20061030-bdbtest/ninja/config/../lib/workers/first_worker.rb:5 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:21:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/dependencies.rb:147:in `require' from ./test/unit/first_worker_test.rb:2 from /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb:5 from /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb:5 /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/rake_test_loader.rb" rake aborted! Test failures -- Joshua Warchol Code Poet, Zaadz Inc. http://joshua.zaadz.com/ From ezmobius at gmail.com Mon Oct 30 15:27:13 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 12:27:13 -0800 Subject: [Backgroundrb-devel] SVN requiring authentication In-Reply-To: <45465DC5.1040906@zaadz.com> References: <45465DC5.1040906@zaadz.com> Message-ID: On Oct 30, 2006, at 12:17 PM, Joshua Warchol wrote: > I can't get to the code anymore. Devjavu's svn is asking for a > username > and password. > > http://svn.devjavu.com/backgroundrb/ > > > Authorization Required > > This server could not verify that you are authorized to access the > document requested. Either you supplied the wrong credentials > (e.g., bad > password), or your browser doesn't understand how to supply the > credentials required. > > ---------------------------------------------------------------------- > -- > Apache/2.0.59 (FreeBSD) mod_fastcgi/2.4.2 DAV/2 PHP/5.1.4 SVN/1.4.0 > Server at svn.devjavu.com Port 80 > > > -- > Joshua Warchol > Code Poet, Zaadz Inc. > http://joshua.zaadz.com/ Yeah sorry Joshua. It looks like devjavu is having some sort of problem right now. It did the same thing to my other projects there as well. I email the owners and it should be restored shortly. Sorry , Thanks -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Oct 30 15:33:52 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 12:33:52 -0800 Subject: [Backgroundrb-devel] SVN requiring authentication In-Reply-To: <45465DC5.1040906@zaadz.com> References: <45465DC5.1040906@zaadz.com> Message-ID: <6AA2C31D-D8B0-4E9E-A25D-222A2D3C909F@brainspl.at> Ok it's fixed now -Ezra On Oct 30, 2006, at 12:17 PM, Joshua Warchol wrote: > I can't get to the code anymore. Devjavu's svn is asking for a > username > and password. > > http://svn.devjavu.com/backgroundrb/ > > > Authorization Required > > This server could not verify that you are authorized to access the > document requested. Either you supplied the wrong credentials > (e.g., bad > password), or your browser doesn't understand how to supply the > credentials required. > > ---------------------------------------------------------------------- > -- > Apache/2.0.59 (FreeBSD) mod_fastcgi/2.4.2 DAV/2 PHP/5.1.4 SVN/1.4.0 > Server at svn.devjavu.com Port 80 > > > -- > Joshua Warchol > Code Poet, Zaadz Inc. > http://joshua.zaadz.com/ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Oct 30 15:52:21 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 12:52:21 -0800 Subject: [Backgroundrb-devel] Testing In-Reply-To: <45465ED7.8000208@zaadz.com> References: <45465ED7.8000208@zaadz.com> Message-ID: Hey Joshua- Sorry about that. I forgot to commit one require statement in the tests. please svn up and run again. also note that most of the tests are now specs. so also run rake specs Cheers- -Ezra On Oct 30, 2006, at 12:21 PM, Joshua Warchol wrote: > With the 0.2.0 code, unit testing of workers isn't working out of > the box. > > I created a bare rails project (dubbed "ninja") and generated one > worker > "FirstWorker" and did rake test > > rake test > (in /home/joshua/tmp/20061030-bdbtest/ninja) > /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/ > lib/rake/rake_test_loader.rb" "test/unit/first_worker_test.rb" > /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/active_support/ > dependencies.rb:123:in `const_missing': uninitialized constant > Worker (NameError) > from /home/joshua/tmp/20061030-bdbtest/ninja/config/../lib/ > workers/first_worker.rb:5 > from /usr/local/lib/site_ruby/1.8/rubygems/ > custom_require.rb:21:in `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.3.1/lib/ > active_support/dependencies.rb:147:in `require' > from ./test/unit/first_worker_test.rb:2 > from /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/ > rake_test_loader.rb:5 > from /usr/lib/ruby/gems/1.8/gems/rake-0.7.1/lib/rake/ > rake_test_loader.rb:5 > /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/ > lib/rake/rake_test_loader.rb" > /usr/bin/ruby1.8 -Ilib:test "/usr/lib/ruby/gems/1.8/gems/rake-0.7.1/ > lib/rake/rake_test_loader.rb" > rake aborted! > Test failures > > > > -- > Joshua Warchol > Code Poet, Zaadz Inc. > http://joshua.zaadz.com/ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Oct 30 16:01:18 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 13:01:18 -0800 Subject: [Backgroundrb-devel] Question about new results hash In-Reply-To: <45465128.1040503@zaadz.com> References: <45465128.1040503@zaadz.com> Message-ID: <5E704F5F-A1F2-4982-82AF-F088102DAECF@brainspl.at> On Oct 30, 2006, at 11:23 AM, Joshua Warchol wrote: > Is it really a hash? As in, can I pass it into a method that a > worker is > calling so that, for example, a method in my ActiveRecord that's > actually doing the work? > > Really liking the new code. Does rubyforge or devjavu have the rdocs > anywhere? > > -- > Joshua Warchol > Code Poet, Zaadz Inc. > http://joshua.zaadz.com/ Hey Joshua- ` I'm not entirely certain what you are asking? Can you clarify a bit please? The rdoc is here: http://backgroundrb.rubyforge.org/ Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Oct 30 16:24:59 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 13:24:59 -0800 Subject: [Backgroundrb-devel] Question about new results hash In-Reply-To: <45466B5E.5020106@zaadz.com> References: <45465128.1040503@zaadz.com> <5E704F5F-A1F2-4982-82AF-F088102DAECF@brainspl.at> <45466B5E.5020106@zaadz.com> Message-ID: On Oct 30, 2006, at 1:15 PM, Joshua Warchol wrote: > Ezra Zygmuntowicz wrote: >> I'm not entirely certain what you are asking? Can you clarify a >> bit please? > Well, I've got a model that's currently doing the heavy lifting > populating entries in a few tables for a newsletter. I'm trying to > make it work with Backgroundrb now. So I've got a > Newsletter#deliver method that looks for subscribers and then > populates a notifications queue that another system uses to pump > out actual emails. > > Calling #deliver from a my PublishWorker is a snap, but I'm looking > at my options for reporting back its progress. Passing in the > results hash was my first idea. >> >> The rdoc is here: http://backgroundrb.rubyforge.org/ > I had the old README cached, sorry! Yes the results hash is really just a hash. But if you plan on getting results throughh the middleman and then passing them into your model the you should call MiddleMan.worker(session [:key]).results.to_hash before you pass them around on the rails side. The whole point of the results hash is to be able to pass hashes back and forth between rails and your workers. Take a look in the examples directory for all gthe stuff you need to set up an ajax progress bar. Cheers- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From ezmobius at gmail.com Mon Oct 30 16:33:11 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Mon, 30 Oct 2006 13:33:11 -0800 Subject: [Backgroundrb-devel] new BackgroundRB In-Reply-To: <005c01c6fc4e$8891e830$0a32330a@zeppelin> References: <005c01c6fc4e$8891e830$0a32330a@zeppelin> Message-ID: <24F2895C-D930-40D8-942E-41EA46F8ED0C@gmail.com> Hey Greg- Yes I am sorry, the new architecture uses fork and named pipes and a bunch of unix stuff to do its magick. Now you may be able to port it to qwork on windows, but I don't think it is possible :( I'm really sorry about this but I need this thing to be as robust and solid as it can be and in the end windows isn't compatible. Now you may be able to still work locally by installing cygwin and running the backgroundrb under cygwin. Cheers- -Ezra On Oct 30, 2006, at 10:09 AM, Rockin' wrote: > Ezra, > > > > Thanks for all your efforts on this. I love BackgroundRb ? it is > super useful. > > > > I?m interested in porting to the new version, but my primary > development platform is Windows. (Although I deploy to FC5.) > > I can?t afford to break my existing stuff right now, however, I do > think the new architecture will solve many of my current problems > and would like to make the switch. > > > > What will I run into on Windows? I don?t care if certain things > are more cumbersome, like start/stop/restart. But I do care if it > flat out doesn?t work because the way that process forking is done > is OS-specific or something. As I mentioned, I don?t have time for > that tangent right now, otherwise, I?d be happy to try it and even > help out with its development. > > > > What is your advice? > > > > Thanks! > > > > -greg > > > > ======================= > > > > Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! > > Complete rewrite. > > To: BackgroundRb ML , > > rubyonrails-talk at googlegroups.com > > Message-ID: > > Content-Type: text/plain; charset=WINDOWS-1252; delsp=yes; > > format=flowed > > > > Hello Folks- > > > > I am really happy to announce the release of BackgrounDRb > 0.2.0 . > > This is a complete rewrite and re-architecture of this codebase. I > > want to > > thank skaar for all of his hard work. I put out a call for help with > > the new > > architecture and skaar stepped up bigtime. I had already written > > the new > > scheduler and cron compatible syntax and moved to multi process > > architecture. But he took what I had that wasn't very whole and did a > > whopping 100 svn commits to the project! Big ups to skaar for this > > release! > > > > > > = BackgrounDRb 0.2.0 > > > > > > !! Note that this is in an entirely different svn repo then the old > > plugin. > > Make sure you remove the old before you install the new. > > > > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0/README > > http://svn.devjavu.com/backgroundrb/tags/release-0.2.0 (latest > release) > > http://svn.devjavu.com/backgroundrb/trunk (svn trunk) > > http://backgroundrb.devjavu.com (trac) > > http://backgroundrb.rubyforge.org (rdoc) > > > > Copyright (c) 2006 Ezra Zygmuntowicz and skaar[at]waste[dot]org > > > > == DISCLAIMER > > > > The 0.2.0 release of BackgrounDRb is a complete re-write of previous > > releases and is to be considered experimental, in-complete and in many > > respect untested. Our goal is to reach a release recommended for > > production use by 0.3.x. This said, this release is a more robust > > solution that the previous release. > > > > Also note that support for Windows is deprecated with this release of > > BackgrounDRb, sorry : ( . It may work under cygwin but we have not > > tried it. > > > > == Introduction > > > > BackgrounDRb is a ruby job server and scheduler. It main intent is > to be > > used with Ruby on Rails applications for offloading long running > tasks. > > Since a rails application blocks while servicing a request it is best > > to move long running tasks off into a background process that is > > divorced > > from the http request/response cycle. > > > > This new release of BackgrounDRb is also modular and can be used > without > > Rails. So any ruby program or framework can use it. > > > > == Technology Overview > > > > This 0.2.x branch of BackgrounDRb introduces a completely new > > architecture > > then the previous versions. Instead of a single process, multi > > threaded environment, > > the new system uses multi process with IPC to manage workers. So each > > of your > > workers will run in their own ruby interpreter. The interface that you > > use within rails remains mostly unchanged. > > > > The new architecture allows for a much more robust system. Each of > your > > worker classes get run in their own ruby interpreter. The main server > > process > > holds a collection of references to your running worker objects. This > > acts as > > the interface or MiddleMan that you use to interact with your remote > > jobs from > > rails or other ruby client code. When workers are spawned an > > interprocess > > communication channel is opened between the MiddleMan and the worker. > > There is a heartbeat setup between parent and child processes so > that if > > you kill the parent, all children processes are killed off the next > > time they try to > > hearbeat. All of this happens transparently to you as a user. > > > > The new release comes with a brand new job scheduler. You can use > > simple triggers like repeat_every or you can get more complex and use > > full cron compatible syntax. > > > > Workers can be scheduled by two built in 'trigger' types. A simple > > 'trigger' is specified with start, stop and interval: > > > > require 'active_support' > > MiddleMan.schedule_worker( > > :class => :example_worker, > > :job_key => :schedule_test, > > :worker_method => :other_method, > > :trigger_args => { > > :start => Time.now + 5.seconds, > > :end => Time.now + 10.minutes, > > :repeat_interval => 30.seconds > > } > > > > The cron trigger uses a similar syntax to cron found on UNIX systems: > > > > MiddleMan.schedule_worker( > > :class => simple_class, > > :job_key => :schedule_test, > > :worker_method => :arg_method, > > :worker_method_args => "my argument to arg_method", > > :trigger_args => "0 15 10 * * * *" > > ) > > > > > > Also note that when the server starts up, you will see 3 processes > > running. > > One of the is the MiddleMan server, one is the results worker and > one is > > the logger worker. When you do a logger.info("foo log!") in your > > workers, > > you are actually logging to the Logger worker. > > > > As you might imagine, this new way of managing multiple processes will > > scale a lot better then the multi threaded single process > version ;) But > > also be aware that there is still a thread pool in the middleman > > that you > > can control the size of. All this does is keep the plugin from > > spawning too > > many processes. It will allow however many workers you specify to > run at > > once and any more then that will just queue up and wit for their > turn to > > spawn. > > > > If you are already a user of the old BackgrounDRb please give the new > > version a tryout. It should not require very much work to port your > > worker > > classes to the new architecture. > > > > This is alpha software folks. It works for me? but we will not be > > held responsible > > if it ruins your day ;) > > > > -- Ezra Zygmuntowicz > > -- Lead Rails Evangelist > > -- ez at engineyard.com > > -- Engine Yard, Serious Rails Hosting > > -- (866) 518-YARD (9273) > > > > -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273) From skaar at waste.org Mon Oct 30 17:28:32 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 16:28:32 -0600 Subject: [Backgroundrb-devel] new BackgroundRB In-Reply-To: <24F2895C-D930-40D8-942E-41EA46F8ED0C@gmail.com> References: <005c01c6fc4e$8891e830$0a32330a@zeppelin> <24F2895C-D930-40D8-942E-41EA46F8ED0C@gmail.com> Message-ID: <20061030222832.GA23410@waste.org> * Ezra Zygmuntowicz (ezmobius at gmail.com) [061030 15:32]: > Hey Greg- > > > Yes I am sorry, the new architecture uses fork and named pipes and a > bunch of unix stuff to do its magick. Now you may be able to port it > to qwork on windows, but I don't think it is possible :( I'm really > sorry about this but I need this thing to be as robust and solid as > it can be and in the end windows isn't compatible. Now you may be > able to still work locally by installing cygwin and running the > backgroundrb under cygwin. that said: we use the Slave library which takes care of all of this for us, i.e. we have no encoded knowledge of the forks and the named pipes, which means that if someone wrote a library with a similar interface as the slave library, it wouldn't be all that much work to plug it in. Porting Slave to windows might now be straight forward however. /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From hutch at recursive.ca Mon Oct 30 17:34:05 2006 From: hutch at recursive.ca (Bob Hutchison) Date: Mon, 30 Oct 2006 17:34:05 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: References: Message-ID: <7D1C9561-037C-4090-9AA8-81196645691A@recursive.ca> On 30-Oct-06, at 3:01 PM, Ezra Zygmuntowicz wrote: > > On Oct 30, 2006, at 11:46 AM, Bob Hutchison wrote: > >> Hi, >> >> On 29-Oct-06, at 8:40 PM, Ezra Zygmuntowicz wrote: >> >>> I am really happy to announce the release of BackgrounDRb 0.2.0 . >> >> >> This is good news. >> >> Say you were running a few background processes and someone or >> something killed your main process. Is there any way of knowing >> which processes died because of heart failure when the server is >> restarted? >> >> Cheers, >> Bob >> > \ > > Hey bob- > > Thats a good question. The current answer is no, if the middleman > server gets killed, all workers will get killed as well. But > looking to the future we will try to have a way to have dual > middlemen so they can watch each other and bring the other one up > if it dies. And maybe a way to serialize the current jobs to disk > like journalling so if things crash they could be restored easily. > Please do give me your thoughts on this. We are going to be working > on this thing to make it awesome by 0.3.0 so if there are things > you want to see then speak up. I'm using slave 0.2.0 in my application right now. I'm using it to run some potentially very long running processes. In my case, it doesn't really matter if they fail, but if they do I'd like know about it so I can tell the user and possibly re-start them. At the moment if that scenario occurs then I can only tell by a time out. If I wasn't so lazy I could do something smarter, like mark the fact that the process is running somewhere in the DB (which might really be the filesystem), maybe with a pid. When I check on progress I can see if slave/backgroundrb has any knowledge of that process, if it doesn't then I can assume that the server failed and as re-started. If slave/backgroundrb knows of the process but the process no longer exists then I can assume that it failed in an unpleasant way. Anyway, that's the kind of thing I had in mind. Cheers, Bob > > Thanks > -- Ezra Zygmuntowicz-- Lead Rails Evangelist > -- ez at engineyard.com > -- Engine Yard, Serious Rails Hosting > -- (866) 518-YARD (9273) > > ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby -- From skaar at waste.org Mon Oct 30 18:32:32 2006 From: skaar at waste.org (skaar) Date: Mon, 30 Oct 2006 17:32:32 -0600 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: <7D1C9561-037C-4090-9AA8-81196645691A@recursive.ca> References: <7D1C9561-037C-4090-9AA8-81196645691A@recursive.ca> Message-ID: <20061030233232.GC23410@waste.org> > I'm using slave 0.2.0 in my application right now. I'm using it to > run some potentially very long running processes. In my case, it > doesn't really matter if they fail, but if they do I'd like know > about it so I can tell the user and possibly re-start them. At the > moment if that scenario occurs then I can only tell by a time out. If > I wasn't so lazy I could do something smarter, like mark the fact > that the process is running somewhere in the DB (which might really > be the filesystem), maybe with a pid. When I check on progress I can > see if slave/backgroundrb has any knowledge of that process, if it > doesn't then I can assume that the server failed and as re-started. > If slave/backgroundrb knows of the process but the process no longer > exists then I can assume that it failed in an unpleasant way. the crude way of doing this right now is to use a singleton/named worker - and call new_worker(:class => :foo_worker, :job_key => :job_name) every time you access it. If the worker exist, you will just get the key back, if it doesn't the server will create the worker, and then return the key. You can the same way, with generated keys, on the client side initially call new_worker without the :job_key argument and store the key - if you next time have a key, you can call new_worker with the job_key that was generated for you and that you, say, have stored in the users session. /skaar -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From chris at etechdata.com.au Mon Oct 30 22:49:45 2006 From: chris at etechdata.com.au (Chris H) Date: Tue, 31 Oct 2006 13:49:45 +1000 Subject: [Backgroundrb-devel] 2 Background processes Message-ID: <4546C7D9.5080900@etechdata.com.au> Is it possible to have 2 seperate instances of backgroundrb running on the same machine? I'm having to develop on the same box as production and I can only get backgroundrb to work for one or the other. Thanks in advance, Chris. From hutch at recursive.ca Tue Oct 31 07:29:17 2006 From: hutch at recursive.ca (Bob Hutchison) Date: Tue, 31 Oct 2006 07:29:17 -0500 Subject: [Backgroundrb-devel] [ANN] BackgrounDRb 0.2.0 Release! Complete rewrite. In-Reply-To: <20061030233232.GC23410@waste.org> References: <7D1C9561-037C-4090-9AA8-81196645691A@recursive.ca> <20061030233232.GC23410@waste.org> Message-ID: On 30-Oct-06, at 6:32 PM, skaar wrote: >> I'm using slave 0.2.0 in my application right now. I'm using it to >> run some potentially very long running processes. In my case, it >> doesn't really matter if they fail, but if they do I'd like know >> about it so I can tell the user and possibly re-start them. At the >> moment if that scenario occurs then I can only tell by a time out. If >> I wasn't so lazy I could do something smarter, like mark the fact >> that the process is running somewhere in the DB (which might really >> be the filesystem), maybe with a pid. When I check on progress I can >> see if slave/backgroundrb has any knowledge of that process, if it >> doesn't then I can assume that the server failed and as re-started. >> If slave/backgroundrb knows of the process but the process no longer >> exists then I can assume that it failed in an unpleasant way. > > the crude way of doing this right now is to use a singleton/named > worker > - and call new_worker(:class => :foo_worker, :job_key => :job_name) > every time you access it. If the worker exist, you will just get > the key > back, if it doesn't the server will create the worker, and then return > the key. > > You can the same way, with generated keys, on the client side > initially > call new_worker without the :job_key argument and store the key - > if you > next time have a key, you can call new_worker with the job_key that > was > generated for you and that you, say, have stored in the users session. Thanks, I can probably do something with that. So much for being lazy :-) Cheers, Bob > > /skaar > > > -- > ---------------------------------------------------------------------- > |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n > | | >=========== W.A.S.T.E. | genarratologies > |/|/ (_) is the wisdom | skaar at waste.org > ---------------------------------------------------------------------- ---- Bob Hutchison -- blogs at Recursive Design Inc. -- Raconteur -- xampl for Ruby -- From skaar at waste.org Tue Oct 31 10:18:12 2006 From: skaar at waste.org (skaar) Date: Tue, 31 Oct 2006 09:18:12 -0600 Subject: [Backgroundrb-devel] 2 Background processes In-Reply-To: <4546C7D9.5080900@etechdata.com.au> References: <4546C7D9.5080900@etechdata.com.au> Message-ID: <20061031151812.GA27635@waste.org> you can certainly do do this my setting up config/backgroundrb.yml with different ports for the two instances. You can also swtich to use domain sockets, which use the :host and :port values to construct a unique file name: /srv/rails_apps/production/conf/backgroundrb.yml: :host: localhost :port: 2001 :protocol: drbunix /srv/rails_apps/development/conf/backgroundrb.yml: :host: localhost :port: 2002 :protocol: drbunix you then need to start the two backgroundrb servers independently: cd /srv/rails_apps/production && ./script/backgroundrb start cd /srv/rails_apps/development && ./script/backgroundrb start /skaar * Chris H (chris at etechdata.com.au) [061030 23:22]: > Is it possible to have 2 seperate instances of backgroundrb running on > the same machine? > > I'm having to develop on the same box as production and I can only get > backgroundrb to work > for one or the other. > > Thanks in advance, > Chris. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- ---------------------------------------------------------------------- |\|\ where in the | s_u_b_s_t_r_u_c_t_i_o_n | | >=========== W.A.S.T.E. | genarratologies |/|/ (_) is the wisdom | skaar at waste.org ---------------------------------------------------------------------- From ezmobius at gmail.com Tue Oct 31 19:00:19 2006 From: ezmobius at gmail.com (Ezra Zygmuntowicz) Date: Tue, 31 Oct 2006 16:00:19 -0800 Subject: [Backgroundrb-devel] Scheduler not useable yet Message-ID: Hey there Gang- The scheduler in the new BackgrounDRb is not really useable yet. So please avoid it for a few days until we get the kinks worked out. Thanks- -- Ezra Zygmuntowicz -- Lead Rails Evangelist -- ez at engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)