From andrew at andrewtimberlake.com Tue Apr 1 16:39:28 2008 From: andrew at andrewtimberlake.com (Andrew Timberlake) Date: Tue, 1 Apr 2008 22:39:28 +0200 Subject: [Backgroundrb-devel] Testing workers Message-ID: <01d201c89438$7d3d54f0$77b7fed0$@com> Could someone help me with some information on how to handle functional tests that rely on Backgroundrb? I didn't find anything on the Backgroundrb website not on a couple of Google searches. Links would be fine. I keep getting the following error in my functional tests: "Error while connecting to the backgroundrb server" Thanks Andrew Timberlake andrew at andrewtimberlake.com 082 415 8283 skype: andrewtimberlake "I have never let my schooling interfere with my education." --Mark Twain From gethemant at gmail.com Wed Apr 2 04:20:20 2008 From: gethemant at gmail.com (hemant kumar) Date: Wed, 02 Apr 2008 13:50:20 +0530 Subject: [Backgroundrb-devel] Testing workers In-Reply-To: <01d201c89438$7d3d54f0$77b7fed0$@com> References: <01d201c89438$7d3d54f0$77b7fed0$@com> Message-ID: <1207124420.29419.7.camel@shire> On Tue, 2008-04-01 at 22:39 +0200, Andrew Timberlake wrote: > Could someone help me with some information on how to handle functional > tests that rely on Backgroundrb? > I didn't find anything on the Backgroundrb website not on a couple of Google > searches. > Links would be fine. > > I keep getting the following error in my functional tests: > "Error while connecting to the backgroundrb server" Did you read this: http://backgroundrb.rubyforge.org/workers/index.html We have some information at the bottom page. From dmarkow at gmail.com Wed Apr 2 10:30:52 2008 From: dmarkow at gmail.com (Dylan Markow) Date: Wed, 2 Apr 2008 07:30:52 -0700 Subject: [Backgroundrb-devel] Specifying environment in backgroundrb.yml Message-ID: <24ad6d8a0804020730r29eb1a49kb1b930fc8fb6b819@mail.gmail.com> It seems that regardless of what environment I specify in my backgroundrb.yml file, it always uses development mode unless I pass "-e production" to the "script/backgroundrb start" command. >From looking at bdrb_config.rb, it looks like immediately after loading the config file it assigns whatever was passed on the command line (which defaults to development if nothing is specified) to config[:backgroundrb][:environment] I can easily set up my deployment file to just run with the -e production line, but it seems misleading that the option is even available in the backgroundrb.yml file if it doesn't affect anything. From sqware.de at gmail.com Wed Apr 2 17:06:45 2008 From: sqware.de at gmail.com (sQware.de) Date: Wed, 02 Apr 2008 23:06:45 +0200 Subject: [Backgroundrb-devel] scheduling worker methods in rails Message-ID: <47F3F565.7060309@googlemail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, i'm trying desperately to schedule some worker methods within rails 2.0.2. The problem is, both methods don't even get called by BackgrounDRb. my background.yml: > --- > :backgroundrb: > :port: 11006 > :ip: 0.0.0.0 > --- > :schedules: > :server_worker: > :update: > :trigger_args: 1 * * * * * * > :monitoring_worker: > :check_services: > :trigger_args: */10 * * * * * * Both Worker gets automatically started if BackgroundRb starts. I want the server_worker to start an update-method every minute, the monitoring_worker to check every 10 seconds if the server is still running. Between two calls both worker are stopped (or sleeping), the creation and the different tasks are shorter than the interval (if that matters) At another point i'm trying to schedule a Task via a MiddelMan.new_worker within rails: > MiddleMan.new_worker(:worker => :account_worker, :job_key => id, :data => id,\ :schedule => { :calculate => { :trigger_args => "*/5 * * * * * *", :data => id \ }}) With this call, the method don't even get executed (tested with logger.info as first line to see if a message gets written in the log-file). > def calculate(id = nil) > logger.info("#{Time.now}: calculate") > end The creation of a worker instance works fine, only the scheduled Part dont gets executed. So my assumption is, that the trigger_args are wrong, but i checked everything a thousend times and maybe i dont see the wood for the trees (maybe I have to sleep a litte bit more ;) ) Even the backgroundrb_11006_debug.log don't say anything i can use to find the error: > 00000012{ > : type:start_worker: > schedule{:calculate{:trigger_args"*/5 * * * * * *: data"6: > job_key@ ; > @ > {:data=>"6", :worker=>:account_worker, :type=>:start_worker, :job_key=>"6",\ :schedule=>{:calculate=>{:data=>"6", :trigger_args=>"*/5 * * * * * *"}}} Everything happens on Ubuntu Linux 7.10 > ak at ak-laptop:~$ uname -r > 2.6.22-14-generic with BackgroundRb 1.0.3, rails: > ak at ak-laptop:~$ rails -v > Rails 2.0.2 and Ruby: > ak at ak-laptop:~$ ruby -v > ruby 1.8.6 (2007-06-07 patchlevel 36) [i486-linux] I hope someone can point me to the source of my Problems. Thanks Andi P.S. excuse my poor english, i hope you understand my needs ;) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH8/VlW5GYaw3DxvwRAneLAKDLWc1dv8S9OMwwNF1li71sHEOMvgCfTCce QmdphYGdVX28T0bAA1Qbxbc= =C2Cb -----END PGP SIGNATURE----- From gethemant at gmail.com Wed Apr 2 19:35:24 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 03 Apr 2008 05:05:24 +0530 Subject: [Backgroundrb-devel] Specifying environment in backgroundrb.yml In-Reply-To: <24ad6d8a0804020730r29eb1a49kb1b930fc8fb6b819@mail.gmail.com> References: <24ad6d8a0804020730r29eb1a49kb1b930fc8fb6b819@mail.gmail.com> Message-ID: <1207179324.29419.19.camel@shire> On Wed, 2008-04-02 at 07:30 -0700, Dylan Markow wrote: > It seems that regardless of what environment I specify in my > backgroundrb.yml file, it always uses development mode unless I pass > "-e production" to the "script/backgroundrb start" command. > > >From looking at bdrb_config.rb, it looks like immediately after > loading the config file it assigns whatever was passed on the command > line (which defaults to development if nothing is specified) to > config[:backgroundrb][:environment] > > I can easily set up my deployment file to just run with the -e > production line, but it seems misleading that the option is even > available in the backgroundrb.yml file if it doesn't affect anything. Some regression has crept in. Can you file this as bug report? Also, if possible can you submit this bug report this appropriate testcase? From ramkaka at gmail.com Thu Apr 3 05:38:58 2008 From: ramkaka at gmail.com (Ram Ravichandran) Date: Thu, 3 Apr 2008 05:38:58 -0400 Subject: [Backgroundrb-devel] remote workers Message-ID: Hi, I am kind of lost on how to configure the backgroundrb workers to run from a remote host? Suppose I have my main rails app in 1.2.3.4, and I want my backgroundrb workers to run from a machine with ip 5.6.7.8, what do I put in my backgroundrb.yml file? How will my workers know which database server to load the models from? Thanks, Ram From gethemant at gmail.com Thu Apr 3 06:28:38 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 03 Apr 2008 15:58:38 +0530 Subject: [Backgroundrb-devel] remote workers In-Reply-To: References: Message-ID: <1207218518.14226.3.camel@shire> When you are configuring BackgrounDRb to run from remote servers, you will also need a snapshot of your rails application on that machine as well. On Thu, 2008-04-03 at 05:38 -0400, Ram Ravichandran wrote: > Hi, > > I am kind of lost on how to configure the backgroundrb workers to run > from a remote host? > Suppose I have my main rails app in 1.2.3.4, and I want my > backgroundrb workers to run from a machine with ip 5.6.7.8, what do I > put > in my backgroundrb.yml file? How will my workers know which database > server to load the models from? > > Thanks, > > Ram > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From gethemant at gmail.com Thu Apr 3 06:30:15 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 03 Apr 2008 16:00:15 +0530 Subject: [Backgroundrb-devel] scheduling worker methods in rails In-Reply-To: <47F3F565.7060309@googlemail.com> References: <47F3F565.7060309@googlemail.com> Message-ID: <1207218615.14226.6.camel@shire> On Wed, 2008-04-02 at 23:06 +0200, sQware.de wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > i'm trying desperately to schedule some worker methods within rails 2.0.2. The > problem is, both methods don't even get called by BackgrounDRb. > > my background.yml: > > > --- > > :backgroundrb: > > :port: 11006 > > :ip: 0.0.0.0 > > --- > > :schedules: > > :server_worker: > > :update: > > :trigger_args: 1 * * * * * * > > :monitoring_worker: > > :check_services: > > :trigger_args: */10 * * * * * * > > Both Worker gets automatically started if BackgroundRb starts. I want the > server_worker to start an update-method every minute, the monitoring_worker to > check every 10 seconds if the server is still running. > Between two calls both worker are stopped (or sleeping), the creation and the > different tasks are shorter than the interval (if that matters) > > At another point i'm trying to schedule a Task via a MiddelMan.new_worker within > rails: > > > MiddleMan.new_worker(:worker => :account_worker, :job_key => id, :data => id,\ > :schedule => { :calculate => { :trigger_args => "*/5 * * * * * *", :data => id \ }}) > > With this call, the method don't even get executed (tested with logger.info as > first line to see if a message gets written in the log-file). > > > def calculate(id = nil) > > logger.info("#{Time.now}: calculate") > > end > > The creation of a worker instance works fine, only the scheduled Part dont gets > executed. > So my assumption is, that the trigger_args are wrong, but i checked everything a > thousend times and maybe i dont see the wood for the trees (maybe I have to > sleep a litte bit more ;) ) > > > Even the backgroundrb_11006_debug.log don't say anything i can use to find the > error: > > 00000012{ > > : type:start_worker: > > schedule{:calculate{:trigger_args"*/5 * * * * * *: data"6: > > job_key@ ; > > @ > > {:data=>"6", :worker=>:account_worker, :type=>:start_worker, :job_key=>"6",\ > :schedule=>{:calculate=>{:data=>"6", :trigger_args=>"*/5 * * * * * *"}}} > > Everything happens on Ubuntu Linux 7.10 > > ak at ak-laptop:~$ uname -r > > 2.6.22-14-generic > with BackgroundRb 1.0.3, > > rails: > > ak at ak-laptop:~$ rails -v > > Rails 2.0.2 > > and Ruby: > > ak at ak-laptop:~$ ruby -v > > ruby 1.8.6 (2007-06-07 patchlevel 36) [i486-linux] > > I hope someone can point me to the source of my Problems. > > Thanks > > Andi > If you don't mind, I will be only be able to look in this in detail when I reach home. Your configuration looks alright. Also, are you telling us that, even plain scheduled worker scheduled through config file are not being scheduled? From sqware.de at gmail.com Thu Apr 3 13:02:41 2008 From: sqware.de at gmail.com (sQware.de) Date: Thu, 03 Apr 2008 19:02:41 +0200 Subject: [Backgroundrb-devel] scheduling worker methods in rails In-Reply-To: <1207218615.14226.6.camel@shire> References: <47F3F565.7060309@googlemail.com> <1207218615.14226.6.camel@shire> Message-ID: <47F50DB1.6050304@googlemail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hemant kumar schrieb: > Also, are you telling us that, even plain scheduled worker scheduled > through config file are not being scheduled? Right, with the background.rb-file cited above the schedules dont work. The Server- and the Monitoring- Workers get automatically started when i start backgroundrb. The create-methods are working without problems ... also if i call the update or check_services methods from within the according create-method everything works fine ... only the triggers dont fire ... I updated all gems (sudo gem update --system), doublechecked if all necessary dependencies are installed (chronic and packet) ... everything ok ... I also tested a completly new worker with following settings: backgroundrb.yml: > --- > :backgroundrb: > :port: 11006 > :ip: 0.0.0.0 > --- > :schedules: > :hello_worker: > :worker_method: say > :trigger_args: 0 * * * * * * hello_worker: > class HelloWorker < BackgrounDRb::MetaWorker > set_worker_name :hello_worker > def create(args = nil) > # this method is called, when worker is loaded for the first time > logger.info("#{Time.now}. worker started") > end > def say(args = "hello") > logger.info("#{args} again, its #{Time.now}") > end > end in backgroundrb_11006.log: > Thu Apr 03 18:19:35 +0200 2008. worker started > hello_worker started > Schedules for worker loaded thats all ... so i think, the trigger don't gets fired. Do you have a clue, where i might have an error, or is there an error in trunk? Unfortunaly there is no error logged in any of the logfiles, so i can not hunt it ;) many thanks for helping Andi -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH9Q2xW5GYaw3DxvwRAiFvAJ0WALHwWy+sPClnSTHC5DIFfNE+HgCgkNDl 4gmpRcL392g2qQt7zh+osro= =nfoE -----END PGP SIGNATURE----- From kitplummer at gmail.com Thu Apr 3 13:15:11 2008 From: kitplummer at gmail.com (Kit Plummer) Date: Thu, 3 Apr 2008 10:15:11 -0700 Subject: [Backgroundrb-devel] Two Rails Apps, One Backgroundrb Server? Message-ID: <5A798449-19F2-4157-B3A0-A9EED77079EE@gmail.com> This might be a really simple question, with a simple solution. I have two rails apps running on the same box. Right now I am starting two Backgroundrb servers (11006 and 11007). Wondering if it might be possible to run only one and have the second app point to the first app's server? TIA, Kit From chris at thenumber.biz Thu Apr 3 12:44:29 2008 From: chris at thenumber.biz (Chris Nohr) Date: Thu, 03 Apr 2008 09:44:29 -0700 Subject: [Backgroundrb-devel] ActiveRecord::Observers? Message-ID: <37D0AC49-069D-4E2F-9706-2055811B21D1@thenumber.biz> wow, backgroundrb is great. in about 3 hours we had converted all our daemons over from daemon-generator to backgroundrb, with much more control and power. it's awesome. the only problem is that we can't seem to get the observers to load. in my worker, I go like this: def create(args = nil) ActiveRecord::Base.observers = :event_observer EventObserver.instance end which indicates it's loaded, but it doesn't seem to trigger itself. does anybody have any known issues with observers not observing? also, I'm a bit confused about how/when it loads environment files - does it load the environment/production.rb file if i'm in production mode? chris. From rogboone at yahoo.com Thu Apr 3 14:25:58 2008 From: rogboone at yahoo.com (Roggie Boone) Date: Thu, 3 Apr 2008 11:25:58 -0700 (PDT) Subject: [Backgroundrb-devel] Packet error on startup Message-ID: <870590.67995.qm@web32505.mail.mud.yahoo.com> Hi, New BackgrounDrb user struggling for a couple of days to get things up an going. System: Ruby 1.8.6 Packet 0.1.5 chronic 0.2.3 BackgrounDrb 1.0.3 (downloaded 4/02/08 and 4/03/08) I installed the packet and chronic gems as per instructions. I used git to download BDrb as per instructions and ran the rake setup script. When I run script/backgroundrb from my rails app main directory, I get the following error: /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:145:in `schedule_write': You have a nil object when you didn't expect it! (NoMethodError) The error occurred while evaluating nil.instance from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:57:in `write_and_schedule' from /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/whiny_nil.rb:35:in `each_with_index' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `each' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `each_with_index' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:51:in `write_and_schedule' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb:72:in `dump_object' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:31:in `send_data' from /home/test/work/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:284:in `register_status' ... 9 levels... from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in `run' from /home/test/work/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 ------------------- For added background, I installed BDrb from svn earlier and setup a simple worker to count to 100,000 and in my Rails app had an ajax view to issue the "ask_status" on the worker and update the web view with the current location of the "count". I was able to get the ajax view to update automatically once, and then it no longer update as if the worker was hung. I saw in the archives that someone else had a similar problem and it was recommended to download BDrb from the git repository as it had a fix that might help. I did so (and removed the old background script that was setup by the svn version and reran the setup script). But the git version does not appeare to even start the background server due to the errors above. Any help is appreciated. Roggie --------------------------------- You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080403/cf395f66/attachment-0001.html From jonathan at column3.net Thu Apr 3 22:30:06 2008 From: jonathan at column3.net (Jonathan Donaldson) Date: Thu, 3 Apr 2008 22:30:06 -0400 Subject: [Backgroundrb-devel] Can't get backgroundrb to work with rails plugin 'batched_sql_fu' Message-ID: <1CB150DA-8720-492C-969B-F120E965498A@column3.net> Trying to use backgroundrb to process and insert a large list of bulk records, but when I try to use it with the batched_sql_fu plugin installed in vendor/plugins, I get the following error: ---SNIP--- /Users/jdonalds/Sites/3dial_ads/config/../vendor/plugins/ batched_sql_fu/lib/batched_sql_fu/active_record.rb:22:in `alias_insert_sql_to_batched_version!': undefined method `insert_sql' for class `ActiveRecord::ConnectionAdapters::MysqlAdapter' (NameError) from /Users/jdonalds/Sites/3dial_ads/config/../vendor/plugins/ batched_sql_fu/lib/batched_sql_fu/active_record.rb:8:in `insert_by_batch' from /Users/jdonalds/Sites/3dial_ads/lib/workers/ process_list_worker.rb:249:in `process_leads' from /Users/jdonalds/Sites/3dial_ads/vendor/plugins/backgroundrb/ server/lib/meta_worker.rb:235:in `send' from /Users/jdonalds/Sites/3dial_ads/vendor/plugins/backgroundrb/ server/lib/meta_worker.rb:235:in `process_request' from /Users/jdonalds/Sites/3dial_ads/vendor/plugins/backgroundrb/ server/lib/meta_worker.rb:219:in `receive_data' from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/ packet_worker.rb:53:in `receive_internal_data' from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `call' from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `extract' ... 24 levels... from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:21:in `run' from /Users/jdonalds/Sites/3dial_ads/vendor/plugins/backgroundrb/ server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 ---SNIP-- The code in the worker works as expected when called directly.... Jonathan Donaldson: CTO at Column3, SMB VoIP Communications ASP contact | jonathan at column3.net - 617-507-6707 | yahoo - jonathan.donaldson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080403/7b5c5153/attachment.html From ramkaka at gmail.com Thu Apr 3 23:00:50 2008 From: ramkaka at gmail.com (Ram Ravichandran) Date: Thu, 3 Apr 2008 23:00:50 -0400 Subject: [Backgroundrb-devel] remote workers In-Reply-To: <1207218518.14226.3.camel@shire> References: <1207218518.14226.3.camel@shire> Message-ID: The IP in the yml file is the one with the workers. right? Thanks, Ram On Thu, Apr 3, 2008 at 6:28 AM, hemant kumar wrote: > When you are configuring BackgrounDRb to run from remote servers, you > will also need a snapshot of your rails application on that machine as > well. > > > > > > > On Thu, 2008-04-03 at 05:38 -0400, Ram Ravichandran wrote: > > Hi, > > > > I am kind of lost on how to configure the backgroundrb workers to run > > from a remote host? > > Suppose I have my main rails app in 1.2.3.4, and I want my > > backgroundrb workers to run from a machine with ip 5.6.7.8, what do I > > put > > in my backgroundrb.yml file? How will my workers know which database > > server to load the models from? > > > > Thanks, > > > > Ram > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > From dmarkow at gmail.com Thu Apr 3 23:57:09 2008 From: dmarkow at gmail.com (Dylan Markow) Date: Thu, 3 Apr 2008 20:57:09 -0700 Subject: [Backgroundrb-devel] First call to worker method doesn't work Message-ID: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> I have a worker as follows: class SampleWorker < BackgrounDRb::MetaWorker set_worker_name :sample_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def my_method # Deliver test e-mail message Notifications.deliver_message(1, "DM") end end I have a rails controller that calls the following code: worker = MiddleMan.worker(:sample_worker) worker.my_method The problem is that the first time I load up the page for this action, the worker method "my_method" doesn't seem to ever get called. There is zero output from any of the backgroundrb log files. If I then reload the page, it works fine (and continues to work fine until backgroundrb is restarted). This happens whether I'm in dev. or prod. mode. From gethemant at gmail.com Fri Apr 4 00:52:39 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 04 Apr 2008 10:22:39 +0530 Subject: [Backgroundrb-devel] remote workers In-Reply-To: References: <1207218518.14226.3.camel@shire> Message-ID: <1207284759.4190.9.camel@shire> On Thu, 2008-04-03 at 23:00 -0400, Ram Ravichandran wrote: > The IP in the yml file is the one with the workers. right? Yes that would be the ip. From dmarkow at gmail.com Fri Apr 4 01:12:36 2008 From: dmarkow at gmail.com (Dylan Markow) Date: Thu, 3 Apr 2008 22:12:36 -0700 Subject: [Backgroundrb-devel] First call to worker method doesn't work In-Reply-To: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> References: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> Message-ID: <24ad6d8a0804032212h2732da01hc448d19ab07bbf95@mail.gmail.com> On Thu, Apr 3, 2008 at 8:57 PM, Dylan Markow wrote: > I have a worker as follows: > > class SampleWorker < BackgrounDRb::MetaWorker > set_worker_name :sample_worker > def create(args = nil) > # this method is called, when worker is loaded for the first time > end > > def my_method > # Deliver test e-mail message > Notifications.deliver_message(1, "DM") > end > end > > I have a rails controller that calls the following code: > > worker = MiddleMan.worker(:sample_worker) > worker.my_method > > The problem is that the first time I load up the page for this action, > the worker method "my_method" doesn't seem to ever get called. There > is zero output from any of the backgroundrb log files. If I then > reload the page, it works fine (and continues to work fine until > backgroundrb is restarted). > > This happens whether I'm in dev. or prod. mode. > To clarify, if I restart rails AND backgroundrb at the same time, everything works fine. it's only if backgroundrb needs to restart on its own for some reason. It looks like when the backgroundrb process is restarted, rails doesn't realize this as it still sees port 11006 open. So my first request to the page results in the rails plugin attempting to write data to the socket. I'm assuming this screws something up, as upon the second attempt at loading my page, "rescue Errno::EPIPE" at line 64 of backgroundrb.rb is called, thus re-establishing the connection. I'm assuming this problem may be more closely related to the packet gem? (Not sure if there is a trac repo for packet). From dmarkow at gmail.com Fri Apr 4 01:24:38 2008 From: dmarkow at gmail.com (Dylan Markow) Date: Thu, 3 Apr 2008 22:24:38 -0700 Subject: [Backgroundrb-devel] First call to worker method doesn't work In-Reply-To: <24ad6d8a0804032212h2732da01hc448d19ab07bbf95@mail.gmail.com> References: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> <24ad6d8a0804032212h2732da01hc448d19ab07bbf95@mail.gmail.com> Message-ID: <24ad6d8a0804032224r62100924h3b62f7ea9bdde28a@mail.gmail.com> On Thu, Apr 3, 2008 at 10:12 PM, Dylan Markow wrote: > > On Thu, Apr 3, 2008 at 8:57 PM, Dylan Markow wrote: > > I have a worker as follows: > > > > class SampleWorker < BackgrounDRb::MetaWorker > > set_worker_name :sample_worker > > def create(args = nil) > > # this method is called, when worker is loaded for the first time > > end > > > > def my_method > > # Deliver test e-mail message > > Notifications.deliver_message(1, "DM") > > end > > end > > > > I have a rails controller that calls the following code: > > > > worker = MiddleMan.worker(:sample_worker) > > worker.my_method > > > > The problem is that the first time I load up the page for this action, > > the worker method "my_method" doesn't seem to ever get called. There > > is zero output from any of the backgroundrb log files. If I then > > reload the page, it works fine (and continues to work fine until > > backgroundrb is restarted). > > > > This happens whether I'm in dev. or prod. mode. > > > > To clarify, if I restart rails AND backgroundrb at the same time, > everything works fine. it's only if backgroundrb needs to restart on > its own for some reason. It looks like when the backgroundrb process > is restarted, rails doesn't realize this as it still sees port 11006 > open. So my first request to the page results in the rails plugin > attempting to write data to the socket. I'm assuming this screws > something up, as upon the second attempt at loading my page, "rescue > Errno::EPIPE" at line 64 of backgroundrb.rb is called, thus > re-establishing the connection. I'm assuming this problem may be more > closely related to the packet gem? (Not sure if there is a trac repo > for packet). > One last note: If you have 5 mongrels running of your rails app, then the worker will fail to fire 5 times (once for each mongrel), though not always in a row (typically my second request works, but then my 3rd or 4th, etc., won't) As it stands now, I'm setting up my deployment files to always restart backgroundrb BEFORE rails to make sure that once each mongrel fires up, they each have a valid instance of backgroundrb to point to. From gethemant at gmail.com Fri Apr 4 04:25:55 2008 From: gethemant at gmail.com (hemant) Date: Fri, 4 Apr 2008 13:55:55 +0530 Subject: [Backgroundrb-devel] First call to worker method doesn't work In-Reply-To: <24ad6d8a0804032224r62100924h3b62f7ea9bdde28a@mail.gmail.com> References: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> <24ad6d8a0804032212h2732da01hc448d19ab07bbf95@mail.gmail.com> <24ad6d8a0804032224r62100924h3b62f7ea9bdde28a@mail.gmail.com> Message-ID: On 4/4/08, Dylan Markow wrote: > On Thu, Apr 3, 2008 at 10:12 PM, Dylan Markow wrote: > > > > On Thu, Apr 3, 2008 at 8:57 PM, Dylan Markow wrote: > > > I have a worker as follows: > > > > > > class SampleWorker < BackgrounDRb::MetaWorker > > > set_worker_name :sample_worker > > > def create(args = nil) > > > # this method is called, when worker is loaded for the first time > > > end > > > > > > def my_method > > > # Deliver test e-mail message > > > Notifications.deliver_message(1, "DM") > > > end > > > end > > > > > > I have a rails controller that calls the following code: > > > > > > worker = MiddleMan.worker(:sample_worker) > > > worker.my_method > > > > > > The problem is that the first time I load up the page for this action, > > > the worker method "my_method" doesn't seem to ever get called. There > > > is zero output from any of the backgroundrb log files. If I then > > > reload the page, it works fine (and continues to work fine until > > > backgroundrb is restarted). > > > > > > This happens whether I'm in dev. or prod. mode. > > > > > > > To clarify, if I restart rails AND backgroundrb at the same time, > > everything works fine. it's only if backgroundrb needs to restart on > > its own for some reason. It looks like when the backgroundrb process > > is restarted, rails doesn't realize this as it still sees port 11006 > > open. So my first request to the page results in the rails plugin > > attempting to write data to the socket. I'm assuming this screws > > something up, as upon the second attempt at loading my page, "rescue > > Errno::EPIPE" at line 64 of backgroundrb.rb is called, thus > > re-establishing the connection. I'm assuming this problem may be more > > closely related to the packet gem? (Not sure if there is a trac repo > > for packet). > > > > > > One last note: If you have 5 mongrels running of your rails app, then > the worker will fail to fire 5 times (once for each mongrel), though > not always in a row (typically my second request works, but then my > 3rd or 4th, etc., won't) > > As it stands now, I'm setting up my deployment files to always restart > backgroundrb BEFORE rails to make sure that once each mongrel fires > up, they each have a valid instance of backgroundrb to point to. > I am tracking down this problem and its related to a bug in ruby itself (See my post on ruby-core mailing list). In the meanwhile, we can switch back to starting new connection on each request as a workaround. From normelton at gmail.com Fri Apr 4 14:38:46 2008 From: normelton at gmail.com (Norman Elton) Date: Fri, 4 Apr 2008 14:38:46 -0400 Subject: [Backgroundrb-devel] Bug in meta_worker.rb Message-ID: <6b3a7f010804041138g14c8a995ve0ae3d3d7f87b4e9@mail.gmail.com> I propose changing meta_worker.rb line 240 from: result = "dummy_result" unless result to result = "dummy_result" unless defined?(result) If my worker is returning false, I get "dummy_result" instead. Thanks! Norman From todd at rubidine.com Tue Apr 8 13:56:29 2008 From: todd at rubidine.com (Todd Willey) Date: Tue, 8 Apr 2008 17:56:29 +0000 Subject: [Backgroundrb-devel] Lingering status? Message-ID: <25e0753e0804081056j44a9043ar8daedd2cf82f545a@mail.gmail.com> Is it expected that MiddleMan.ask_stats will return a status for a worker that has been MiddleMan.delete_worker'ed? The worker isn't showing up in MiddleMan.query_all_workers, so it seems that is shouldn't have a status either. Is there a way to define a callback that is called at deletion, to reset the status, or do I need to write my own method to reset the status before I delete the worker? TIA -todd[1] -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080408/acb1ae5f/attachment-0001.html From paul.kmiec at appfolio.com Tue Apr 8 14:23:14 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Tue, 8 Apr 2008 11:23:14 -0700 Subject: [Backgroundrb-devel] cron trigger problem Message-ID: <674a03dc0804081123i2b887926h3815ae22d7350d9e@mail.gmail.com> Hi, I am playing around with backgroundrb 1.0.3. I have the following backgroundrb.yml, --- :backgroundrb: :port: 11006 :ip: 0.0.0.0 :schedules: :task_worker: :run: :trigger_args: */10 * * * * * My task worker is just, class TaskWorker < BackgrounDRb:MetaWorker set_worker_name :task_worker def run logger.info "begin" sleep 60 logger.info "end" end end After seeing a few begin's and end's in the log, I get the following exception: /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/cron_trigger.rb:119:in `parse': wrong number of arguments (2 for 1) (ArgumentError) from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/cron_trigger.rb:119:in `fire_after_time' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/cron_trigger.rb:21:in `loop' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/cron_trigger.rb:21:in `fire_after_time' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:336:in `check_for_timer_events' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:325:in `each' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:325:in `check_for_timer_events' from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:125:in `start_reactor' from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/packet_core.rb:124:in `loop' ... 6 levels... from /Library/Ruby/Gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in `run' from /Users/pkmiec/src/propertyadmin/trunk/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 Line 119 is, next_min = Chronic.parse("next minute",:now => current_time) Not sure what is going on since cron trigger does work correctly few times before this error occurs. Is it possible for Chronic to be reloaded? I am running in development mode with chronic (0.2.3). Has anyone seen something similar? Obviously, I can run my task every 10 secs using the add_periodic_timer, but I was hoping to use cron-style config. Thanks, Paul From Ivan.Manida at Sun.COM Wed Apr 9 05:26:59 2008 From: Ivan.Manida at Sun.COM (Ivan S. Manida) Date: Wed, 09 Apr 2008 13:26:59 +0400 Subject: [Backgroundrb-devel] Two Rails Apps, One Backgroundrb Server? In-Reply-To: <5A798449-19F2-4157-B3A0-A9EED77079EE@gmail.com> References: <5A798449-19F2-4157-B3A0-A9EED77079EE@gmail.com> Message-ID: <47FC8BE3.9070206@Sun.COM> Of course that is possible (just specify the same port and run it from one of the apps), but the bdrb server will only load models for the one you start it from, so if they are different tasks will fail (unless you don't use application's models in the workers). Kit Plummer wrote: > This might be a really simple question, with a simple solution. I > have two rails apps running on the same box. Right now I am starting > two Backgroundrb servers (11006 and 11007). Wondering if it might be > possible to run only one and have the second app point to the first > app's server? > > TIA, > Kit > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From kitplummer at gmail.com Wed Apr 9 08:06:15 2008 From: kitplummer at gmail.com (Kit Plummer) Date: Wed, 9 Apr 2008 05:06:15 -0700 Subject: [Backgroundrb-devel] Two Rails Apps, One Backgroundrb Server? In-Reply-To: <47FC8BE3.9070206@Sun.COM> References: <5A798449-19F2-4157-B3A0-A9EED77079EE@gmail.com> <47FC8BE3.9070206@Sun.COM> Message-ID: Yes, this is my problem...getting the workers to start. On Apr 9, 2008, at 2:26 AM, Ivan S. Manida wrote: > Of course that is possible (just specify the same port and run it > from one of the apps), but the bdrb server will only load models for > the one you start it from, so if they are different tasks will fail > (unless you don't use application's models in the workers). > > > Kit Plummer wrote: >> This might be a really simple question, with a simple solution. I >> have two rails apps running on the same box. Right now I am >> starting two Backgroundrb servers (11006 and 11007). Wondering if >> it might be possible to run only one and have the second app point >> to the first app's server? >> TIA, >> Kit >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel From scott at shefield.com Wed Apr 9 19:09:10 2008 From: scott at shefield.com (Scott Ward) Date: Wed, 9 Apr 2008 16:09:10 -0700 Subject: [Backgroundrb-devel] reload_on_schedule Message-ID: <20080409231822.7FCC618585CE@rubyforge.org> Hello, Does reload_on_schedule force the worker to be reloaded even if it is already running? Or does it just load the worker if its not running when the schedule fires? I want to load the worker at the scheduled time if its not running, but I don't want to reload the worker if its already running. Make sense? Thanks, Scott From raghu.srinivasan at gmail.com Wed Apr 9 19:46:57 2008 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Wed, 9 Apr 2008 16:46:57 -0700 Subject: [Backgroundrb-devel] Need help in passing parameters to a background job method Message-ID: <6482c06a0804091646s3d34f80dka9274390c834e551@mail.gmail.com> I am trying to send email as a background process from within a Rails app. The goal is that when I go to http://mysite.com/foo/bar, the background job kicks in and sends an email out. The relevant code snippets are below. This errors out with a wrong number of arguments passed (2 for 1) message. I think I am making a mess of passing parameters to the email_admin method in the worker. How should I be passing multiple parameters to email_admin? Thanks ============================== # Worker Code class PostalWorker < BackgrounDRb::MetaWorker set_worker_name :postal_worker def create # this method is called, when worker is loaded for the first time end def email_admin(p_subject,p_body) Emailer.deliver_send_info(p_subject,p_body,Time.now) end end ============================== # Rails controller code class FooController < ApplicationController def bar worker = MiddleMan.worker(:postal_worker) result = worker.email_admin('Some subject','And some body') render :text => "Check your email " + Time.now.to_s end ================================ # Rails nodel code class Emailer < ActionMailer::Base def send_info(subject,message,sent_at = Time.now) @subject = subject @body = {:message => message} @recipients = 'raghu.srinivasan at gmail.com' @from = '' @sent_on = sent_at @headers = {} end end ====================================== -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080409/4a680ad6/attachment.html From leavengood at gmail.com Wed Apr 9 20:45:08 2008 From: leavengood at gmail.com (Ryan Leavengood) Date: Wed, 9 Apr 2008 20:45:08 -0400 Subject: [Backgroundrb-devel] Need help in passing parameters to a background job method In-Reply-To: <6482c06a0804091646s3d34f80dka9274390c834e551@mail.gmail.com> References: <6482c06a0804091646s3d34f80dka9274390c834e551@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 7:46 PM, Raghu Srinivasan wrote: > > This errors out with a wrong number of arguments passed (2 for 1) message. I > think I am making a mess of passing parameters to the email_admin method in > the worker. How should I be passing multiple parameters to email_admin? Worker methods are only supposed to have one argument, usually called args. If this needs to take multiple values, use a hash: def email_admin(args) Emailer.deliver_send_info(args[:p_subject],args[:p_body],Time.now) end Ryan From gethemant at gmail.com Thu Apr 10 00:16:11 2008 From: gethemant at gmail.com (hemant) Date: Thu, 10 Apr 2008 09:46:11 +0530 Subject: [Backgroundrb-devel] reload_on_schedule In-Reply-To: <20080409231822.7FCC618585CE@rubyforge.org> References: <20080409231822.7FCC618585CE@rubyforge.org> Message-ID: reload_on_schedule will not force your existing workers to be reloaded, but rather it will start new one if one is already running. On 4/10/08, Scott Ward wrote: > Hello, > > Does reload_on_schedule force the worker to be reloaded even if it is > already running? > > Or does it just load the worker if its not running when the schedule fires? > > I want to load the worker at the scheduled time if its not running, but I > don't want to reload the worker if its already running. > > Make sense? > > Thanks, > > Scott > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Thu Apr 10 00:18:45 2008 From: gethemant at gmail.com (hemant) Date: Thu, 10 Apr 2008 09:48:45 +0530 Subject: [Backgroundrb-devel] Lingering status? In-Reply-To: <25e0753e0804081056j44a9043ar8daedd2cf82f545a@mail.gmail.com> References: <25e0753e0804081056j44a9043ar8daedd2cf82f545a@mail.gmail.com> Message-ID: yes worker status keeps lingering even after a worker has been deleted. You can search mailing list archives on how to write a custom handler which will handle delete request and let you attach custom handlers. On 4/8/08, Todd Willey wrote: > Is it expected that MiddleMan.ask_stats will return a status for a worker > that has been MiddleMan.delete_worker'ed? The worker isn't showing up in > MiddleMan.query_all_workers, so it seems that is shouldn't have a status > either. > > Is there a way to define a callback that is called at deletion, to reset the > status, or do I need to write my own method to reset the status before I > delete the worker? > > TIA > -todd[1] > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From veraticus at gmail.com Thu Apr 10 09:37:37 2008 From: veraticus at gmail.com (Josh Symonds) Date: Thu, 10 Apr 2008 08:37:37 -0500 Subject: [Backgroundrb-devel] Communicating Worker-to-Worker Message-ID: <76341dbf0804100637q4b7d6e95i41f24c056a4e399e@mail.gmail.com> Hi everyone, I have two workers, a connection_worker and an interface_worker, that I want to be able to call methods on each other. Initially I thought I could do this: connection_worker def alive? return true end end interface_worker def test_connection if MiddleMan.worker(:connection_worker).alive?(true) return true end end end Obviously this is a somewhat contrived test but I was just trying to make sure everything worked. I use alive?(true) because, according to http://backgroundrb.rubyforge.org/rails/index.html, passing true should cause the worker to wait for a result and return that result. Unfortunately that doesn't seem to be the cause: every time I run this method I get nil back from the connection_worker. I know that in this specific case I could just check whether the connection_worker is running, but what I'm really trying to test here is how to communicate between the two workers, not whether the connection_worker is alive. So I'm wondering what the best way to communicate between these workers is. I took a look at the advanced stuff and I think that maybe connect and start_server could work for me, but the documentation on them leaves me confused as to how to actually implement them. Any suggestions or help would be much appreciated. Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080410/1018d044/attachment-0001.html From scott at shefield.com Thu Apr 10 11:22:42 2008 From: scott at shefield.com (Scott Ward) Date: Thu, 10 Apr 2008 08:22:42 -0700 Subject: [Backgroundrb-devel] reload_on_schedule In-Reply-To: Message-ID: <20080410151926.EA1F61858662@rubyforge.org> And, also, it will start a new one if the worker is not running at all, correct? Thanks, Scott -----Original Message----- From: hemant [mailto:gethemant at gmail.com] Sent: Wednesday, April 09, 2008 9:16 PM To: Scott Ward Cc: backgroundrb-devel at rubyforge.org Subject: Re: [Backgroundrb-devel] reload_on_schedule reload_on_schedule will not force your existing workers to be reloaded, but rather it will start new one if one is already running. On 4/10/08, Scott Ward wrote: > Hello, > > Does reload_on_schedule force the worker to be reloaded even if it is > already running? > > Or does it just load the worker if its not running when the schedule fires? > > I want to load the worker at the scheduled time if its not running, > but I don't want to reload the worker if its already running. > > Make sense? > > Thanks, > > Scott > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Thu Apr 10 12:24:54 2008 From: gethemant at gmail.com (hemant kumar) Date: Thu, 10 Apr 2008 21:54:54 +0530 Subject: [Backgroundrb-devel] reload_on_schedule In-Reply-To: <20080410151926.EA1F61858662@rubyforge.org> References: <20080410151926.EA1F61858662@rubyforge.org> Message-ID: <1207844694.32671.6.camel@shire> On Thu, 2008-04-10 at 08:22 -0700, Scott Ward wrote: > And, also, it will start a new one if the worker is not running at all, > correct? Yes thats right. You can see master_worker.rb code for example. > > Thanks, > > Scott > > -----Original Message----- > From: hemant [mailto:gethemant at gmail.com] > Sent: Wednesday, April 09, 2008 9:16 PM > To: Scott Ward > Cc: backgroundrb-devel at rubyforge.org > Subject: Re: [Backgroundrb-devel] reload_on_schedule > > reload_on_schedule will not force your existing workers to be reloaded, but > rather it will start new one if one is already running. > > On 4/10/08, Scott Ward wrote: > > Hello, > > > > Does reload_on_schedule force the worker to be reloaded even if it is > > already running? > > > > Or does it just load the worker if its not running when the schedule > fires? > > > > I want to load the worker at the scheduled time if its not running, > > but I don't want to reload the worker if its already running. > > > > Make sense? > > > > Thanks, > > > > Scott > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -- > Let them talk of their oriental summer climes of everlasting conservatories; > give me the privilege of making my own summer with my own coals. > > http://gnufied.org > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From todd at rubidine.com Thu Apr 10 13:54:55 2008 From: todd at rubidine.com (Todd Willey) Date: Thu, 10 Apr 2008 17:54:55 +0000 Subject: [Backgroundrb-devel] Communicating Worker-to-Worker In-Reply-To: <76341dbf0804100637q4b7d6e95i41f24c056a4e399e@mail.gmail.com> References: <76341dbf0804100637q4b7d6e95i41f24c056a4e399e@mail.gmail.com> Message-ID: <25e0753e0804101054u2821241vf687b82c18be4533@mail.gmail.com> I think you can use (nil, true) as your arguments, or call MiddleMan.send_request(...). The RailsWorkerProxy takes the first argument as the argument to pass to the method alive, and the second argument to be the synchronous flag. See RailsWorkerProxy#method_missing. -todd[1] On Thu, Apr 10, 2008 at 1:37 PM, Josh Symonds wrote: > Hi everyone, > > I have two workers, a connection_worker and an interface_worker, that I > want to be able to call methods on each other. Initially I thought I could > do this: > > connection_worker > def alive? > return true > end > end > > interface_worker > def test_connection > if MiddleMan.worker(:connection_worker).alive?(true) > return true > end > end > end > > Obviously this is a somewhat contrived test but I was just trying to make > sure everything worked. I use alive?(true) because, according to > http://backgroundrb.rubyforge.org/rails/index.html, passing true should > cause the worker to wait for a result and return that result. Unfortunately > that doesn't seem to be the cause: every time I run this method I get nil > back from the connection_worker. I know that in this specific case I could > just check whether the connection_worker is running, but what I'm really > trying to test here is how to communicate between the two workers, not > whether the connection_worker is alive. > > So I'm wondering what the best way to communicate between these workers > is. I took a look at the advanced stuff and I think that maybe connect and > start_server could work for me, but the documentation on them leaves me > confused as to how to actually implement them. Any suggestions or help would > be much appreciated. > > Josh > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080410/ce8c2390/attachment.html From gethemant at gmail.com Thu Apr 10 18:17:35 2008 From: gethemant at gmail.com (hemant kumar) Date: Fri, 11 Apr 2008 03:47:35 +0530 Subject: [Backgroundrb-devel] First call to worker method doesn't work In-Reply-To: <24ad6d8a0804032224r62100924h3b62f7ea9bdde28a@mail.gmail.com> References: <24ad6d8a0804032057j2b5e5269ma67b8a275c3ad5d3@mail.gmail.com> <24ad6d8a0804032212h2732da01hc448d19ab07bbf95@mail.gmail.com> <24ad6d8a0804032224r62100924h3b62f7ea9bdde28a@mail.gmail.com> Message-ID: <1207865855.32671.12.camel@shire> On Thu, 2008-04-03 at 22:24 -0700, Dylan Markow wrote: > On Thu, Apr 3, 2008 at 10:12 PM, Dylan Markow wrote: > > > > On Thu, Apr 3, 2008 at 8:57 PM, Dylan Markow wrote: > > > I have a worker as follows: > > > > > > class SampleWorker < BackgrounDRb::MetaWorker > > > set_worker_name :sample_worker > > > def create(args = nil) > > > # this method is called, when worker is loaded for the first time > > > end > > > > > > def my_method > > > # Deliver test e-mail message > > > Notifications.deliver_message(1, "DM") > > > end > > > end > > > > > > I have a rails controller that calls the following code: > > > > > > worker = MiddleMan.worker(:sample_worker) > > > worker.my_method > > > > > > The problem is that the first time I load up the page for this action, > > > the worker method "my_method" doesn't seem to ever get called. There > > > is zero output from any of the backgroundrb log files. If I then > > > reload the page, it works fine (and continues to work fine until > > > backgroundrb is restarted). > > > > > > This happens whether I'm in dev. or prod. mode. > > > > > > > To clarify, if I restart rails AND backgroundrb at the same time, > > everything works fine. it's only if backgroundrb needs to restart on > > its own for some reason. It looks like when the backgroundrb process > > is restarted, rails doesn't realize this as it still sees port 11006 > > open. So my first request to the page results in the rails plugin > > attempting to write data to the socket. I'm assuming this screws > > something up, as upon the second attempt at loading my page, "rescue > > Errno::EPIPE" at line 64 of backgroundrb.rb is called, thus > > re-establishing the connection. I'm assuming this problem may be more > > closely related to the packet gem? (Not sure if there is a trac repo > > for packet). > > > > > One last note: If you have 5 mongrels running of your rails app, then > the worker will fail to fire 5 times (once for each mongrel), though > not always in a row (typically my second request works, but then my > 3rd or 4th, etc., won't) > > As it stands now, I'm setting up my deployment files to always restart > backgroundrb BEFORE rails to make sure that once each mongrel fires > up, they each have a valid instance of backgroundrb to point to. Hi Dylan, Sorry for late reply, but I think I nailed down issues you mentioned. But you need to get latest code, which is not yet in git repo: 1. First update your packet. Get packet code from here: http://code.google.com/p/packet/source/checkout Install using: rake gem; sudo gem install packet-0.1.6.gem 2. Get backgroundrb code with fixes from: http://backgroundrb.gnufied.org/manual/backgroundrb.tar.gz Now, you are set. Your problems were because of a bug in Ruby itself and hence I have reverted connection to old style of connecting on each request. There was a bug in packet itself for frequent connection/disconnection. See, if above fixes your problem. I will be publishing both softwares upstream in few days. From paul.kmiec at appfolio.com Fri Apr 11 18:15:23 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Fri, 11 Apr 2008 15:15:23 -0700 Subject: [Backgroundrb-devel] does backgroundrb server need rails environment? Message-ID: <674a03dc0804111515x76e7b7a3u59a7a7e72032618e@mail.gmail.com> Hi everyone, I noticed that script/backgroudrb requires config/environment which causes the backgroundrb server as well as the log worker to 'bloat' to 35MB each. I am kind of sensitive to memory issues, so I patched the code and essentially moved the require of environment from script/backgroundrb to the meta_worker. Everything seems good and now both backgroundrb server and log worker hang around 12MB. Is there a reason why script/backgroundrb loads the environment? I suppose it makes spawning rail workers faster? Anyways, if anyone is interested, I can post my simple patch. Paul From gethemant at gmail.com Fri Apr 11 19:15:50 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 12 Apr 2008 04:45:50 +0530 Subject: [Backgroundrb-devel] does backgroundrb server need rails environment? In-Reply-To: <674a03dc0804111515x76e7b7a3u59a7a7e72032618e@mail.gmail.com> References: <674a03dc0804111515x76e7b7a3u59a7a7e72032618e@mail.gmail.com> Message-ID: <1207955750.27774.15.camel@shire> On Fri, 2008-04-11 at 15:15 -0700, Paul Kmiec wrote: > Hi everyone, > > I noticed that script/backgroudrb requires config/environment which > causes the backgroundrb server as well as the log worker to 'bloat' to > 35MB each. I am kind of sensitive to memory issues, so I patched the > code and essentially moved the require of environment from > script/backgroundrb to the meta_worker. Everything seems good and now > both backgroundrb server and log worker hang around 12MB. > > Is there a reason why script/backgroundrb loads the environment? I > suppose it makes spawning rail workers faster? > > Anyways, if anyone is interested, I can post my simple patch. > Paul Environment loading is a little messy right now. I wanted to work on it and make it clean, so i will welcome any patch. From gethemant at gmail.com Fri Apr 11 20:22:44 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 12 Apr 2008 05:52:44 +0530 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue Message-ID: <1207959764.27774.27.camel@shire> Hi Folks, I have been working to fix BackgrounDRb memory usage as such. Ruby1.8 GC isn't fork friendly and it seems to use a lot of memory because when you fork, it sets a bit in all the objects in global scope which causes OS to all pages in child process. A classic solution is to use fork and exec, rather than just fork. Its working and pretty stable, but you loose ability to pass complex arguments when invoking MiddleMan.new_worker, because essentially I can only pass arguments in form of strings when using exec. This help significantly in improving stability of BackgrounDRb and code written on top of packet. So what you guys think? Shall we go for this change? From leavengood at gmail.com Fri Apr 11 20:54:59 2008 From: leavengood at gmail.com (Ryan Leavengood) Date: Fri, 11 Apr 2008 20:54:59 -0400 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue In-Reply-To: <1207959764.27774.27.camel@shire> References: <1207959764.27774.27.camel@shire> Message-ID: On Fri, Apr 11, 2008 at 8:22 PM, hemant kumar wrote: > > A classic solution is to use fork and exec, rather than just fork. Its > working and pretty stable, but you loose ability to pass complex > arguments when invoking MiddleMan.new_worker, because essentially I can > only pass arguments in form of strings when using exec. Is that really a problem? irb(main):001:0> a = [{:key=>'value','another one'=>'something'}, 3, 56, "foobar"] => [{"another one"=>"something", :key=>"value"}, 3, 56, "foobar"] irb(main):002:0> s = Marshal.dump(a) => "\004\b[\t{\a\"\020another one\"\016something:\bkey\"\nvaluei\bi=\"\vfoobar" irb(main):003:0> b = Marshal.load(s) => [{"another one"=>"something", :key=>"value"}, 3, 56, "foobar"] irb(main):004:0> a == b => true > This help significantly in improving stability of BackgrounDRb and code > written on top of packet. So what you guys think? Shall we go for this > change? It sounds good to me. Anything that helps stability and improves memory usage will always get my vote. Regards, Ryan From lists at wildgooses.com Fri Apr 11 16:43:54 2008 From: lists at wildgooses.com (Ed W) Date: Fri, 11 Apr 2008 21:43:54 +0100 Subject: [Backgroundrb-devel] Can I do this, question Message-ID: <47FFCD8A.9090404@wildgooses.com> I have a bunch of real uses for BackgroundDrb, but I also a very simple need to delay sending an email for about 5 seconds after a rails action completes. Basically I need to leave time for the DB to commit and sync to the slaves before the email can be sent Is BackgroundDb the right tool for this job? Any other suggestions on ways to implement this? (I have seen suggestions to create a new thread and run the mailer from that thread, but it's not clear that actionmailer is actually thread safe..?) I'm thinking just to fire up worker threads which pause for a length of time before sending the email - is this likely the best solution? Any other suggestions? Cheers Ed W From raghu.srinivasan at gmail.com Sat Apr 12 01:57:57 2008 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Fri, 11 Apr 2008 22:57:57 -0700 Subject: [Backgroundrb-devel] Sequence of multiple background process submissions Message-ID: <6482c06a0804112257o682f11a2m94fbbf342c860333@mail.gmail.com> I have a question on how multiple b/g processes that are kicked off get handled in terms of sequence/schedule. On my site, a user enters an RSS feed to be processed and since this takes about 5-10 secs, I pass the process off to a background job and meanwhile do some Ajaxy spinners and tap-dancing until the job completes and then redirect_to the appropriate page. This works great. I also need to do a batch refresh of all feeds at night. I am doing these also in the b/g. Now, in a loop, if I submit 100 b/g jobs, the jobs get processed sequentially. However, if a web user comes along during this batch job and adds his feed, then the b/g process to serve him gets called only after the 100th batch job has completed. Am I correct in assuming that this is expected behavior? Next, is there a way around this? Can I have 2 threads/processes/ports/etc for Bdrb so that the batch job doesn't interfere with a live user's experience. Or any other workaround for this? Right now if the web user comes along when 50 jobs are left and each job takes 10 secs, then he has a nearly 10 minute wait, which sucks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080411/07df8aa1/attachment.html From leavengood at gmail.com Sat Apr 12 02:30:16 2008 From: leavengood at gmail.com (Ryan Leavengood) Date: Sat, 12 Apr 2008 02:30:16 -0400 Subject: [Backgroundrb-devel] Sequence of multiple background process submissions In-Reply-To: <6482c06a0804112257o682f11a2m94fbbf342c860333@mail.gmail.com> References: <6482c06a0804112257o682f11a2m94fbbf342c860333@mail.gmail.com> Message-ID: On Sat, Apr 12, 2008 at 1:57 AM, Raghu Srinivasan wrote: > > On my site, a user enters an RSS feed to be processed and since this takes > about 5-10 secs, I pass the process off to a background job and meanwhile do > some Ajaxy spinners and tap-dancing until the job completes and then > redirect_to the appropriate page. This works great. Tap-dancing, LOL. I love that. > Next, is there a way around this? Can I have 2 threads/processes/ports/etc > for Bdrb so that the batch job doesn't interfere with a live user's > experience. Or any other workaround for this? Right now if the web user > comes along when 50 jobs are left and each job takes 10 secs, then he has a > nearly 10 minute wait, which sucks. In my opinion you should always separate scheduled long running processes from user spawned ones. What I would do in this case is extract the common RSS processing functionality into a class in your Rails lib directory, then create two different BackgrounDRb workers that make use of that class. One would be UserRSSWorker and the other could be ScheduledRSSWorker. The first should only be used for user requests (and in addition you should use thread_pool.defer to allow multiple requests at once, which also might solve your original problem with one worker) and the other can be set up on your schedule. Also I am not sure how to do it offhand, but you should try to set up the ScheduledRSSWorker so that BackgrounDRb instantiates it fresh to run and then kills it once it is done, since you don't need it sitting around all day doing nothing. But as I said above in parenthesis in general if you want a worker to be able to handle many jobs at once, use thread_pool.defer. Be sure to read the documentation because since this is threaded code there are things you need to be careful about. Ryan From raghu.srinivasan at gmail.com Sat Apr 12 02:53:19 2008 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Fri, 11 Apr 2008 23:53:19 -0700 Subject: [Backgroundrb-devel] Sequence of multiple background process submissions In-Reply-To: References: <6482c06a0804112257o682f11a2m94fbbf342c860333@mail.gmail.com> Message-ID: <6482c06a0804112353k2f5ce0b7o34ba641a77c85c8e@mail.gmail.com> On Fri, Apr 11, 2008 at 11:30 PM, Ryan Leavengood wrote: > On Sat, Apr 12, 2008 at 1:57 AM, Raghu Srinivasan > wrote: > > > > On my site, a user enters an RSS feed to be processed and since this > takes > > about 5-10 secs, I pass the process off to a background job and > meanwhile do > > some Ajaxy spinners and tap-dancing until the job completes and then > > redirect_to the appropriate page. This works great. > > Tap-dancing, LOL. I love that. > > > Next, is there a way around this? Can I have 2 > threads/processes/ports/etc > > for Bdrb so that the batch job doesn't interfere with a live user's > > experience. Or any other workaround for this? Right now if the web user > > comes along when 50 jobs are left and each job takes 10 secs, then he > has a > > nearly 10 minute wait, which sucks. > > In my opinion you should always separate scheduled long running > processes from user spawned ones. What I would do in this case is > extract the common RSS processing functionality into a class in your > Rails lib directory, then create two different BackgrounDRb workers > that make use of that class. One would be UserRSSWorker and the other > could be ScheduledRSSWorker. The first should only be used for user > requests (and in addition you should use thread_pool.defer to allow > multiple requests at once, which also might solve your original > problem with one worker) and the other can be set up on your schedule. > Also I am not sure how to do it offhand, but you should try to set up > the ScheduledRSSWorker so that BackgrounDRb instantiates it fresh to > run and then kills it once it is done, since you don't need it sitting > around all day doing nothing. > The RSS processing is already extracted into a .rb in my lib directory. Right now I have just one worker (called parse_feeds). If I understand you correctly, here's what I need to do: duplicate parse_feeds such that I have two identical workers, once called parse_feeds_web and the other parse_feeds_batch. And in my Rails controller, kick off parse_feeds_web for a live user but use the parse_feeds_batch for the nightly job. This will ensure that parse_feeds_web need not wait for parse_feeds_batch processes to complete. Sounds great. I'll try that. > > But as I said above in parenthesis in general if you want a worker to > be able to handle many jobs at once, use thread_pool.defer. Be sure to > read the documentation because since this is threaded code there are > things you need to be careful about. > I did not know about thread_pool.defer. Thanks. I'll be sure to read and understand it before I mess with threaded stuff. But it will be good in case several users all plug in their feeds at the same time. > > Ryan > Thanks Ryan! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080411/76bca6a6/attachment.html From gethemant at gmail.com Sat Apr 12 05:27:32 2008 From: gethemant at gmail.com (hemant kumar) Date: Sat, 12 Apr 2008 14:57:32 +0530 Subject: [Backgroundrb-devel] Can I do this, question In-Reply-To: <47FFCD8A.9090404@wildgooses.com> References: <47FFCD8A.9090404@wildgooses.com> Message-ID: <1207992452.8435.1.camel@shire> On Fri, 2008-04-11 at 21:43 +0100, Ed W wrote: > I have a bunch of real uses for BackgroundDrb, but I also a very simple > need to delay sending an email for about 5 seconds after a rails action > completes. Basically I need to leave time for the DB to commit and sync > to the slaves before the email can be sent > > Is BackgroundDb the right tool for this job? Any other suggestions on > ways to implement this? (I have seen suggestions to create a new thread > and run the mailer from that thread, but it's not clear that > actionmailer is actually thread safe..?) > > I'm thinking just to fire up worker threads which pause for a length of > time before sending the email - is this likely the best solution? Any > other suggestions? you can use : add_timer(5) { send_delayed_mail(mail_content_id) } It will send all mails delayed by 5 seconds also, you should not use sleep in your code to simulate "pause". sleep doesn't work well with nonblocking IO reactor loops. From jeroen at laika.nl Sat Apr 12 12:54:15 2008 From: jeroen at laika.nl (jeroen janssen) Date: Sat, 12 Apr 2008 18:54:15 +0200 Subject: [Backgroundrb-devel] Startup error Message-ID: <2FD5B29C-D7B5-4299-824D-217698D6CCAD@laika.nl> Hi all, I'm new to this list (and pretty new to backgroundrb). I'd like to jump in with a question, I hope you don't mind. I'm trying to get my site working with backgroundrb to sent mail. It's pretty basic stuff and I got it working pretty fast on my local machine (os x leopard). Unfortunately, when I try to deploy to my production machine I get an error when I try to start backgroundrb. I'm not really sure how to debug this issue, I don't see anything in the log. My production machine is hosted by Slicehost. and is running Ubuntu. I get the following error: /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_meta_pimp.rb:48:in `process_request': undefined method `send_request' for nil:NilClass (NoMethodError) from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_meta_pimp.rb:27:in `handle_object' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_meta_pimp.rb:20:in `receive_data' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `call' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `extract' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_meta_pimp.rb:18:in `receive_data' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:64:in `handle_internal_messages' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:176:in `handle_read_event' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `each' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `handle_read_event' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:130:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `loop' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:21:in `run' from /var/www/apps/lego/releases/20080412142415/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_nbio.rb: 25:in `read_data': Packet::DisconnectError (Packet::DisconnectError) from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_worker.rb:46:in `handle_internal_messages' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:176:in `handle_read_event' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `each' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `handle_read_event' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:130:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `loop' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `start_reactor' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_worker.rb:20:in `start_worker' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:133:in `fork_and_load' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:96:in `load_workers' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:91:in `each' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:91:in `load_workers' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:20:in `run' from /var/www/apps/lego/releases/20080412142415/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 From paul.kmiec at appfolio.com Sat Apr 12 20:45:46 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Sat, 12 Apr 2008 17:45:46 -0700 Subject: [Backgroundrb-devel] does backgroundrb server need rails environment? In-Reply-To: <1207955750.27774.15.camel@shire> References: <674a03dc0804111515x76e7b7a3u59a7a7e72032618e@mail.gmail.com> <1207955750.27774.15.camel@shire> Message-ID: <674a03dc0804121745v4c7cd35dqe4ef121231ad908f@mail.gmail.com> Hi, Attached are the changes that prevent backgroundrb server from loading the rail's boot & environment. Removing environment has the implication that MasterProxy does not load_rails_env ... therefore, the lazy load flag (if needed as a feature) needs to be moved into meta_worker's load_rails_env. After removing environment, I noticed that we still needed active_support. I was able to identify and fix the two issues caused by not having active_support, - find_reloadable_worker used classify ... turns out packet has its own version of classify in Packet::NbioHelper which I was able to use. - MissingSourceFile is defined in active support. The usages of MissingSourceFile in backgroundrb & packet are the same as LoadError. I just defined MissingSourceFile. I think packet's use of MissingSourceFile is a bug ... packet requires all the libraries it needs in packet.rb, but does not include active_resource. Also, since they have implemented their own version of classify, I think this further supports their intent not to rely on active_support. What do you think? Also, I added yield to MasterProxy which allows additional custom initialization, which I happen to do :). Paul On Fri, Apr 11, 2008 at 4:15 PM, hemant kumar wrote: > > > On Fri, 2008-04-11 at 15:15 -0700, Paul Kmiec wrote: > > Hi everyone, > > > > I noticed that script/backgroudrb requires config/environment which > > causes the backgroundrb server as well as the log worker to 'bloat' to > > 35MB each. I am kind of sensitive to memory issues, so I patched the > > code and essentially moved the require of environment from > > script/backgroundrb to the meta_worker. Everything seems good and now > > both backgroundrb server and log worker hang around 12MB. > > > > Is there a reason why script/backgroundrb loads the environment? I > > suppose it makes spawning rail workers faster? > > > > Anyways, if anyone is interested, I can post my simple patch. > > Paul > > Environment loading is a little messy right now. I wanted to work on it > and make it clean, so i will welcome any patch. > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: backgroundrb.diff Type: application/octet-stream Size: 4423 bytes Desc: not available Url : http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080412/5d7e95db/attachment-0001.obj From lists at wildgooses.com Sun Apr 13 05:41:27 2008 From: lists at wildgooses.com (Ed W) Date: Sun, 13 Apr 2008 10:41:27 +0100 Subject: [Backgroundrb-devel] Can I do this, question In-Reply-To: <1207992452.8435.1.camel@shire> References: <47FFCD8A.9090404@wildgooses.com> <1207992452.8435.1.camel@shire> Message-ID: <4801D547.2040806@wildgooses.com> hemant kumar wrote: > you can use : > > add_timer(5) { send_delayed_mail(mail_content_id) } > > It will send all mails delayed by 5 seconds also, you should not use > sleep in your code to simulate "pause". sleep doesn't work well with > nonblocking IO reactor loops. > > > OK, cool. So if I want to delay the start by five seconds AND I expect the process I am running to take (say) 60 seconds to complete (some theoretical long running process), what's the best way to structure that so that I don't block the IO loop? Thanks Ed W From paul.kmiec at appfolio.com Sun Apr 13 19:41:16 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Sun, 13 Apr 2008 16:41:16 -0700 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue Message-ID: <674a03dc0804131641x3f66a918xf49c8d239f7af9c6@mail.gmail.com> > > > This help significantly in improving stability of BackgrounDRb and code > > written on top of packet. So what you guys think? Shall we go for this > > change? > > It sounds good to me. Anything that helps stability and improves > memory usage will always get my vote. I agree. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080413/1d589f50/attachment.html From brian at morearty.org Sun Apr 13 20:44:52 2008 From: brian at morearty.org (Brian Morearty) Date: Sun, 13 Apr 2008 17:44:52 -0700 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows Message-ID: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> Hello all, I have some good news to announce. I have made some changes to a local copy of BackgrounDRb 1.0.3 (actually most of the changes were in Packet 0.1.5) and I've got it working on Windows. From what I've read, this is the first time it's worked on Windows in a year and a half, since 0.2 was released. In my research I can see that a number of people have asked for Windows support so I'm excited that I can help out. I'm new to this mailing list and new to BackgrounDRb so I would appreciate advice and help from you all on things like: - how to test my changes. When I run rake it just exits without running any tests. I'm not sure why. I also don't know how to test Packet. - how to send a patch. I mostly do Windows development where textual patches are not commonplace. But I did submit a patch to Rails that was accepted, so at least I've done it once with success. :-) - a code review to make sure I'm doing things in an approved way - someone with *nix to make sure I didn't break anything by accident. I tried hard to not but you can't be sure without testing. Since I haven't figured out how to run automated tests I have created some ad hoc tests of my own. These cases work: - ad hoc scheduling by calling a method on the worker - passing parameters to the worker - cron scheduling - register_status and ask_status - making a synchronous call and getting a result back kind of works, except I get the whole hash back with the result in the :data key instead of just getting the result alone. In other words I get a hash like this: {:type=>:response, :client_signature=>25, :result=>true, :data=>1} which is odd because it's calling the same "extract" function that ask_status uses, so I'm not sure why this is any different. There are some things I haven't tried yet but it's great that this many cases work. I definitely have to try to figure out why the return value of a synchronous call is the whole hash, and I might need help on that because I'm mystified. The primary obstacles to making BackgrounDRb work on Windows were: 1. UNIXSocket - it wasn't too hard to add code that uses TCPSocket unless defined? UNIXSocket. And there were four lines of code that checked if a socket was a UNIXSocket to decide whether it's an "internal" read or write, but those lines were crashing because UNIXSocket isn't even defined on Windows, much less used. My fix for that was to create a marker Module called "Packet::InternalSocket." Each time I created an internal socket (where the code used to create a UNIXSocket) I now call socket.extend Packet::InternalSocket so that later on the code can distinguish the internal ones from the others. 2. fork - well that's a harder matter. Windows doesn't support fork() and probably never will. I saw a recent post by Hemant Kumar mentioning a fix that uses fork and exec rather than just fork, but it still requires fork. Simulating fork in a generic way is nearly impossible but replacing it in a single application is conceivable, depending on what the app does. And in this case I was able to do this: a) Leave the existing fork call there b) Fall back on IO.popen for operating systems where fork is not supported What I did is for each worker I launch (instead of fork) a new child process, pass the read/write port numbers to the child on the command line, and wait for it to connect to them. Because Rails takes a long time to start up on Windows (11 seconds, ugh) you can wait a pretty long time before all your workers are ready (11sec * (number of workers+2)). But still, it's better than not being able to use it at all. It's perfectly acceptable for a development machine, which is how I use Rails on Windows. (My project is going to deploy on something that ends in nix.) On the other hand if you're going to be dynamically creating background workers (e.g. reload_on_schedule true or set_no_auto_load true) you'll wait a while for the worker to start up on Windows. Will this slow startup time ever block a Rails server request? Not that I've seen. From what I've seen BackgrounDRb never starts a new worker process because of an API call. The only case I've seen where it starts a new worker process on the fly is for a scheduled background task using the cron feature. But I may have missed something. Another thing that I didn't implement (at least not for now): support for the "start" parameter to run the background processes as hidden processes. For now on Windows you just have to run script/backgroundrb without the start parameter and just leave it running in a console. (But I did make the error message nicer.) The reason I didn't: Ruby does not seem to have implemented a way to kill a process tree in Windows. Windows doesn't have a single API to kill a process tree. You can add all the processes to a Job and kill that, or you can enumerate the descendants of a process and kill them all one by one. I just didn't want to bother adding the code to Ruby do that, at least not now, because it works well enough in a console. And the nice thing is Ctrl+C from the console does indeed kill the whole process tree. That's it for now. Please let me know how to proceed so us Windows users can benefit from the wonderfulness of BackgrounDRb. -- Brian Morearty -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080413/dd7c953d/attachment.html From gethemant at gmail.com Sun Apr 13 21:16:08 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 14 Apr 2008 06:46:08 +0530 Subject: [Backgroundrb-devel] does backgroundrb server need rails environment? In-Reply-To: <674a03dc0804121745v4c7cd35dqe4ef121231ad908f@mail.gmail.com> References: <674a03dc0804111515x76e7b7a3u59a7a7e72032618e@mail.gmail.com> <1207955750.27774.15.camel@shire> <674a03dc0804121745v4c7cd35dqe4ef121231ad908f@mail.gmail.com> Message-ID: <1208135768.14339.5.camel@shire> On Sat, 2008-04-12 at 17:45 -0700, Paul Kmiec wrote: > Hi, > > Attached are the changes that prevent backgroundrb server from loading > the rail's boot & environment. > > Removing environment has the implication that MasterProxy does not > load_rails_env ... therefore, the lazy load flag (if needed as a > feature) needs to be moved into meta_worker's load_rails_env. So, if users who want to pass AR objects from rails, they will be in trouble because of no environment loading at all. I think we do need an option like 'master_rails_env' enable or disable. > > After removing environment, I noticed that we still needed > active_support. I was able to identify and fix the two issues caused > by not having active_support, > - find_reloadable_worker used classify ... turns out packet has its > own version of classify in Packet::NbioHelper which I was able to use. > - MissingSourceFile is defined in active support. The usages of > MissingSourceFile in backgroundrb & packet are the same as LoadError. > I just defined MissingSourceFile. > > I think packet's use of MissingSourceFile is a bug ... packet requires > all the libraries it needs in packet.rb, but does not include > active_resource. Also, since they have implemented their own version > of classify, I think this further supports their intent not to rely on > active_support. What do you think? Yes, thats a bug in packet. I have made those changes. Anyone interested in hacking packet itself can start here (I am looking at you Brian too) : http://github.com/gnufied/packet/tree/master > > Also, I added yield to MasterProxy which allows additional custom > initialization, which I happen to do :). > Paul Hmm, thats nice. I will get this patch in. From gethemant at gmail.com Sun Apr 13 21:21:09 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 14 Apr 2008 06:51:09 +0530 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue In-Reply-To: References: <1207959764.27774.27.camel@shire> Message-ID: <1208136069.14339.9.camel@shire> On Fri, 2008-04-11 at 20:54 -0400, Ryan Leavengood wrote: > On Fri, Apr 11, 2008 at 8:22 PM, hemant kumar wrote: > > > > A classic solution is to use fork and exec, rather than just fork. Its > > working and pretty stable, but you loose ability to pass complex > > arguments when invoking MiddleMan.new_worker, because essentially I can > > only pass arguments in form of strings when using exec. > > Is that really a problem? > > irb(main):001:0> a = [{:key=>'value','another one'=>'something'}, 3, > 56, "foobar"] > => [{"another one"=>"something", :key=>"value"}, 3, 56, "foobar"] > irb(main):002:0> s = Marshal.dump(a) > => "\004\b[\t{\a\"\020another one\"\016something:\bkey\"\nvaluei\bi=\"\vfoobar" > irb(main):003:0> b = Marshal.load(s) > => [{"another one"=>"something", :key=>"value"}, 3, 56, "foobar"] > irb(main):004:0> a == b > => true Nevermind, that wasn't a problem at all. As a simple fix, I am avoiding passing anything other than file descriptors as command line argument and before forking and execing the process, I am writing worker_options to write end of master process. Seems to be working well. Something like this: require "socket" def enable_nonblock io f = io.fcntl(Fcntl::F_GETFL,0) io.fcntl(Fcntl::F_SETFL,Fcntl::O_NONBLOCK | f) end master_read_end,worker_write_end = UNIXSocket.pair(Socket::SOCK_STREAM) worker_read_end,master_write_end = UNIXSocket.pair(Socket::SOCK_STREAM) master_write_end.puts "Hello from parent" if(!(pid = fork)) [master_write_end,master_read_end].each { |x| x.close } [worker_read_end,worker_write_end].each { |x| enable_nonblock(x) } exec("bar.rb","#{worker_read_end.fileno}:#{worker_write_end.fileno}") end [master_write_end,master_read_end].each { |x| enable_nonblock(x) } p pid worker_read_end.close worker_write_end.close a = master_read_end.gets puts a From brian at morearty.org Sun Apr 13 22:03:55 2008 From: brian at morearty.org (Brian Morearty) Date: Sun, 13 Apr 2008 19:03:55 -0700 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows In-Reply-To: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> References: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> Message-ID: <3f8e47bb0804131903g30ff691di4107cd196fe4b065@mail.gmail.com> I have a couple of patch files I can send the the appropriate place. One for packet and one for backgroundrb. Am I supposed to send them to this list for review, email them to hemant, request write access to a SubVersion or Git repository, or something else? I need a little handholding with this part so please be patient. :-) I haven't participated in open source very much so I'm not sure what the conventions are. - Brian On Sun, Apr 13, 2008 at 5:44 PM, Brian Morearty wrote: > Hello all, > > I have some good news to announce. I have made some changes to a local > copy of BackgrounDRb 1.0.3 (actually most of the changes were in Packet > 0.1.5) and I've got it working on Windows. From what I've read, this is the > first time it's worked on Windows in a year and a half, since 0.2 was > released. > > In my research I can see that a number of people have asked for Windows > support so I'm excited that I can help out. > > I'm new to this mailing list and new to BackgrounDRb so I would appreciate > advice and help from you all on things like: > - how to test my changes. When I run rake it just exits without running > any tests. I'm not sure why. I also don't know how to test Packet. > - how to send a patch. I mostly do Windows development where textual > patches are not commonplace. But I did submit a patch to Rails that was > accepted, so at least I've done it once with success. :-) > - a code review to make sure I'm doing things in an approved way > - someone with *nix to make sure I didn't break anything by accident. I > tried hard to not but you can't be sure without testing. > > Since I haven't figured out how to run automated tests I have created some > ad hoc tests of my own. These cases work: > - ad hoc scheduling by calling a method on the worker > - passing parameters to the worker > - cron scheduling > - register_status and ask_status > - making a synchronous call and getting a result back kind of works, > except I get the whole hash back with the result in the :data key instead of > just getting the result alone. In other words I get a hash like this: > {:type=>:response, :client_signature=>25, :result=>true, :data=>1} > which is odd because it's calling the same "extract" function that > ask_status uses, so I'm not sure why this is any different. > > There are some things I haven't tried yet but it's great that this many > cases work. > > I definitely have to try to figure out why the return value of a > synchronous call is the whole hash, and I might need help on that because > I'm mystified. > > The primary obstacles to making BackgrounDRb work on Windows were: > 1. UNIXSocket - it wasn't too hard to add code that uses TCPSocket unless > defined? UNIXSocket. And there were four lines of code that checked if a > socket was a UNIXSocket to decide whether it's an "internal" read or write, > but those lines were crashing because UNIXSocket isn't even defined on > Windows, much less used. My fix for that was to create a marker Module > called "Packet::InternalSocket." Each time I created an internal socket > (where the code used to create a UNIXSocket) I now call socket.extend > Packet::InternalSocket so that later on the code can distinguish the > internal ones from the others. > 2. fork - well that's a harder matter. Windows doesn't support fork() and > probably never will. I saw a recent post by Hemant Kumar mentioning a fix > that uses fork and exec rather than just fork, but it still requires fork. > Simulating fork in a generic way is nearly impossible but replacing it in a > single application is conceivable, depending on what the app does. And in > this case I was able to do this: > a) Leave the existing fork call there > b) Fall back on IO.popen for operating systems where fork is not > supported > What I did is for each worker I launch (instead of fork) a new child > process, pass the read/write port numbers to the child on the command line, > and wait for it to connect to them. > > Because Rails takes a long time to start up on Windows (11 seconds, ugh) > you can wait a pretty long time before all your workers are ready (11sec * > (number of workers+2)). But still, it's better than not being able to use it > at all. It's perfectly acceptable for a development machine, which is how I > use Rails on Windows. (My project is going to deploy on something that ends > in nix.) > > On the other hand if you're going to be dynamically creating background > workers (e.g. reload_on_schedule true or set_no_auto_load true) you'll wait > a while for the worker to start up on Windows. > > Will this slow startup time ever block a Rails server request? Not that > I've seen. From what I've seen BackgrounDRb never starts a new worker > process because of an API call. The only case I've seen where it starts a > new worker process on the fly is for a scheduled background task using the > cron feature. But I may have missed something. > > Another thing that I didn't implement (at least not for now): support for > the "start" parameter to run the background processes as hidden processes. > For now on Windows you just have to run script/backgroundrb without the > start parameter and just leave it running in a console. (But I did make the > error message nicer.) The reason I didn't: Ruby does not seem to have > implemented a way to kill a process tree in Windows. Windows doesn't have a > single API to kill a process tree. You can add all the processes to a Job > and kill that, or you can enumerate the descendants of a process and kill > them all one by one. I just didn't want to bother adding the code to Ruby do > that, at least not now, because it works well enough in a console. > > And the nice thing is Ctrl+C from the console does indeed kill the whole > process tree. > > That's it for now. Please let me know how to proceed so us Windows users > can benefit from the wonderfulness of BackgrounDRb. > > -- > Brian Morearty > > -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080413/5e1f4521/attachment-0001.html From gethemant at gmail.com Sun Apr 13 21:53:59 2008 From: gethemant at gmail.com (hemant kumar) Date: Mon, 14 Apr 2008 07:23:59 +0530 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue In-Reply-To: <674a03dc0804131641x3f66a918xf49c8d239f7af9c6@mail.gmail.com> References: <674a03dc0804131641x3f66a918xf49c8d239f7af9c6@mail.gmail.com> Message-ID: <1208138039.14339.19.camel@shire> On Sun, 2008-04-13 at 16:41 -0700, Paul Kmiec wrote: > > This help significantly in improving stability of > BackgrounDRb and code > > written on top of packet. So what you guys think? Shall we > go for this > > change? > > It sounds good to me. Anything that helps stability and > improves > memory usage will always get my vote. > > I agree. > > Paul Okay folks, I have updated git repo of just packet with latest changes wherein I have removed fork in favour of fork and exec: http://github.com/gnufied/packet/tree/master The only catch is, now we need some kinda runner to run our workers since exec takes argument in form of string only. For example: exec "packet_worker_runner ##{t_worker_name}:{worker_read_end.fileno}:#{worker_write_end.fileno}:#{option_dump_length}" One option we have is to have "packet_worker_runner" as installed executable when user installs packet gem. Any other ideas? From brian at morearty.org Sun Apr 13 22:07:26 2008 From: brian at morearty.org (Brian Morearty) Date: Sun, 13 Apr 2008 19:07:26 -0700 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows In-Reply-To: <3f8e47bb0804131903g30ff691di4107cd196fe4b065@mail.gmail.com> References: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> <3f8e47bb0804131903g30ff691di4107cd196fe4b065@mail.gmail.com> Message-ID: <3f8e47bb0804131907r790a47c3u209393a4a802a707@mail.gmail.com> P.S. Hemant I did see your note "(I am looking at you Brian too)" in the other thread where you mentioned anyone interested in hacking at Packet can start with that Git repository. And my name's Brian so maybe you meant me. :-) I'm just not sure what I'm supposed to do there. Just check it in? (Is that repository a playground that's open to anyone?) Thanks again. - Brian On Sun, Apr 13, 2008 at 7:03 PM, Brian Morearty wrote: > I have a couple of patch files I can send the the appropriate place. One > for packet and one for backgroundrb. Am I supposed to send them to this list > for review, email them to hemant, request write access to a SubVersion or > Git repository, or something else? > > I need a little handholding with this part so please be patient. :-) I > haven't participated in open source very much so I'm not sure what the > conventions are. > > - Brian > > On Sun, Apr 13, 2008 at 5:44 PM, Brian Morearty > wrote: > > > Hello all, > > > > I have some good news to announce. I have made some changes to a local > > copy of BackgrounDRb 1.0.3 (actually most of the changes were in Packet > > 0.1.5) and I've got it working on Windows. From what I've read, this is the > > first time it's worked on Windows in a year and a half, since 0.2 was > > released. > > > > In my research I can see that a number of people have asked for Windows > > support so I'm excited that I can help out. > > > > I'm new to this mailing list and new to BackgrounDRb so I would > > appreciate advice and help from you all on things like: > > - how to test my changes. When I run rake it just exits without running > > any tests. I'm not sure why. I also don't know how to test Packet. > > - how to send a patch. I mostly do Windows development where textual > > patches are not commonplace. But I did submit a patch to Rails that was > > accepted, so at least I've done it once with success. :-) > > - a code review to make sure I'm doing things in an approved way > > - someone with *nix to make sure I didn't break anything by accident. I > > tried hard to not but you can't be sure without testing. > > > > Since I haven't figured out how to run automated tests I have created > > some ad hoc tests of my own. These cases work: > > - ad hoc scheduling by calling a method on the worker > > - passing parameters to the worker > > - cron scheduling > > - register_status and ask_status > > - making a synchronous call and getting a result back kind of works, > > except I get the whole hash back with the result in the :data key instead of > > just getting the result alone. In other words I get a hash like this: > > {:type=>:response, :client_signature=>25, :result=>true, :data=>1} > > which is odd because it's calling the same "extract" function that > > ask_status uses, so I'm not sure why this is any different. > > > > There are some things I haven't tried yet but it's great that this many > > cases work. > > > > I definitely have to try to figure out why the return value of a > > synchronous call is the whole hash, and I might need help on that because > > I'm mystified. > > > > The primary obstacles to making BackgrounDRb work on Windows were: > > 1. UNIXSocket - it wasn't too hard to add code that uses TCPSocket > > unless defined? UNIXSocket. And there were four lines of code that checked > > if a socket was a UNIXSocket to decide whether it's an "internal" read or > > write, but those lines were crashing because UNIXSocket isn't even defined > > on Windows, much less used. My fix for that was to create a marker Module > > called "Packet::InternalSocket." Each time I created an internal socket > > (where the code used to create a UNIXSocket) I now call socket.extend > > Packet::InternalSocket so that later on the code can distinguish the > > internal ones from the others. > > 2. fork - well that's a harder matter. Windows doesn't support fork() > > and probably never will. I saw a recent post by Hemant Kumar mentioning a > > fix that uses fork and exec rather than just fork, but it still requires > > fork. Simulating fork in a generic way is nearly impossible but replacing it > > in a single application is conceivable, depending on what the app does. And > > in this case I was able to do this: > > a) Leave the existing fork call there > > b) Fall back on IO.popen for operating systems where fork is not > > supported > > What I did is for each worker I launch (instead of fork) a new child > > process, pass the read/write port numbers to the child on the command line, > > and wait for it to connect to them. > > > > Because Rails takes a long time to start up on Windows (11 seconds, ugh) > > you can wait a pretty long time before all your workers are ready (11sec * > > (number of workers+2)). But still, it's better than not being able to use it > > at all. It's perfectly acceptable for a development machine, which is how I > > use Rails on Windows. (My project is going to deploy on something that ends > > in nix.) > > > > On the other hand if you're going to be dynamically creating background > > workers (e.g. reload_on_schedule true or set_no_auto_load true) you'll wait > > a while for the worker to start up on Windows. > > > > Will this slow startup time ever block a Rails server request? Not that > > I've seen. From what I've seen BackgrounDRb never starts a new worker > > process because of an API call. The only case I've seen where it starts a > > new worker process on the fly is for a scheduled background task using the > > cron feature. But I may have missed something. > > > > Another thing that I didn't implement (at least not for now): support > > for the "start" parameter to run the background processes as hidden > > processes. For now on Windows you just have to run script/backgroundrb > > without the start parameter and just leave it running in a console. (But I > > did make the error message nicer.) The reason I didn't: Ruby does not seem > > to have implemented a way to kill a process tree in Windows. Windows doesn't > > have a single API to kill a process tree. You can add all the processes to a > > Job and kill that, or you can enumerate the descendants of a process and > > kill them all one by one. I just didn't want to bother adding the code to > > Ruby do that, at least not now, because it works well enough in a console. > > > > And the nice thing is Ctrl+C from the console does indeed kill the whole > > process tree. > > > > That's it for now. Please let me know how to proceed so us Windows users > > can benefit from the wonderfulness of BackgrounDRb. > > > > -- > > Brian Morearty > > > > > > > > -- > Brian -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080413/562cb0f5/attachment.html From siebertm85 at googlemail.com Mon Apr 14 03:03:53 2008 From: siebertm85 at googlemail.com (Michael Siebert) Date: Mon, 14 Apr 2008 09:03:53 +0200 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows In-Reply-To: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> References: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> Message-ID: Brain, this is great news (although i use windows only for playing games). when it comes to fork, i can remember from my before-mac-time that there is a rubyforge project called win32-utils (http://rubyforge.org/projects/win32utils/ ) which implements things like fork and possibly killing of a process tree. the thing with the tests ist: backgroundrb doesnt seem to have tests. include them :) Am 14.04.2008 um 02:44 schrieb Brian Morearty: > Hello all, > > I have some good news to announce. I have made some changes to a > local copy of BackgrounDRb 1.0.3 (actually most of the changes were > in Packet 0.1.5) and I've got it working on Windows. From what I've > read, this is the first time it's worked on Windows in a year and a > half, since 0.2 was released. > > In my research I can see that a number of people have asked for > Windows support so I'm excited that I can help out. > > I'm new to this mailing list and new to BackgrounDRb so I would > appreciate advice and help from you all on things like: > - how to test my changes. When I run rake it just exits without > running any tests. I'm not sure why. I also don't know how to test > Packet. > - how to send a patch. I mostly do Windows development where textual > patches are not commonplace. But I did submit a patch to Rails that > was accepted, so at least I've done it once with success. :-) > - a code review to make sure I'm doing things in an approved way > - someone with *nix to make sure I didn't break anything by > accident. I tried hard to not but you can't be sure without testing. > > Since I haven't figured out how to run automated tests I have > created some ad hoc tests of my own. These cases work: > - ad hoc scheduling by calling a method on the worker > - passing parameters to the worker > - cron scheduling > - register_status and ask_status > - making a synchronous call and getting a result back kind of works, > except I get the whole hash back with the result in the :data key > instead of just getting the result alone. In other words I get a > hash like this: > {:type=>:response, :client_signature=>25, :result=>true, :data=>1} > which is odd because it's calling the same "extract" function that > ask_status uses, so I'm not sure why this is any different. > > There are some things I haven't tried yet but it's great that this > many cases work. > > I definitely have to try to figure out why the return value of a > synchronous call is the whole hash, and I might need help on that > because I'm mystified. > > The primary obstacles to making BackgrounDRb work on Windows were: > 1. UNIXSocket - it wasn't too hard to add code that uses TCPSocket > unless defined? UNIXSocket. And there were four lines of code that > checked if a socket was a UNIXSocket to decide whether it's an > "internal" read or write, but those lines were crashing because > UNIXSocket isn't even defined on Windows, much less used. My fix for > that was to create a marker Module called "Packet::InternalSocket." > Each time I created an internal socket (where the code used to > create a UNIXSocket) I now call socket.extend Packet::InternalSocket > so that later on the code can distinguish the internal ones from the > others. > 2. fork - well that's a harder matter. Windows doesn't support > fork() and probably never will. I saw a recent post by Hemant Kumar > mentioning a fix that uses fork and exec rather than just fork, but > it still requires fork. Simulating fork in a generic way is nearly > impossible but replacing it in a single application is conceivable, > depending on what the app does. And in this case I was able to do > this: > a) Leave the existing fork call there > b) Fall back on IO.popen for operating systems where fork is not > supported > What I did is for each worker I launch (instead of fork) a new > child process, pass the read/write port numbers to the child on the > command line, and wait for it to connect to them. > > Because Rails takes a long time to start up on Windows (11 seconds, > ugh) you can wait a pretty long time before all your workers are > ready (11sec * (number of workers+2)). But still, it's better than > not being able to use it at all. It's perfectly acceptable for a > development machine, which is how I use Rails on Windows. (My > project is going to deploy on something that ends in nix.) > > On the other hand if you're going to be dynamically creating > background workers (e.g. reload_on_schedule true or set_no_auto_load > true) you'll wait a while for the worker to start up on Windows. > > Will this slow startup time ever block a Rails server request? Not > that I've seen. From what I've seen BackgrounDRb never starts a new > worker process because of an API call. The only case I've seen where > it starts a new worker process on the fly is for a scheduled > background task using the cron feature. But I may have missed > something. > > Another thing that I didn't implement (at least not for now): > support for the "start" parameter to run the background processes as > hidden processes. For now on Windows you just have to run script/ > backgroundrb without the start parameter and just leave it running > in a console. (But I did make the error message nicer.) The reason I > didn't: Ruby does not seem to have implemented a way to kill a > process tree in Windows. Windows doesn't have a single API to kill a > process tree. You can add all the processes to a Job and kill that, > or you can enumerate the descendants of a process and kill them all > one by one. I just didn't want to bother adding the code to Ruby do > that, at least not now, because it works well enough in a console. > > And the nice thing is Ctrl+C from the console does indeed kill the > whole process tree. > > That's it for now. Please let me know how to proceed so us Windows > users can benefit from the wonderfulness of BackgrounDRb. > > -- > Brian Morearty > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel --- Siebert WD Michael Siebert Trappenweg 6, 15749 Mittenwalde, Deutschland From kiran.soumya at gmail.com Mon Apr 14 05:08:10 2008 From: kiran.soumya at gmail.com (Kiran Soumya) Date: Mon, 14 Apr 2008 14:38:10 +0530 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on windows Message-ID: <471d7b3a0804140208q2a76abddmb79aa6ff626426b2@mail.gmail.com> Brian, If you have want to submit a patch to backgroundrb and want to contribute, you need to create an account with gitorious to have commit access. But before that please make sure that you are done with perfect set of test cases. Kindly refer to the organized test cases for backgroundrb [ http://gitorious.org/projects/backgroundrb/repos/mainline/trees/master/test] inorder to follow the same coding style. You need get it tested in all versions of windows too... 98,Xp,Vista for confirmation so that you can create a perfect benchmark of your code. If you need help in testing or benchmarking or remodelling your code to suit the exact requirements, then you have to contact hemant for help or else keep going... Regards, Kiran. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080414/ef5a57b1/attachment-0001.html From paul.kmiec at appfolio.com Mon Apr 14 15:23:23 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Mon, 14 Apr 2008 12:23:23 -0700 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows In-Reply-To: <3f8e47bb0804131907r790a47c3u209393a4a802a707@mail.gmail.com> References: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> <3f8e47bb0804131903g30ff691di4107cd196fe4b065@mail.gmail.com> <3f8e47bb0804131907r790a47c3u209393a4a802a707@mail.gmail.com> Message-ID: <674a03dc0804141223ie077945wc5545db4bff0d43a@mail.gmail.com> Yeah, I am not sure either. Truthfully, I have not used git before and haven't quite grasped it yet. Seems projects (like backgroundrb, RoR) are starting to move their code trees to git while maintaining svn access. Paul On Sun, Apr 13, 2008 at 7:07 PM, Brian Morearty wrote: > P.S. Hemant I did see your note "(I am looking at you Brian too)" in the > other thread where you mentioned anyone interested in hacking at Packet can > start with that Git repository. And my name's Brian so maybe you meant me. > :-) > > I'm just not sure what I'm supposed to do there. Just check it in? (Is that > repository a playground that's open to anyone?) > > Thanks again. > > - Brian > > > > On Sun, Apr 13, 2008 at 7:03 PM, Brian Morearty wrote: > > > > > I have a couple of patch files I can send the the appropriate place. One > for packet and one for backgroundrb. Am I supposed to send them to this list > for review, email them to hemant, request write access to a SubVersion or > Git repository, or something else? > > > > I need a little handholding with this part so please be patient. :-) I > haven't participated in open source very much so I'm not sure what the > conventions are. > > > > - Brian > > > > > > > > > > > > On Sun, Apr 13, 2008 at 5:44 PM, Brian Morearty > wrote: > > > > > > > > Hello all, > > > > > > I have some good news to announce. I have made some changes to a local > copy of BackgrounDRb 1.0.3 (actually most of the changes were in Packet > 0.1.5) and I've got it working on Windows. From what I've read, this is the > first time it's worked on Windows in a year and a half, since 0.2 was > released. > > > > > > > > > In my research I can see that a number of people have asked for Windows > support so I'm excited that I can help out. > > > > > > I'm new to this mailing list and new to BackgrounDRb so I would > appreciate advice and help from you all on things like: > > > - how to test my changes. When I run rake it just exits without running > any tests. I'm not sure why. I also don't know how to test Packet. > > > - how to send a patch. I mostly do Windows development where textual > patches are not commonplace. But I did submit a patch to Rails that was > accepted, so at least I've done it once with success. :-) > > > - a code review to make sure I'm doing things in an approved way > > > - someone with *nix to make sure I didn't break anything by accident. I > tried hard to not but you can't be sure without testing. > > > > > > Since I haven't figured out how to run automated tests I have created > some ad hoc tests of my own. These cases work: > > > > > > - ad hoc scheduling by calling a method on the worker- passing > parameters to the worker > > > - cron scheduling > > > - register_status and ask_status > > > - making a synchronous call and getting a result back kind of works, > except I get the whole hash back with the result in the :data key instead of > just getting the result alone. In other words I get a hash like this: > > > {:type=>:response, :client_signature=>25, :result=>true, :data=>1} > > > which is odd because it's calling the same "extract" function that > ask_status uses, so I'm not sure why this is any different. > > > > > > There are some things I haven't tried yet but it's great that this many > cases work. > > > > > > I definitely have to try to figure out why the return value of a > synchronous call is the whole hash, and I might need help on that because > I'm mystified. > > > > > > The primary obstacles to making BackgrounDRb work on Windows were: > > > 1. UNIXSocket - it wasn't too hard to add code that uses TCPSocket > unless defined? UNIXSocket. And there were four lines of code that checked > if a socket was a UNIXSocket to decide whether it's an "internal" read or > write, but those lines were crashing because UNIXSocket isn't even defined > on Windows, much less used. My fix for that was to create a marker Module > called "Packet::InternalSocket." Each time I created an internal socket > (where the code used to create a UNIXSocket) I now call socket.extend > Packet::InternalSocket so that later on the code can distinguish the > internal ones from the others. > > > 2. fork - well that's a harder matter. Windows doesn't support fork() > and probably never will. I saw a recent post by Hemant Kumar mentioning a > fix that uses fork and exec rather than just fork, but it still requires > fork. Simulating fork in a generic way is nearly impossible but replacing it > in a single application is conceivable, depending on what the app does. And > in this case I was able to do this: > > > a) Leave the existing fork call there > > > b) Fall back on IO.popen for operating systems where fork is not > supported > > > What I did is for each worker I launch (instead of fork) a new child > process, pass the read/write port numbers to the child on the command line, > and wait for it to connect to them. > > > > > > Because Rails takes a long time to start up on Windows (11 seconds, ugh) > you can wait a pretty long time before all your workers are ready (11sec * > (number of workers+2)). But still, it's better than not being able to use it > at all. It's perfectly acceptable for a development machine, which is how I > use Rails on Windows. (My project is going to deploy on something that ends > in nix.) > > > > > > On the other hand if you're going to be dynamically creating background > workers (e.g. reload_on_schedule true or set_no_auto_load true) you'll wait > a while for the worker to start up on Windows. > > > > > > Will this slow startup time ever block a Rails server request? Not that > I've seen. From what I've seen BackgrounDRb never starts a new worker > process because of an API call. The only case I've seen where it starts a > new worker process on the fly is for a scheduled background task using the > cron feature. But I may have missed something. > > > > > > Another thing that I didn't implement (at least not for now): support > for the "start" parameter to run the background processes as hidden > processes. For now on Windows you just have to run script/backgroundrb > without the start parameter and just leave it running in a console. (But I > did make the error message nicer.) The reason I didn't: Ruby does not seem > to have implemented a way to kill a process tree in Windows. Windows doesn't > have a single API to kill a process tree. You can add all the processes to a > Job and kill that, or you can enumerate the descendants of a process and > kill them all one by one. I just didn't want to bother adding the code to > Ruby do that, at least not now, because it works well enough in a console. > > > > > > And the nice thing is Ctrl+C from the console does indeed kill the whole > process tree. > > > > > > That's it for now. Please let me know how to proceed so us Windows users > can benefit from the wonderfulness of BackgrounDRb. > > > > > > -- > > > Brian Morearty > > > > > > > > > > > -- > > Brian > > > > -- > Brian > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From leavengood at gmail.com Mon Apr 14 15:39:58 2008 From: leavengood at gmail.com (Ryan Leavengood) Date: Mon, 14 Apr 2008 15:39:58 -0400 Subject: [Backgroundrb-devel] Good news: BackgrounDRb now works on Windows In-Reply-To: <674a03dc0804141223ie077945wc5545db4bff0d43a@mail.gmail.com> References: <3f8e47bb0804131744j1100a704h1089186bc831f0fe@mail.gmail.com> <3f8e47bb0804131903g30ff691di4107cd196fe4b065@mail.gmail.com> <3f8e47bb0804131907r790a47c3u209393a4a802a707@mail.gmail.com> <674a03dc0804141223ie077945wc5545db4bff0d43a@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 3:23 PM, Paul Kmiec wrote: > Yeah, I am not sure either. Truthfully, I have not used git before and > haven't quite grasped it yet. Seems projects (like backgroundrb, RoR) > are starting to move their code trees to git while maintaining svn > access. Git is really worth learning. I'm only just beginning with it myself, but I plan to move all my projects to it eventually. It has nice SVN integration so you can actually play with it against an SVN repo. Of course GitHub.com or gitorious.org have plenty of interesting projects to play with too. Search video.google.com for a couple Google Tech Talks about Git, one from Linus himself (more about why he created it) and one from Randal Swartz that goes into more detail about how to use it. The Git manual is quite good and can get you going fairly quick, but also delves into some detail: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html There are also lots of other articles and screencasts on the blogosphere. This is a great (and free) one about using git on a Rails project: http://www.vimeo.com/369095 Regards, Ryan From brian at morearty.org Tue Apr 15 13:29:22 2008 From: brian at morearty.org (Brian Morearty) Date: Tue, 15 Apr 2008 10:29:22 -0700 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue In-Reply-To: <1208138039.14339.19.camel@shire> References: <674a03dc0804131641x3f66a918xf49c8d239f7af9c6@mail.gmail.com> <1208138039.14339.19.camel@shire> Message-ID: <3f8e47bb0804151029p789a3ae9paeef6f07786a4898@mail.gmail.com> > One option we have is to have "packet_worker_runner" as installed executable when user installs packet gem. The more I think about the idea of a packet runner, the more I like it. One reason I like it: it solves a problem with the Windows port I wrote. The Windows implementation needs to know what process to launch to create a new worker. It currently relaunches PACKET_APP, but I was concerned that might be a misuse of PACKET_APP and it forces the author of a PACKET_APP to implement code like what I wrote in script/backgroundrb to handle a "worker parameter. If there's a separate app whose only purpose in life is to be the runner, no one else has to write code to handle those parameters. - Brian On Sun, Apr 13, 2008 at 6:53 PM, hemant kumar wrote: > > On Sun, 2008-04-13 at 16:41 -0700, Paul Kmiec wrote: > > > This help significantly in improving stability of > > BackgrounDRb and code > > > written on top of packet. So what you guys think? Shall we > > go for this > > > change? > > > > It sounds good to me. Anything that helps stability and > > improves > > memory usage will always get my vote. > > > > I agree. > > > > Paul > > Okay folks, > > I have updated git repo of just packet with latest changes wherein > I have removed fork in favour of fork and exec: > > http://github.com/gnufied/packet/tree/master > > The only catch is, now we need some kinda runner to run our workers > since exec takes argument in form of string only. For example: > > exec "packet_worker_runner > ##{t_worker_name}:{worker_read_end.fileno}:#{worker_write_end.fileno}:#{option_dump_length}" > > One option we have is to have "packet_worker_runner" as installed > executable when user installs packet gem. > > Any other ideas? > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080415/873e2d17/attachment.html From paul.kmiec at appfolio.com Tue Apr 15 18:29:12 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Tue, 15 Apr 2008 15:29:12 -0700 Subject: [Backgroundrb-devel] how to do and how not to do (in the background) Message-ID: <674a03dc0804151529s4618bef5s2cc6230e960df8bb@mail.gmail.com> Hi everyone, I recently upgraded to backgroundrb 1.0.3 (which is much more stable than the old drb version). However, I am not sure whether backgroundrb is the right solution nor whether I am using it the right way. For a while now, I have followed various discussions out there about background-fu, spawn, workling, startling, ap4r, sqs etc. Seems the world of background processing is quite undecided. I think the best way to start is to describe the three main use cases (I think they are pretty universal). Feedback is highly appreciated. Periodic Tasks - automatic tasks that execute at specific intervals or certain times of day (think cron as in * */4 * * *). Example is charging monthly subscriptions or sending nightly digest emails. Background Tasks with status - user initiated tasks that need to execute outside of the request / response for which the client periodically check their status. Example is charging credit card at checkout or generating a report. Background Tasks without status - the spawn and forget tasks. Example is solr commit. In our world, we have multiple mongrels spread over more than one physical machine for hardware redudancy. In addition, we have monitoring (in my case monit) that watches processes and restarts them if they crash or exceed limits. Having all of this in mind, I currently have the following: I have backgroundrb running on multiple machines. I don't use MiddleMan in RoR because that doesn't deal with a backgroundrb or a worker being down. Instead, I use the database to communicate between RoR and the workers (I suppose home grown queue). RoR inserts a job to be done. Workers polls the database, lock the job to ensure only one worker executes the job, execute the jobs, and update the job progress / status. This essentially handles background tasks. For periodic tasks, I have scheduled backgroundrb workers. Some of them just perform an action like delete temporary files. Other ones query the database to see what to do (i.e. who to charge monthly subscription or who should receive the digest email). Depending on the task, there is some status that tells who was already charged or who already received the digest email to avoid doing something twice. Does that sound reasonable? Couple additional things I am not sure about ... I mentioned I use monit to stop/stop/monitor backgroundrb server. But then the backgroundrb server spawns off workers. At this point, I don't see how to monitor a worker and restart it were it to die. One workaround I can think off is to have a periodic worker that is reloaded by backgroundrb server that starts the other workers (MiddleMan.start_worker will not create a new worker if one with same task name and job key already exists). Can this same worker tell me another worker has died (i.e. MiddleMan.all_worker_info and ensure all workers are :running)? Paul ps. I always see people talking trash about backgroundrb because of its previous implementation ... and there is hemant always trying to fend off the assaults :). Good job! From tawan.sierek at gmail.com Wed Apr 16 08:00:58 2008 From: tawan.sierek at gmail.com (Tawan Sierek) Date: Wed, 16 Apr 2008 14:00:58 +0200 Subject: [Backgroundrb-devel] Loading Rails Environment Message-ID: Hi, I pulled commit 00ecd76da9a0df5f8ec0836c462b6e9a1a0f3ba2 and ran rake backgroundrb:setup to replace my current script/backgroundrb file. I tried to start the server, but the log says: /backgroundrb/server/lib/master_worker.rb:288:in `load_rails_env': uninitialized constant BackgrounDRb::MasterProxy::ActiveRecord (NameError) I dont really get it how the rails environment is loaded, though I tried to figure out whats wrong, but I gave up after while. Am I the only one who runs in this error? Thanx for ya help tawan -- Tawan Sierek tawan at sierek.at -- Phorusgasse 10/7 - 1040 Vienna Mobile: ++43 650 700 20 12 Skype: Tawan Sierek From gethemant at gmail.com Wed Apr 16 11:23:05 2008 From: gethemant at gmail.com (hemant kumar) Date: Wed, 16 Apr 2008 20:53:05 +0530 Subject: [Backgroundrb-devel] Loading Rails Environment In-Reply-To: References: Message-ID: <1208359385.6874.0.camel@shire> On Wed, 2008-04-16 at 14:00 +0200, Tawan Sierek wrote: > Hi, > > I pulled commit 00ecd76da9a0df5f8ec0836c462b6e9a1a0f3ba2 and ran rake > backgroundrb:setup to replace my current script/backgroundrb file. > I tried to start the server, but the log says: > /backgroundrb/server/lib/master_worker.rb:288:in `load_rails_env': > uninitialized constant BackgrounDRb::MasterProxy::ActiveRecord > (NameError) > > I dont really get it how the rails environment is loaded, though I > tried to figure out whats wrong, but I gave up after while. > Am I the only one who runs in this error? > Git repository is in flux and is broken right now. Please use SVN copy. From rogboone at yahoo.com Fri Apr 18 14:07:11 2008 From: rogboone at yahoo.com (Roggie Boone) Date: Fri, 18 Apr 2008 11:07:11 -0700 (PDT) Subject: [Backgroundrb-devel] [Newbie] Trouble with methods? Message-ID: <245506.72667.qm@web32505.mail.mud.yahoo.com> Hi, I'm new to BackgroundRB and am having some trouble figuring out what must be a simple thing. I'm in time crunch where I need my Rails app to generate PDF files on the fly for my website users. But before I leap into that, I'm trying to get familiar with the basics. So I have this worker in my rails app lib/worker folder: ============== class PdfmakerWorker < BackgrounDRb::MetaWorker set_worker_name :pdfmaker_worker set_no_auto_load(true) def create(args = nil) # this method is called, when worker is loaded for the first time register_status(:cnt => 0) 1.upto 100000000 do |x| if x % 1000 == 0 register_status(:cnt => x) end end #exit end def getres res = "Just a test" return res end end ============================================================ I've written a simple program to test that worker and it is as follows: jk = MiddleMan.new_worker(:worker => :pdfmaker_worker, :job_key => "123", :data => nil) sleep (5) count = 0 while count < 1 pct = MiddleMan.ask_status(:worker => :pdfmaker_worker, :job_key => jk) sleep(2) puts "Count is: " + (pct[:cnt]).to_s res = MiddleMan.worker(:pdfmaker_worker, jk).getres puts "Res is: " + res.to_s end ============================================================ I run script/console and load the above program. The worker counts up to 10000000 and my little test program repeatedly queries the worker and writes "Count is " to the screen as it should. However, when it get to my custom method "getres" in the last two lines of the test program, the return result for "res" is nil. The worker doesn't crash, it just appears not to return a result. I'm sure it's something simple, but I don't see why. Here is a sample output of one iteration of the simple program: ================ Count is: 970000000 {:type=>:do_work, :worker=>:pdfmaker_worker, :worker_method=>:getres, :job_key=>"123"} Res is: {:type=>:get_status, :worker=>:pdfmaker_worker, :job_key=>"123"} ================ It looks like the call to "getres" is being recognized, but I'm obviously missing some critical link. My setup: Ubuntu 7.10 Ruby 1.8.6 Rails 2.0.2 BackgroundRB (been through several versions, latest of which was retrieved via svn on 4/17/08 and I updated the backgroundrb script as per install guidelines) Thanks in advance for your help. Rogelio --------------------------------- Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080418/ed9dca84/attachment-0001.html From brian at morearty.org Fri Apr 18 15:26:25 2008 From: brian at morearty.org (Brian Morearty) Date: Fri, 18 Apr 2008 12:26:25 -0700 Subject: [Backgroundrb-devel] [Newbie] Trouble with methods? In-Reply-To: <245506.72667.qm@web32505.mail.mud.yahoo.com> References: <245506.72667.qm@web32505.mail.mud.yahoo.com> Message-ID: <3f8e47bb0804181226r137c80dbvd02f735a4733703f@mail.gmail.com> Sorry, I meant to reply-all. Here's what I wrote: Hello Rogelio, If you want a return value you have to call it synchronously. The default call is asynchronous--because after all, it's a background call. That's why you're getting nil back. To make a synchronous call, pass "true" as the 2nd argument to your worker: res = MiddleMan.worker(:pdfmaker_worker, jk).getres(nil, true) In this example I used nil for the first argument because your getres function doesn't take an argument. For more info see http://backgroundrb.rubyforge.org/rails/index.html and read the section "Invoke a method on worker and get results." - Brian Morearty On Fri, Apr 18, 2008 at 11:07 AM, Roggie Boone wrote: > Hi, > > I'm new to BackgroundRB and am having some trouble figuring > out what must be a simple thing. I'm in time crunch where I > need my Rails app to generate PDF files on the fly for my website > users. But before I leap into that, I'm trying to get familiar with > the basics. So I have this worker in my rails app lib/worker folder: > ============== > class PdfmakerWorker < BackgrounDRb::MetaWorker > set_worker_name :pdfmaker_worker > set_no_auto_load(true) > def create(args = nil) > # this method is called, when worker is loaded for the first time > register_status(:cnt => 0) > 1.upto 100000000 do |x| > if x % 1000 == 0 > register_status(:cnt => x) > end > end > #exit > end > > def getres > res = "Just a test" > return res > end > end > > ============================================================ > > I've written a simple program to test that worker and it is as follows: > > jk = MiddleMan.new_worker(:worker => :pdfmaker_worker, :job_key => "123", > :data => nil) > > sleep (5) > > count = 0 > > while count < 1 > pct = MiddleMan.ask_status(:worker => :pdfmaker_worker, :job_key => jk) > sleep(2) > puts "Count is: " + (pct[:cnt]).to_s > > res = MiddleMan.worker(:pdfmaker_worker, jk).getres > puts "Res is: " + res.to_s > end > ============================================================ > > I run script/console and load the above program. The worker counts up > to 10000000 and my little test program repeatedly queries the worker > and writes "Count is " to the screen as it should. > > However, when it get to my custom method "getres" in the last > two lines of the test program, the return result for "res" is nil. The > worker doesn't crash, it just appears not to return a result. I'm sure it's > something simple, but I don't see why. > > Here is a sample output of one iteration of the simple program: > ================ > Count is: 970000000 > {:type=>:do_work, :worker=>:pdfmaker_worker, :worker_method=>:getres, > :job_key=>"123"} > Res is: > {:type=>:get_status, :worker=>:pdfmaker_worker, :job_key=>"123"} > ================ > > It looks like the call to "getres" is being recognized, but I'm obviously > missing some critical link. > > My setup: > > Ubuntu 7.10 > Ruby 1.8.6 > Rails 2.0.2 > BackgroundRB (been through several versions, latest of > which was retrieved via svn on 4/17/08 and I updated the > backgroundrb script as per install guidelines) > > Thanks in advance for your help. > > Rogelio > > ------------------------------ > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it > now. > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080418/17d8228e/attachment.html From gethemant at gmail.com Fri Apr 18 16:06:52 2008 From: gethemant at gmail.com (hemant) Date: Sat, 19 Apr 2008 01:36:52 +0530 Subject: [Backgroundrb-devel] Cutting down BackgrounDRb memory issue In-Reply-To: <3f8e47bb0804151029p789a3ae9paeef6f07786a4898@mail.gmail.com> References: <674a03dc0804131641x3f66a918xf49c8d239f7af9c6@mail.gmail.com> <1208138039.14339.19.camel@shire> <3f8e47bb0804151029p789a3ae9paeef6f07786a4898@mail.gmail.com> Message-ID: On Tue, Apr 15, 2008 at 10:59 PM, Brian Morearty wrote: > > > One option we have is to have "packet_worker_runner" as installed > executable when user installs packet gem. > > The more I think about the idea of a packet runner, the more I like it. > > One reason I like it: it solves a problem with the Windows port I wrote. The > Windows implementation needs to know what process to launch to create a new > worker. It currently relaunches PACKET_APP, but I was concerned that might > be a misuse of PACKET_APP and it forces the author of a PACKET_APP to > implement code like what I wrote in script/backgroundrb to handle a "worker > parameter. If there's a separate app whose only purpose in life is to be the > runner, no one else has to write code to handle those parameters. > Brian, Just a quick update. I have updated both BackgrounDRb and packet with new changes of fork, exec and runner binaries.I will start merging your changes and making it run on Windows sometime tomorrow. Also, this is our opportunity to have 100% test case coverage of BackgrounDRb. If you can be online on IRC tomorrow, it will be awesome. Time is around 19th April, 18.23 UTC. From rogboone at yahoo.com Fri Apr 18 16:14:20 2008 From: rogboone at yahoo.com (Roggie Boone) Date: Fri, 18 Apr 2008 13:14:20 -0700 (PDT) Subject: [Backgroundrb-devel] [Newbie] Trouble with methods? In-Reply-To: <3f8e47bb0804181226r137c80dbvd02f735a4733703f@mail.gmail.com> Message-ID: <610633.8372.qm@web32503.mail.mud.yahoo.com> Yes, thank you! I had read about the optional 2nd argument but misunderstood its usefulness. It's working now. But I also learned that the return result is not just my string but a hash containing my string in the "data" hash member. Rog Brian Morearty wrote: Sorry, I meant to reply-all. Here's what I wrote: Hello Rogelio, If you want a return value you have to call it synchronously. The default call is asynchronous--because after all, it's a background call. That's why you're getting nil back. To make a synchronous call, pass "true" as the 2nd argument to your worker: res = MiddleMan.worker(:pdfmaker_worker, jk).getres(nil, true) In this example I used nil for the first argument because your getres function doesn't take an argument. For more info see http://backgroundrb.rubyforge.org/rails/index.html and read the section "Invoke a method on worker and get results." - Brian Morearty On Fri, Apr 18, 2008 at 11:07 AM, Roggie Boone wrote: Hi, I'm new to BackgroundRB and am having some trouble figuring out what must be a simple thing. I'm in time crunch where I need my Rails app to generate PDF files on the fly for my website users. But before I leap into that, I'm trying to get familiar with the basics. So I have this worker in my rails app lib/worker folder: ============== class PdfmakerWorker < BackgrounDRb::MetaWorker set_worker_name :pdfmaker_worker set_no_auto_load(true) def create(args = nil) # this method is called, when worker is loaded for the first time register_status(:cnt => 0) 1.upto 100000000 do |x| if x % 1000 == 0 register_status(:cnt => x) end end #exit end def getres res = "Just a test" return res end end ============================================================ I've written a simple program to test that worker and it is as follows: jk = MiddleMan.new_worker(:worker => :pdfmaker_worker, :job_key => "123", :data => nil) sleep (5) count = 0 while count < 1 pct = MiddleMan.ask_status(:worker => :pdfmaker_worker, :job_key => jk) sleep(2) puts "Count is: " + (pct[:cnt]).to_s res = MiddleMan.worker(:pdfmaker_worker, jk).getres puts "Res is: " + res.to_s end ============================================================ I run script/console and load the above program. The worker counts up to 10000000 and my little test program repeatedly queries the worker and writes "Count is " to the screen as it should. However, when it get to my custom method "getres" in the last two lines of the test program, the return result for "res" is nil. The worker doesn't crash, it just appears not to return a result. I'm sure it's something simple, but I don't see why. Here is a sample output of one iteration of the simple program: ================ Count is: 970000000 {:type=>:do_work, :worker=>:pdfmaker_worker, :worker_method=>:getres, :job_key=>"123"} Res is: {:type=>:get_status, :worker=>:pdfmaker_worker, :job_key=>"123"} ================ It looks like the call to "getres" is being recognized, but I'm obviously missing some critical link. My setup: Ubuntu 7.10 Ruby 1.8.6 Rails 2.0.2 BackgroundRB (been through several versions, latest of which was retrieved via svn on 4/17/08 and I updated the backgroundrb script as per install guidelines) Thanks in advance for your help. Rogelio --------------------------------- Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. _______________________________________________ Backgroundrb-devel mailing list Backgroundrb-devel at rubyforge.org http://rubyforge.org/mailman/listinfo/backgroundrb-devel -- Brian --------------------------------- Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080418/2a140a30/attachment.html From brian at morearty.org Fri Apr 18 17:18:35 2008 From: brian at morearty.org (Brian Morearty) Date: Fri, 18 Apr 2008 14:18:35 -0700 Subject: [Backgroundrb-devel] [Newbie] Trouble with methods? In-Reply-To: <610633.8372.qm@web32503.mail.mud.yahoo.com> References: <3f8e47bb0804181226r137c80dbvd02f735a4733703f@mail.gmail.com> <610633.8372.qm@web32503.mail.mud.yahoo.com> Message-ID: <3f8e47bb0804181418x11b909ecs8752b8b4ad03b6d9@mail.gmail.com> > But I also learned that the return result is not just my string but a hash containing my string in the "data" hash member. That's useful for me to know because that's what was happening to me when I ported BackgrounDRb to Windows and I kept wondering if it was a bug on my side. But I guess that's what (the current version of) BackgrounDRb does on all platforms, so it's not just me. (I hadn't tried it on another platform because all I have for now is Windows.) - Brian On 4/18/08, Roggie Boone wrote: > > Yes, thank you! I had read about the optional 2nd argument > but misunderstood its usefulness. It's working now. But I also > learned that the return result is not just my string but a hash > containing my string in the "data" hash member. > > Rog > > *Brian Morearty * wrote: > > Sorry, I meant to reply-all. Here's what I wrote: > > Hello Rogelio, > > If you want a return value you have to call it synchronously. The default > call is asynchronous--because after all, it's a background call. That's why > you're getting nil back. > > To make a synchronous call, pass "true" as the 2nd argument to your > worker: > > res = MiddleMan.worker(:pdfmaker_worker, jk).getres(nil, true) > > In this example I used nil for the first argument because your getres > function doesn't take an argument. > > For more info see http://backgroundrb.rubyforge.org/rails/index.html and > read the section "Invoke a method on worker and get results." > > - Brian Morearty > > > > > On Fri, Apr 18, 2008 at 11:07 AM, Roggie Boone wrote: > > > Hi, > > > > I'm new to BackgroundRB and am having some trouble figuring > > out what must be a simple thing. I'm in time crunch where I > > need my Rails app to generate PDF files on the fly for my website > > users. But before I leap into that, I'm trying to get familiar with > > the basics. So I have this worker in my rails app lib/worker folder: > > ============== > > class PdfmakerWorker < BackgrounDRb::MetaWorker > > set_worker_name :pdfmaker_worker > > set_no_auto_load(true) > > def create(args = nil) > > # this method is called, when worker is loaded for the first time > > register_status(:cnt => 0) > > 1.upto 100000000 do |x| > > if x % 1000 == 0 > > register_status(:cnt => x) > > end > > end > > #exit > > end > > > > def getres > > res = "Just a test" > > return res > > end > > end > > > > ============================================================ > > > > I've written a simple program to test that worker and it is as follows: > > > > jk = MiddleMan.new_worker(:worker => :pdfmaker_worker, :job_key => > > "123", :data => nil) > > > > sleep (5) > > > > count = 0 > > > > while count < 1 > > pct = MiddleMan.ask_status(:worker => :pdfmaker_worker, :job_key => > > jk) > > sleep(2) > > puts "Count is: " + (pct[:cnt]).to_s > > > > res = MiddleMan.worker(:pdfmaker_worker, jk).getres > > puts "Res is: " + res.to_s > > end > > ============================================================ > > > > I run script/console and load the above program. The worker counts up > > to 10000000 and my little test program repeatedly queries the worker > > and writes "Count is " to the screen as it should. > > > > However, when it get to my custom method "getres" in the last > > two lines of the test program, the return result for "res" is nil. The > > worker doesn't crash, it just appears not to return a result. I'm sure it's > > something simple, but I don't see why. > > > > Here is a sample output of one iteration of the simple program: > > ================ > > Count is: 970000000 > > {:type=>:do_work, :worker=>:pdfmaker_worker, :worker_method=>:getres, > > :job_key=>"123"} > > Res is: > > {:type=>:get_status, :worker=>:pdfmaker_worker, :job_key=>"123"} > > ================ > > > > It looks like the call to "getres" is being recognized, but I'm > > obviously > > missing some critical link. > > > > My setup: > > > > Ubuntu 7.10 > > Ruby 1.8.6 > > Rails 2.0.2 > > BackgroundRB (been through several versions, latest of > > which was retrieved via svn on 4/17/08 and I updated the > > backgroundrb script as per install guidelines) > > > > Thanks in advance for your help. > > > > Rogelio > > > > ------------------------------ > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try > > it now. > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Brian > > > ------------------------------ > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it > now. > > -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080418/88af9644/attachment-0001.html From brian at morearty.org Fri Apr 18 17:23:32 2008 From: brian at morearty.org (Brian Morearty) Date: Fri, 18 Apr 2008 14:23:32 -0700 Subject: [Backgroundrb-devel] Return result from synchronous call is a hash containing the result in the "data" hash member. Message-ID: <3f8e47bb0804181423ofd45ff6w2f8cf61d8cc1c132@mail.gmail.com> When calling a BackgrounDRb function synchronously (putting true in the 2nd param), the return result is a hash containing the result in the :data hash member. Question for people in the know: it should just be the return value itself, right? Not a hash with the return value in the :data member? (Perhaps this is a bug with the current version?) -- Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080418/6df5e748/attachment.html From raghu.srinivasan at gmail.com Sun Apr 20 22:03:33 2008 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Sun, 20 Apr 2008 19:03:33 -0700 Subject: [Backgroundrb-devel] Has anyone seen this error message? Message-ID: <6482c06a0804201903u6013751evfc04217f07b4fbbf@mail.gmail.com> Hi all, I have a background job that parses RSS Feeds. This works just fine when called from a batch process in a loop, but when triggered by a user i/p, it *always* craps out with this error message. ========================== /usr/lib/ruby/1.8/uri/common.rb:436:in `split': bad URI(is not URI?): http://rss.netflix.com/ReviewsRSS?id=XYZ123 (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/common.rb:485:in `parse' from /usr/lib/ruby/1.8/open-uri.rb:29:in `open' from /home/raghus/public_html/feedflix.com/ff/lib/netflix.rb:467:in`process_netflix_reviews' from /home/raghus/public_html/ feedflix.com/ff/lib/workers/netflix_worker.rb:48:in `parse_netflix_feeds' from /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:235:in`send' from /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:235:in`process_request' from /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:219:in`receive_data' from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_worker.rb:53:in `receive_internal_data' ... 15 levels... from /usr/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/packet_master.rb:20:in `run' from /home/raghus/public_html/ feedflix.com/ff/vendor/plugins/backgroundrb/server/lib/master_worker.rb:166:in`initialize' from ./script/backgroundrb:42:in `new' from ./script/backgroundrb:42 ============================ I googled around and saw a few suggestions to escape pipes ("|") and the like but this doesn't have any pipes. And actually the code doesn't seem to have any problems running when called in a loop as part of a batch process. It's only when a FORM submit calls it that it craps. In both the controllers, the call is *exactly* the same: worker = MiddleMan.worker(:netflix_worker) result = worker.parse_netflix_feeds(l_netflix_rss_code) Any ideas what I am missing? Something super-obvious? Thanks! Raghu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080420/0c11a9a6/attachment.html From julien.genestoux at gmail.com Tue Apr 22 19:36:23 2008 From: julien.genestoux at gmail.com (Julien Genestoux) Date: Tue, 22 Apr 2008 16:36:23 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? Message-ID: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> Hello, I am trying to build an application that will parse thousands of XML Feeds in continue in the back. I really have no idea on how to do this "correctly" with Rails. Here is my code : class Feed < ActiveRecord::Base def parse # Parsing using external lib (syndication gem) end end class FeedsController < ApplicationController def parse feed = Feed.find(params[:id]) feed.parse end end So for now, if I want to parse all my feeds forever, what I have to do is to call http://myapp/feeds/1/parse, and then http://myapp/feeds/2/parse ... This is definetely not a good solution! How can I use Backgroundrb to do this? Thanks for your help! -- Julien Genestoux julien.genestoux at gmail.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080422/d86201e4/attachment.html From adam at thewilliams.ws Tue Apr 22 21:43:31 2008 From: adam at thewilliams.ws (Adam Williams) Date: Tue, 22 Apr 2008 21:43:31 -0400 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> Message-ID: <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> On Apr 22, 2008, at 7:36 PM, Julien Genestoux wrote: > So for now, if I want to parse all my feeds forever, what I have to > do is to call http://myapp/feeds/1/parse, and then http://myapp/feeds/2/parse > ... > This is definetely not a good solution! > > How can I use Backgroundrb to do this? 1. Use the version of backgroundrb from subversion. The git one was having problems for me. 2. Follow these instructions: http://backgroundrb.rubyforge.org/ 3. Then read this: http://backgroundrb.rubyforge.org/rails/index.html 4. Create a worker of your own. Schedule it according to http://backgroundrb.rubyforge.org/scheduling/index.html adam (a 3 day old user of backgroundrb) From julien.genestoux at gmail.com Wed Apr 23 01:07:12 2008 From: julien.genestoux at gmail.com (Julien Genestoux) Date: Tue, 22 Apr 2008 22:07:12 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> Message-ID: <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> Thanks Adam for your help... I still have a few questions : shoud I have one worker for each feed that is called periodically (add_periodic_timer) or rather one single worker that calls every feed one by one? What is the best solution, perfomance-wise? Thanks again for your help! Best On 4/22/08, Adam Williams wrote: > On Apr 22, 2008, at 7:36 PM, Julien Genestoux wrote: > > > So for now, if I want to parse all my feeds forever, what I have to > > do is to call http://myapp/feeds/1/parse, and then http://myapp/feeds/2/parse > > ... > > This is definetely not a good solution! > > > > How can I use Backgroundrb to do this? > > > 1. Use the version of backgroundrb from subversion. The git one was > having problems for me. > 2. Follow these instructions: http://backgroundrb.rubyforge.org/ > 3. Then read this: http://backgroundrb.rubyforge.org/rails/index.html > 4. Create a worker of your own. Schedule it according to http://backgroundrb.rubyforge.org/scheduling/index.html > > adam (a 3 day old user of backgroundrb) > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- -- Julien Genestoux julien.genestoux at gmail.com http://www.ouvre-boite.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 From supriya.iiita at gmail.com Wed Apr 23 01:50:18 2008 From: supriya.iiita at gmail.com (supriya agarwal) Date: Wed, 23 Apr 2008 11:20:18 +0530 Subject: [Backgroundrb-devel] Problem with starting backgroundrb server! Message-ID: Hi! Im using this backgroundrb plugin! The steps i followed for using this plugin are as follows: Kindly correct me where i went wrong! 1)rails_apps/app_name/ruby script/plugin install http://svn.devjavu.com/backgroundrb/trunk 2) rake backgroundrb:setup 3) ruby script/generate worker example 4)opened example_worker.rb and just wrote : 1. class ExampleWorker < BackgrounDRb::MetaWorker 2. set_worker_name :example_worker 3. def create(args = nil) 4. # this method is called, when worker is loaded for the first time 5. logger.info "here" 6. end 7. end after that i tried to run #ruby script/backgroundrb start and it gave me load error :bdrb_config.rb not found! I tried copying it to some directory and then ultimately it popped up another error : fork function is umimplemented on this machine! I have also installed the required gems :packets and chronic.. Can someone help me where i went wrong! or give me a step by step procedure to start backgroundrb running. Thanks in advance! Sup -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080423/8675ef7c/attachment.html From dlanger at swissonline.ch Wed Apr 23 03:12:42 2008 From: dlanger at swissonline.ch (Dominik Langer) Date: Wed, 23 Apr 2008 09:12:42 +0200 Subject: [Backgroundrb-devel] Problem with starting backgroundrb server! In-Reply-To: References: Message-ID: <480EE16A.2040705@swissonline.ch> Hi Sup Regarding your first error (i.e., bdrb_config.rb; which you seem to have solved by yourself): I had the same error first, because after checkout I had backgroundrb in the wrong folder. - be sure that BackgrounDrb is installed in /vendor/plugins/backgroundrb/ - this means bdrb_config.rb should be at /vendor/plugins/backgroundrb/lib/bdrb_config.rb Regarding your second error (i.e., unimplemented fork function): - what kind operating system are you running? Fork is not implemented on Windows, but only on Unix/Linux machines. - If you are running Windows, switch to a Linux machine (a virtual machine on the same PC might also work) Cheers, Dominik supriya agarwal schrieb: > Hi! > > Im using this backgroundrb plugin! The steps i followed for using this > plugin are as follows: Kindly correct me where i went wrong! > > 1)rails_apps/app_name/ruby script/plugin install > http://svn.devjavu.com/backgroundrb/trunk > > 2) rake backgroundrb:setup > > 3) ruby script/generate worker example > > 4)opened example_worker.rb and just wrote : > > 1. class ExampleWorker < BackgrounDRb::MetaWorker > 2. set_worker_name :example_worker > 3. def create(args = nil) > 4. # this method is called, when worker is loaded for the first > time > 5. logger.info "here" > 6. end > 7. end > > after that i tried to run #ruby script/backgroundrb start > and it gave me load error :bdrb_config.rb not found! I tried copying > it to some directory and then ultimately it popped up another error : > fork function is umimplemented on this machine! > I have also installed the required gems :packets and chronic.. > > Can someone help me where i went wrong! or give me a step by step > procedure to start backgroundrb running. > Thanks in advance! > Sup > > > ------------------------------------------------------------------------ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From adam at thewilliams.ws Wed Apr 23 06:45:10 2008 From: adam at thewilliams.ws (Adam Williams) Date: Wed, 23 Apr 2008 06:45:10 -0400 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> Message-ID: On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > I still have a few questions : shoud I have one worker for each feed > that is called periodically (add_periodic_timer) or rather one single > worker that calls every feed one by one? > > What is the best solution, perfomance-wise? Good question... I don't suppose I know exactly. I would start by processing all the feeds in one worker invocation - that is what I have done for sending an unknown amount of email. It just seems wrong to me to invoke a worker for one email at a time. The right answer likely lies in understanding the whole MasterWorker, Packet::Reactor/handler_instance.ask_work bits of the puzzle... adam From julien.genestoux at gmail.com Wed Apr 23 10:30:58 2008 From: julien.genestoux at gmail.com (Julien Genestoux) Date: Wed, 23 Apr 2008 07:30:58 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> Message-ID: <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> Thanks Adam, That sounded weird to me as well to have one worker for each feed... However, if I only have one worker, that also means that I am parsing one feed only at any moment. An option, is maybe to have a few workers (denpending on the number of feeds) that parse feeds concurrently? If I only have one worker, according to you what should be the winnning strategy to choose the "right" parse to feed? Obviously some feeds need to be parsed one every few minutes, while some other might no need to be parse more than every hour... Any idea/tip on this? On 4/23/08, Adam Williams wrote: > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > I still have a few questions : shoud I have one worker for each feed > > that is called periodically (add_periodic_timer) or rather one single > > worker that calls every feed one by one? > > > > What is the best solution, perfomance-wise? > > > Good question... I don't suppose I know exactly. I would start by > processing all the feeds in one worker invocation - that is what I > have done for sending an unknown amount of email. It just seems wrong > to me to invoke a worker for one email at a time. > > The right answer likely lies in understanding the whole MasterWorker, > Packet::Reactor/handler_instance.ask_work bits of the puzzle... > > > adam > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- -- Julien Genestoux julien.genestoux at gmail.com http://www.ouvre-boite.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 From stevie at animoto.com Wed Apr 23 12:14:24 2008 From: stevie at animoto.com (Stevie Clifton) Date: Wed, 23 Apr 2008 12:14:24 -0400 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> Message-ID: Hey Julien/Adam, There was a great thread about a similar situation about 10 days ago. Check it out here: http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001681.html Julien, you definitely don't want a worker for each feed, and you'll want to use thread_pool.defer, which will allow you to concurrently process as many feeds as you want (or as many as your system can handle). From what you've said, it sounds like you'll only need one worker coded up, but probably set multiple periodic timers (e.g one of hourly parsing of high-priority feeds, one for nightlies, etc). The method you specify in the periodic timer should use thread_pool.defer to handle processing of multiple feeds at a time -- there's no reason to do them sequentially. stevie On Wed, Apr 23, 2008 at 10:30 AM, Julien Genestoux wrote: > Thanks Adam, > > That sounded weird to me as well to have one worker for each feed... > However, if I only have one worker, that also means that I am parsing > one feed only at any moment. An option, is maybe to have a few workers > (denpending on the number of feeds) that parse feeds concurrently? > > If I only have one worker, according to you what should be the > winnning strategy to choose the "right" parse to feed? Obviously some > feeds need to be parsed one every few minutes, while some other might > no need to be parse more than every hour... > > Any idea/tip on this? > > > > > > > On 4/23/08, Adam Williams wrote: > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > I still have a few questions : shoud I have one worker for each feed > > > that is called periodically (add_periodic_timer) or rather one single > > > worker that calls every feed one by one? > > > > > > What is the best solution, perfomance-wise? > > > > > > Good question... I don't suppose I know exactly. I would start by > > processing all the feeds in one worker invocation - that is what I > > have done for sending an unknown amount of email. It just seems wrong > > to me to invoke a worker for one email at a time. > > > > The right answer likely lies in understanding the whole MasterWorker, > > Packet::Reactor/handler_instance.ask_work bits of the puzzle... > > > > > > adam > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > -- > Julien Genestoux > julien.genestoux at gmail.com > http://www.ouvre-boite.com > +1 (415) 254 7340 > +33 (0)8 70 44 76 29 > _______________________________________________ > > > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From paul.kmiec at appfolio.com Wed Apr 23 12:17:29 2008 From: paul.kmiec at appfolio.com (Paul Kmiec) Date: Wed, 23 Apr 2008 09:17:29 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> Message-ID: <674a03dc0804230917w254f9acdyc46ba8562728c1d3@mail.gmail.com> You can use the built build thread pool to process more than one feed within the same worker. So within the worker, you'd do, def parse_feeds loop do feed = Feed.find_feed_to_process thread_pool.defer do feed.parse end end end I think the default pool size is 20. You can control the size of the thread pool using a class level method, as I recall it is pool_size x Paul On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux < julien.genestoux at gmail.com> wrote: > Thanks Adam, > > That sounded weird to me as well to have one worker for each feed... > However, if I only have one worker, that also means that I am parsing > one feed only at any moment. An option, is maybe to have a few workers > (denpending on the number of feeds) that parse feeds concurrently? > > If I only have one worker, according to you what should be the > winnning strategy to choose the "right" parse to feed? Obviously some > feeds need to be parsed one every few minutes, while some other might > no need to be parse more than every hour... > > Any idea/tip on this? > > > > > On 4/23/08, Adam Williams wrote: > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > I still have a few questions : shoud I have one worker for each feed > > > that is called periodically (add_periodic_timer) or rather one single > > > worker that calls every feed one by one? > > > > > > What is the best solution, perfomance-wise? > > > > > > Good question... I don't suppose I know exactly. I would start by > > processing all the feeds in one worker invocation - that is what I > > have done for sending an unknown amount of email. It just seems wrong > > to me to invoke a worker for one email at a time. > > > > The right answer likely lies in understanding the whole MasterWorker, > > Packet::Reactor/handler_instance.ask_work bits of the puzzle... > > > > > > adam > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -- > -- > Julien Genestoux > julien.genestoux at gmail.com > http://www.ouvre-boite.com > +1 (415) 254 7340 > +33 (0)8 70 44 76 29 > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080423/2437376c/attachment.html From julien.genestoux at gmail.com Wed Apr 23 17:46:52 2008 From: julien.genestoux at gmail.com (Julien Genestoux) Date: Wed, 23 Apr 2008 14:46:52 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <674a03dc0804230917w254f9acdyc46ba8562728c1d3@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> <674a03dc0804230917w254f9acdyc46ba8562728c1d3@mail.gmail.com> Message-ID: <26c0cf900804231446g74f54c4bu207051c10d320f9d@mail.gmail.com> Thanks guys... that's a ton of info! I am definetely gonna use the thread_pool... as soon as I can find the documentation ;D 1- For each feed, I define a "frequency" (every minute, every hour... every 30 minutes...) that will be updated every time I'm parsing the feed: if the parser returns "new" element, I am increasding the frequency (from 1 time per hour, to 1 time per 30 min.), if not, I'm decreasing the frequency... 2- I also have a "last_update" field which remembers the time when the feed was parsed for the last time. 3- With 1 & 2, I know how "late" I am to parse a feed... so when I choose my next feed to parse, I am always choosing the one that is the most "late" I am not sure if Steevie's approach of having multiple tasks for the worker applies here. Actually, I am not even schedulling my worker, I am just launching it once, and the parse_feeds runs forever (while true do... end) Also, if I understand well Paul's code, his approach allows my worker to be more efficient always, but doesn't take into account the "lateness" of my feeds. My idea would be to add/remove worker according to "how late" I am in parsing feeds. If my the the lastest feed is late by more than 10min, I would add one worker... and If my latest feed is late by less than 5 minutes, I would remove one worker Does this approach makes sense to you? Thanks a lot for your help guys... On 4/23/08, Paul Kmiec wrote: > You can use the built build thread pool to process more than one feed within > the same worker. So within the worker, you'd do, > > def parse_feeds > loop do > feed = Feed.find_feed_to_process > thread_pool.defer do > feed.parse > end > end > end > > I think the default pool size is 20. You can control the size of the thread > pool using a class level method, as I recall it is > > pool_size x > > Paul > > > On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux > wrote: > > Thanks Adam, > > > > That sounded weird to me as well to have one worker for each feed... > > However, if I only have one worker, that also means that I am parsing > > one feed only at any moment. An option, is maybe to have a few workers > > (denpending on the number of feeds) that parse feeds concurrently? > > > > If I only have one worker, according to you what should be the > > winnning strategy to choose the "right" parse to feed? Obviously some > > feeds need to be parsed one every few minutes, while some other might > > no need to be parse more than every hour... > > > > Any idea/tip on this? > > > > > > > > > > > > > > > > On 4/23/08, Adam Williams wrote: > > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > > > I still have a few questions : shoud I have one worker for each feed > > > > that is called periodically (add_periodic_timer) or rather one single > > > > worker that calls every feed one by one? > > > > > > > > What is the best solution, perfomance-wise? > > > > > > > > > Good question... I don't suppose I know exactly. I would start by > > > processing all the feeds in one worker invocation - that is what I > > > have done for sending an unknown amount of email. It just seems wrong > > > to me to invoke a worker for one email at a time. > > > > > > The right answer likely lies in understanding the whole MasterWorker, > > > Packet::Reactor/handler_instance.ask_work bits of the > puzzle... > > > > > > > > > adam > > > > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > -- > > -- > > Julien Genestoux > > julien.genestoux at gmail.com > > http://www.ouvre-boite.com > > +1 (415) 254 7340 > > +33 (0)8 70 44 76 29 > > _______________________________________________ > > > > > > > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > -- -- Julien Genestoux julien.genestoux at gmail.com http://www.ouvre-boite.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 From stevie at slowbicycle.com Fri Apr 25 10:54:13 2008 From: stevie at slowbicycle.com (Stevie Clifton) Date: Fri, 25 Apr 2008 10:54:13 -0400 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: <26c0cf900804231446g74f54c4bu207051c10d320f9d@mail.gmail.com> References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> <674a03dc0804230917w254f9acdyc46ba8562728c1d3@mail.gmail.com> <26c0cf900804231446g74f54c4bu207051c10d320f9d@mail.gmail.com> Message-ID: Hey Julien, It sounds like you are planning on using one "long running" feed parsing loop with a do...while. This is exactly the sort of thing you want to avoid in new bdrb, especially if you know you want to do something at discrete time periods--it totally goes against the twisted paradigm. After thinking about it for a bit, I would recommend setting just one periodic_timer for every minute, and then determining in your parse_feeds method which feeds need to be parsed. If I were you, I wouldn't use last_updated to determine when to parse your feeds -- it adds unnecessary complexity to your system. You can of course save that value for reference, but it's not necessary for your requirements. In your db you could have a field for every feed call "interval" that would determine the minute intervals to parse the feeds. Then every minute when parse_feed gets called, you could parse every feed with an interval of "1", and then determine based on the current minute in the hour whether or not to try to parse the 15, 30, or 60 minute feeds. And you'll of course want to use thread_pool.defer. So, using Paul's code as a starting point, something like this: def parse_feeds feeds = Feed.find_feeds_to_process feeds.each do |feed| thread_pool.defer do feed.parse end end end class Feed def find_feeds_to_process feeds = [] [1, 15, 30, 60].each |interval| feeds << Feeds.find_by_interval( interval ) if Time.now.min % interval == 0 end end def parse # parsing code end end On my way home yesterday I thought of another sexy addition you could add to this. In the above code, you know that you'll be parsing _every_ feed in your db on the hour, which isn't a very efficient setup. If possible, you want to set it up so that you have even parsing distribution throughout the hour so you're not getting hammered. You could add a pretty simple heuristic that would give you a relatively even distribution across the hour by using the hash of the feed url. Along with the url and the interval, save an "offset" value like this example: feed = Feed.new feed.url = ''my_feed_url' feed.interval = 15 feed.offset = feed.url.hash % 60 feed.save Then in find_feeds_to_process, you can do this (untested): # the select will return any feed for which its interval offset matches the current minute's offset for the same interval def find_feeds_to_process feeds = Feed.find(:all).select do |feed| [15, 30, 60].detect { |interval| feed.offset % interval == Time.now.min % interval } end end Doing a Feed.find(:all) is probably not the best idea if you have a ton of records, so you might want to do multiple db finds to get the same results. stevie On Wed, Apr 23, 2008 at 5:46 PM, Julien Genestoux wrote: > Thanks guys... that's a ton of info! I am definetely gonna use the > thread_pool... as soon as I can find the documentation ;D > > 1- For each feed, I define a "frequency" (every minute, every hour... > every 30 minutes...) that will be updated every time I'm parsing the > feed: if the parser returns "new" element, I am increasding the > frequency (from 1 time per hour, to 1 time per 30 min.), if not, I'm > decreasing the frequency... > > 2- I also have a "last_update" field which remembers the time when the > feed was parsed for the last time. > > 3- With 1 & 2, I know how "late" I am to parse a feed... so when I > choose my next feed to parse, I am always choosing the one that is the > most "late" > > I am not sure if Steevie's approach of having multiple tasks for the > worker applies here. Actually, I am not even schedulling my worker, I > am just launching it once, and the parse_feeds runs forever (while > true do... end) > > Also, if I understand well Paul's code, his approach allows my worker > to be more efficient always, but doesn't take into account the > "lateness" of my feeds. > > > My idea would be to add/remove worker according to "how late" I am in > parsing feeds. > If my the the lastest feed is late by more than 10min, I would add one > worker... and If my latest feed is late by less than 5 minutes, I > would remove one worker > > Does this approach makes sense to you? > > Thanks a lot for your help guys... > > > > > On 4/23/08, Paul Kmiec wrote: > > You can use the built build thread pool to process more than one feed within > > the same worker. So within the worker, you'd do, > > > > def parse_feeds > > loop do > > feed = Feed.find_feed_to_process > > thread_pool.defer do > > feed.parse > > end > > end > > end > > > > I think the default pool size is 20. You can control the size of the thread > > pool using a class level method, as I recall it is > > > > pool_size x > > > > Paul > > > > > > On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux > > wrote: > > > Thanks Adam, > > > > > > That sounded weird to me as well to have one worker for each feed... > > > However, if I only have one worker, that also means that I am parsing > > > one feed only at any moment. An option, is maybe to have a few workers > > > (denpending on the number of feeds) that parse feeds concurrently? > > > > > > If I only have one worker, according to you what should be the > > > winnning strategy to choose the "right" parse to feed? Obviously some > > > feeds need to be parsed one every few minutes, while some other might > > > no need to be parse more than every hour... > > > > > > Any idea/tip on this? > > > > > > > > > > > > > > > > > > > > > > > > On 4/23/08, Adam Williams wrote: > > > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > > > > > I still have a few questions : shoud I have one worker for each feed > > > > > that is called periodically (add_periodic_timer) or rather one single > > > > > worker that calls every feed one by one? > > > > > > > > > > What is the best solution, perfomance-wise? > > > > > > > > > > > > Good question... I don't suppose I know exactly. I would start by > > > > processing all the feeds in one worker invocation - that is what I > > > > have done for sending an unknown amount of email. It just seems wrong > > > > to me to invoke a worker for one email at a time. > > > > > > > > The right answer likely lies in understanding the whole MasterWorker, > > > > Packet::Reactor/handler_instance.ask_work bits of the > > puzzle... > > > > > > > > > > > > adam > > > > > > > > _______________________________________________ > > > > Backgroundrb-devel mailing list > > > > Backgroundrb-devel at rubyforge.org > > > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > > > > > > -- > > > -- > > > Julien Genestoux > > > julien.genestoux at gmail.com > > > http://www.ouvre-boite.com > > > +1 (415) 254 7340 > > > +33 (0)8 70 44 76 29 > > > _______________________________________________ > > > > > > > > > > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > -- > > > -- > Julien Genestoux > julien.genestoux at gmail.com > http://www.ouvre-boite.com > +1 (415) 254 7340 > +33 (0)8 70 44 76 29 > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From julien.genestoux at gmail.com Mon Apr 28 16:43:17 2008 From: julien.genestoux at gmail.com (Julien Genestoux) Date: Mon, 28 Apr 2008 13:43:17 -0700 Subject: [Backgroundrb-devel] Using Backgroundrb? In-Reply-To: References: <26c0cf900804221636l7757d60bg9621bc3f486a496d@mail.gmail.com> <6C487FC2-F5BB-4B92-AF45-AA277C75B58A@thewilliams.ws> <26c0cf900804222207i1dc12ba9na70b37322a0cf031@mail.gmail.com> <26c0cf900804230730x6b7a9624j346ca1a50440723a@mail.gmail.com> <674a03dc0804230917w254f9acdyc46ba8562728c1d3@mail.gmail.com> <26c0cf900804231446g74f54c4bu207051c10d320f9d@mail.gmail.com> Message-ID: <26c0cf900804281343s7d36e167i667ab6c3edb612c7@mail.gmail.com> Thanks a lot for this very helpful answer. I implement a solution very similar to yours and it runs, but I have 2 big problems. The first one is "throughput". If I have a periodic timer of 1 minute, I can only parse 20 (number of threads) feeds per minute, which leads to 1200 per hour (since I want to parse a feed at least once every hour). The problem is that I really need to be able to parse at least 10 times this number of feeds... and probably closer to 100k! What if I increase the number of threads? Will I be able to parse more feeds? The second one is actually a lot worse. I've had my system running for a little more than a day... without monitoring it, and well, this morning, everything was "down". I did a "ps aux" and here is what I got : USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 21697 0.0 0.8 32524 15620 ? D Apr27 0:13 ruby /mnt/app/current/script/backgroundrb start -e production root 21698 0.0 0.2 32504 4736 ? D Apr27 0:08 ruby log_worker root 21699 1.1 90.5 2170872 1576364 ? D Apr27 25:58 ruby parser_worker As you can see, my parser_worker is consuming a little over 1,5Gb of RAM : wayyyy too much ;) it seems that the vars are not destroyed in my worker? Any idea of what's wrong? Thanks a lot once again for your help! Best, On 4/25/08, Stevie Clifton wrote: > Hey Julien, > > It sounds like you are planning on using one "long running" feed > parsing loop with a do...while. This is exactly the sort of thing you > want to avoid in new bdrb, especially if you know you want to do > something at discrete time periods--it totally goes against the > twisted paradigm. After thinking about it for a bit, I would > recommend setting just one periodic_timer for every minute, and then > determining in your parse_feeds method which feeds need to be parsed. > If I were you, I wouldn't use last_updated to determine when to parse > your feeds -- it adds unnecessary complexity to your system. You can > of course save that value for reference, but it's not necessary for > your requirements. > > In your db you could have a field for every feed call "interval" that > would determine the minute intervals to parse the feeds. Then every > minute when parse_feed gets called, you could parse every feed with an > interval of "1", and then determine based on the current minute in the > hour whether or not to try to parse the 15, 30, or 60 minute feeds. > And you'll of course want to use thread_pool.defer. So, using Paul's > code as a starting point, something like this: > > def parse_feeds > feeds = Feed.find_feeds_to_process > feeds.each do |feed| > > thread_pool.defer do > feed.parse > end > end > end > > > class Feed > def find_feeds_to_process > feeds = [] > [1, 15, 30, 60].each |interval| > feeds << Feeds.find_by_interval( interval ) if Time.now.min % > interval == 0 > end > end > def parse > # parsing code > end > end > > On my way home yesterday I thought of another sexy addition you could > add to this. In the above code, you know that you'll be parsing > _every_ feed in your db on the hour, which isn't a very efficient > setup. If possible, you want to set it up so that you have even > parsing distribution throughout the hour so you're not getting > hammered. You could add a pretty simple heuristic that would give you > a relatively even distribution across the hour by using the hash of > the feed url. Along with the url and the interval, save an "offset" > value like this example: > > feed = Feed.new > feed.url = ''my_feed_url' > feed.interval = 15 > feed.offset = feed.url.hash % 60 > feed.save > > Then in find_feeds_to_process, you can do this (untested): > > # the select will return any feed for which its interval offset > matches the current minute's offset for the same interval > def find_feeds_to_process > feeds = Feed.find(:all).select do |feed| > [15, 30, 60].detect { |interval| feed.offset % interval == > Time.now.min % interval } > end > end > > Doing a Feed.find(:all) is probably not the best idea if you have a > ton of records, so you might want to do multiple db finds to get the > same results. > > stevie > > > On Wed, Apr 23, 2008 at 5:46 PM, Julien Genestoux > > wrote: > > Thanks guys... that's a ton of info! I am definetely gonna use the > > thread_pool... as soon as I can find the documentation ;D > > > > 1- For each feed, I define a "frequency" (every minute, every hour... > > every 30 minutes...) that will be updated every time I'm parsing the > > feed: if the parser returns "new" element, I am increasding the > > frequency (from 1 time per hour, to 1 time per 30 min.), if not, I'm > > decreasing the frequency... > > > > 2- I also have a "last_update" field which remembers the time when the > > feed was parsed for the last time. > > > > 3- With 1 & 2, I know how "late" I am to parse a feed... so when I > > choose my next feed to parse, I am always choosing the one that is the > > most "late" > > > > I am not sure if Steevie's approach of having multiple tasks for the > > worker applies here. Actually, I am not even schedulling my worker, I > > am just launching it once, and the parse_feeds runs forever (while > > true do... end) > > > > Also, if I understand well Paul's code, his approach allows my worker > > to be more efficient always, but doesn't take into account the > > "lateness" of my feeds. > > > > > > My idea would be to add/remove worker according to "how late" I am in > > parsing feeds. > > If my the the lastest feed is late by more than 10min, I would add one > > worker... and If my latest feed is late by less than 5 minutes, I > > would remove one worker > > > > Does this approach makes sense to you? > > > > Thanks a lot for your help guys... > > > > > > > > > > On 4/23/08, Paul Kmiec wrote: > > > You can use the built build thread pool to process more than one feed within > > > the same worker. So within the worker, you'd do, > > > > > > def parse_feeds > > > loop do > > > feed = Feed.find_feed_to_process > > > thread_pool.defer do > > > feed.parse > > > end > > > end > > > end > > > > > > I think the default pool size is 20. You can control the size of the thread > > > pool using a class level method, as I recall it is > > > > > > pool_size x > > > > > > Paul > > > > > > > > > On Wed, Apr 23, 2008 at 7:30 AM, Julien Genestoux > > > wrote: > > > > Thanks Adam, > > > > > > > > That sounded weird to me as well to have one worker for each feed... > > > > However, if I only have one worker, that also means that I am parsing > > > > one feed only at any moment. An option, is maybe to have a few workers > > > > (denpending on the number of feeds) that parse feeds concurrently? > > > > > > > > If I only have one worker, according to you what should be the > > > > winnning strategy to choose the "right" parse to feed? Obviously some > > > > feeds need to be parsed one every few minutes, while some other might > > > > no need to be parse more than every hour... > > > > > > > > Any idea/tip on this? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 4/23/08, Adam Williams wrote: > > > > > On Apr 23, 2008, at 1:07 AM, Julien Genestoux wrote: > > > > > > > > > > > I still have a few questions : shoud I have one worker for each feed > > > > > > that is called periodically (add_periodic_timer) or rather one single > > > > > > worker that calls every feed one by one? > > > > > > > > > > > > What is the best solution, perfomance-wise? > > > > > > > > > > > > > > > Good question... I don't suppose I know exactly. I would start by > > > > > processing all the feeds in one worker invocation - that is what I > > > > > have done for sending an unknown amount of email. It just seems wrong > > > > > to me to invoke a worker for one email at a time. > > > > > > > > > > The right answer likely lies in understanding the whole MasterWorker, > > > > > Packet::Reactor/handler_instance.ask_work bits of the > > > puzzle... > > > > > > > > > > > > > > > adam > > > > > > > > > > _______________________________________________ > > > > > Backgroundrb-devel mailing list > > > > > Backgroundrb-devel at rubyforge.org > > > > > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > > > > > > > > > > > -- > > > > -- > > > > Julien Genestoux > > > > julien.genestoux at gmail.com > > > > http://www.ouvre-boite.com > > > > +1 (415) 254 7340 > > > > +33 (0)8 70 44 76 29 > > > > _______________________________________________ > > > > > > > > > > > > > > > > Backgroundrb-devel mailing list > > > > Backgroundrb-devel at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > > > > > > -- > > > > > > -- > > Julien Genestoux > > julien.genestoux at gmail.com > > http://www.ouvre-boite.com > > +1 (415) 254 7340 > > +33 (0)8 70 44 76 29 > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > -- -- Julien Genestoux julien.genestoux at gmail.com http://www.ouvre-boite.com +1 (415) 254 7340 +33 (0)8 70 44 76 29 From theeuniv at gmail.com Mon Apr 28 17:35:01 2008 From: theeuniv at gmail.com (Brian) Date: Mon, 28 Apr 2008 14:35:01 -0700 Subject: [Backgroundrb-devel] First call to worker method doesn't work Message-ID: <7B7D6FAE-EB1F-4148-8BF0-717D264D63BC@gmail.com> I'm facing this same issue. Tried to follow hemant's recommendation http://rubyforge.org/pipermail/backgroundrb-devel/2008-April/001675.html But get an error when running the Rake task: rake aborted! no such file to load -- spec/rake/spectask /Users/brian/packet-read-only/rakefile:7 (See full trace by running task with --trace) I'm not a rake or a gem guy, so have no clue what to do. The file / spec/rake/spectask doesn't exist. Thanks. From theeuniv at gmail.com Mon Apr 28 18:12:47 2008 From: theeuniv at gmail.com (Brian) Date: Mon, 28 Apr 2008 15:12:47 -0700 Subject: [Backgroundrb-devel] update_attribute overwrites existing attributes with NULL. Message-ID: <5F65FC7F-9B0C-48E8-A0B2-A4883F59D4A0@gmail.com> Hmm. When using update_attribute from within a worker method, it seems that Backgroundrb inserts null values into all attributes that are not explicitly set. Is this possible/true? From brian at morearty.org Mon Apr 28 18:45:59 2008 From: brian at morearty.org (Brian Morearty) Date: Mon, 28 Apr 2008 15:45:59 -0700 Subject: [Backgroundrb-devel] update_attribute overwrites existing attributes with NULL. In-Reply-To: <5F65FC7F-9B0C-48E8-A0B2-A4883F59D4A0@gmail.com> References: <5F65FC7F-9B0C-48E8-A0B2-A4883F59D4A0@gmail.com> Message-ID: <3f8e47bb0804281545o23e9ded3o33d2a4f9344ea94d@mail.gmail.com> Hi Brian, I'm not sure but perhaps this is the answer to your question? http://lists.rubyonrails.org/pipermail/rails/2006-May/039493.html Summary: put a :select on your "find" call so it only updates the attributes that were selected. (Not a backgroundrb issue, but an ActiveRecord one. Seems like an ActiveRecord deficiency to me. The link above is pretty old. Not sure if it's fixed nowadays.) - Brian (not you) On Mon, Apr 28, 2008 at 3:12 PM, Brian wrote: > Hmm. > > When using update_attribute from within a worker method, it seems that > Backgroundrb inserts null values into all attributes that are not explicitly > set. Is this possible/true? > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin at yardbarker.com Tue Apr 29 12:17:05 2008 From: kevin at yardbarker.com (Kevin Olsen) Date: Tue, 29 Apr 2008 09:17:05 -0700 Subject: [Backgroundrb-devel] writing a module for bdrb workers Message-ID: <6FC8CFF2-171E-466D-ADF0-B85867A551AE@yardbarker.com> I'm working on a suite of workers that all inherit a lot of their core functionality from a plugin that i'm writing. the plugin works fine when included within my models so the logic there has been tested and works correctly. the goal here is to be able to be able to extend a suite of bdrb workers that will have their basic transactional logic encapsulated in the plugin/module. I'm thoroughly perplexed and I would love some suggestions on this. example worker: class CommentNotificationWorker < BackgrounDRb::MetaWorker include ActionQueue set_worker_name :comment_notification_worker def create(args = nil) end end I'm trying to call it from inside my comment model in an :after_create hook like so (where self is an instance of class Comment, and enqueue_action is inherited from the ActionQueue module): MiddleMan.worker(:comment_notification_worker).enqueue_action(self) I'm consistently generating this error: vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in `load': undefined class/module Comment (ArgumentError) from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:32:in `receive_data' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `call' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_parser.rb:29:in `extract' from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:31:in `receive_data' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:212:in `read_external_socket' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:204:in `handle_external_messages' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:178:in `handle_read_event' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `each' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `handle_read_event' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:130:in `start_reactor' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `loop' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `start_reactor' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:21:in `run' from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 followed by a heap of: /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_nbio.rb:25:in `read_data': Packet::DisconnectError (Packet::DisconnectError) from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_worker.rb:46:in `handle_internal_messages' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:176:in `handle_read_event' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `each' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:174:in `handle_read_event' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:130:in `start_reactor' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `loop' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_core.rb:124:in `start_reactor' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_worker.rb:20:in `start_worker' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:133:in `fork_and_load' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:96:in `load_workers' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:91:in `each' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:91:in `load_workers' from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ packet_master.rb:20:in `run' from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ backgroundrb/server/lib/master_worker.rb:166:in `initialize' from script/backgroundrb:60:in `new' from script/backgroundrb:60 any recommendations would be appreciated. From kevin at yardbarker.com Tue Apr 29 14:19:55 2008 From: kevin at yardbarker.com (Kevin Olsen) Date: Tue, 29 Apr 2008 11:19:55 -0700 Subject: [Backgroundrb-devel] writing a module for bdrb workers References: <015C172C-8807-4706-A618-F58CCB8B0D8A@yardbarker.com> Message-ID: <7474D715-5254-4536-AB3C-BE53775DA228@yardbarker.com> An addendum to my previous post: this process works from the rails console: I've removed the specifics of the model and its contents, but they are correct when they come back >>MiddleMan.worker (:comment_notification_worker).enqueue_action( SomeModel.find(:first) ) {:type=>:do_work, :data=># , :worker_method =>:enqueue_action, :worker=>:comment_notification_worker} => nil backgroundrb receives the object and handles it properly, writing to the database. if anyone has any thoughts on this disparity in behavior I'd really appreciate some feedback kevin On Apr 29, 2008, at 9:17 AM, Kevin Olsen wrote: > I'm working on a suite of workers that all inherit a lot of their > core functionality from a plugin that i'm writing. the plugin works > fine when included within my models so the logic there has been > tested and works correctly. the goal here is to be able to be able > to extend a suite of bdrb workers that will have their basic > transactional logic encapsulated in the plugin/module. > > I'm thoroughly perplexed and I would love some suggestions on this. > > > example worker: > > class CommentNotificationWorker < BackgrounDRb::MetaWorker > include ActionQueue > > set_worker_name :comment_notification_worker > > def create(args = nil) > end > > end > > I'm trying to call it from inside my comment model in > an :after_create hook like so (where self is an instance of class > Comment, and enqueue_action is inherited from the ActionQueue module): > > > MiddleMan.worker(:comment_notification_worker).enqueue_action(self) > > > > I'm consistently generating this error: > > vendor/plugins/backgroundrb/server/lib/master_worker.rb:32:in > `load': undefined class/module Comment (ArgumentError) > from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ > backgroundrb/server/lib/master_worker.rb:32:in `receive_data' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_parser.rb:29:in `call' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_parser.rb:29:in `extract' > from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ > backgroundrb/server/lib/master_worker.rb:31:in `receive_data' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:212:in `read_external_socket' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:204:in `handle_external_messages' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:178:in `handle_read_event' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:174:in `each' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:174:in `handle_read_event' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:130:in `start_reactor' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:124:in `loop' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:124:in `start_reactor' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:21:in `run' > from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ > backgroundrb/server/lib/master_worker.rb:166:in `initialize' > from script/backgroundrb:60:in `new' > from script/backgroundrb:60 > > followed by a heap of: > > /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_nbio.rb:25:in `read_data': Packet::DisconnectError > (Packet::DisconnectError) > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_worker.rb:46:in `handle_internal_messages' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:176:in `handle_read_event' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:174:in `each' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:174:in `handle_read_event' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:130:in `start_reactor' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:124:in `loop' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_core.rb:124:in `start_reactor' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_worker.rb:20:in `start_worker' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:133:in `fork_and_load' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:96:in `load_workers' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:91:in `each' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:91:in `load_workers' > from /opt/local/lib/ruby/gems/1.8/gems/packet-0.1.5/lib/packet/ > packet_master.rb:20:in `run' > from /Users/kevin/_yardbarker/action_queue_2008/vendor/plugins/ > backgroundrb/server/lib/master_worker.rb:166:in `initialize' > from script/backgroundrb:60:in `new' > from script/backgroundrb:60 > > > > any recommendations would be appreciated. > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From dave at 3mix.com Tue Apr 29 14:59:56 2008 From: dave at 3mix.com (David Masover) Date: Tue, 29 Apr 2008 13:59:56 -0500 Subject: [Backgroundrb-devel] SSL? Unix sockets? Message-ID: <1abf6d300804291159sad2962yde399d183e235a59@mail.gmail.com> It seems to me that BackgrounDRb would be a good fit for a central job server shared across applications, but I want to use it with one app -- it would be easier and safer to simply use a DRbUNIXSocket on a single machine. On multiple machines, the only way I know of to add any kind of security (even authentication) is to use SSL. Does BackgrounDRb support either of these options? If not, where's a good place to start on adding that functionality? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasonlee9 at gmail.com Wed Apr 30 01:16:00 2008 From: jasonlee9 at gmail.com (Jason Lee) Date: Tue, 29 Apr 2008 22:16:00 -0700 Subject: [Backgroundrb-devel] strange deployment issue Message-ID: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> Hello, I'm using just the scheduler with my workers like cron jobs. I have 3 different ones that poll my db for specific object states and do work accordingly. This all works perfectly on my development machine. However, when I deploy to my staging server - with its own staging configuration - the scheduler works, the worker processes, but my ActiveRecord object never returns any data back. I explicitly include the model in the top of the worker and when the worker processes, I can see the query logging in the log file. I can even cut the query, paste into a sql window and get back the expected results. But whenever the query returns in my non-development deployed worker, the result set is empty. My backgroundrb.yml configuration is super simple: --- :backgroundrb: :ip: 0.0.0.0 :port: 11006 Again, the above is only just for scheduling and works perfectly on my local env. I also explicitly executed brb with the env specific setting: script/console backgroundrb -e stage Still no luck. Has anyone see this weird issue with AR not returning back any results, even tho the query has been executed in the db? tia, - jason From swertyui at gmail.com Wed Apr 30 04:05:57 2008 From: swertyui at gmail.com (steef nobody) Date: Wed, 30 Apr 2008 04:05:57 -0400 Subject: [Backgroundrb-devel] [Newbie] Questions on having one worker being called by multiple users. Message-ID: <3262a2db0804300105x4feffb0dta6bd0c3c6d82cfdd@mail.gmail.com> Hello, As a certified newbie, I have a few questions on how to properly handle a situation where multiple users may want to perform the same worker task at the same time, but with different arguments. Initially I opted towards making many very specific workers to have a situation where it was highly unlikely that any two users would be asking the same exact worker to do something. Now I'm trying to consolidate everything into as few workers as possible to conserve memory (I think?). What it's come down to at this point is that I have 2 workers whose methods are scheduled in config/backgroundrb.yml. And one worker whose methods are not scheduled and are intended to be called through user initiated actions. My question is then - how should I go about handling the case where user1 has asked the user_worker to do_something at the same time user2 has asked user_worker to do_something or do_something_else. What happens when backgroundrb receives a worker request on a worker that is half way through a method from a previous request? Is there a way to ask a worker something along the lines of "are you busy with another user's request right now?". I'm trying not to spawn unique workers as I'm not sure how best to get rid of them. Would it make sense to create a new worker for the specific task, run the task, and kill the worker if the user may want to run the task fairly frequently? - Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam at thewilliams.ws Wed Apr 30 07:00:56 2008 From: adam at thewilliams.ws (Adam Williams) Date: Wed, 30 Apr 2008 07:00:56 -0400 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> Message-ID: <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> I'm not certain at all about this, but the -e thing was not working well for me. Try spitting out the RAILS_ENV value in your worker, to be sure it is the environment your expect. Or, perhaps you could use 'sudo lsof -Pni -a -c ruby' to see what database ruby processes are connecting to. On Apr 30, 2008, at 1:16 AM, Jason Lee wrote: > Hello, > > I'm using just the scheduler with my workers like cron jobs. I have 3 > different ones that poll my db for specific object states and do work > accordingly. > > This all works perfectly on my development machine. However, when I > deploy to my staging server - with its own staging configuration - the > scheduler works, the worker processes, but my ActiveRecord object > never returns any data back. > > I explicitly include the model in the top of the worker and when the > worker processes, I can see the query logging in the log file. I can > even cut the query, paste into a sql window and get back the expected > results. But whenever the query returns in my non-development deployed > worker, the result set is empty. > > My backgroundrb.yml configuration is super simple: > > --- > :backgroundrb: > :ip: 0.0.0.0 > :port: 11006 > > Again, the above is only just for scheduling and works perfectly on my > local env. I also explicitly executed brb with the env specific > setting: > > script/console backgroundrb -e stage > > Still no luck. > > Has anyone see this weird issue with AR not returning back any > results, even tho the query has been executed in the db? > > tia, > > - jason > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From jasonlee9 at gmail.com Wed Apr 30 10:32:51 2008 From: jasonlee9 at gmail.com (Jason Lee) Date: Wed, 30 Apr 2008 07:32:51 -0700 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> Message-ID: <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> Ah, that's it, it's still in development env. So neither of these commands worked for me: RAILS_ENV=stage rake backgroundrb:start script/backgroundrb start -e stage both go to development. Right now I'm kicking them off manually at the command line. Maybe the rake command is not taking the environment in to account? Now that I know the cause, I guess I have to try to fix it. I saw the environment issue mentioned in previous postings on the list, but it didn't look like there was a solution yet (I think). I'm running rev 234 of BRb. If I figure this out or someone else does, please post. Thx. - jason On Wed, Apr 30, 2008 at 4:00 AM, Adam Williams wrote: > I'm not certain at all about this, but the -e thing was not working well for > me. Try spitting out the RAILS_ENV value in your worker, to be sure it is > the environment your expect. Or, perhaps you could use 'sudo lsof -Pni -a -c > ruby' to see what database ruby processes are connecting to. > > > > On Apr 30, 2008, at 1:16 AM, Jason Lee wrote: > > > > > > > > > > Hello, > > > > I'm using just the scheduler with my workers like cron jobs. I have 3 > > different ones that poll my db for specific object states and do work > > accordingly. > > > > This all works perfectly on my development machine. However, when I > > deploy to my staging server - with its own staging configuration - the > > scheduler works, the worker processes, but my ActiveRecord object > > never returns any data back. > > > > I explicitly include the model in the top of the worker and when the > > worker processes, I can see the query logging in the log file. I can > > even cut the query, paste into a sql window and get back the expected > > results. But whenever the query returns in my non-development deployed > > worker, the result set is empty. > > > > My backgroundrb.yml configuration is super simple: > > > > --- > > :backgroundrb: > > :ip: 0.0.0.0 > > :port: 11006 > > > > Again, the above is only just for scheduling and works perfectly on my > > local env. I also explicitly executed brb with the env specific > > setting: > > > > script/console backgroundrb -e stage > > > > Still no luck. > > > > Has anyone see this weird issue with AR not returning back any > > results, even tho the query has been executed in the db? > > > > tia, > > > > - jason > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > From jasonlee9 at gmail.com Wed Apr 30 11:28:29 2008 From: jasonlee9 at gmail.com (Jason Lee) Date: Wed, 30 Apr 2008 08:28:29 -0700 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> Message-ID: <972417190804300828g6ea9b150he5bc183ca82e3601@mail.gmail.com> Ok, I'll try that. Yes, I have an environment separately configured for 'stage' and I explicitly set it where needed. Thanks, - jason On Wed, Apr 30, 2008 at 8:15 AM, Stevie Clifton wrote: > Hey Jason, > > To get the environment passed in correctly, I use: > RAILS_ENV=staging ./script/backgroundrb start > > And just to make sure, you're aren't actually calling your staging > environment "stage" are you? Or if you are, you have an environment > intentionally named that, right? Just had to ask :). > > stevie > > > > On Wed, Apr 30, 2008 at 10:32 AM, Jason Lee wrote: > > Ah, that's it, it's still in development env. So neither of these > > commands worked for me: > > > > RAILS_ENV=stage rake backgroundrb:start > > > > script/backgroundrb start -e stage > > > > both go to development. Right now I'm kicking them off manually at the > > command line. Maybe the rake command is not taking the environment in > > to account? > > > > Now that I know the cause, I guess I have to try to fix it. I saw the > > environment issue mentioned in previous postings on the list, but it > > didn't look like there was a solution yet (I think). I'm running rev > > 234 of BRb. > > > > If I figure this out or someone else does, please post. Thx. > > > > - jason > > > > > > > > On Wed, Apr 30, 2008 at 4:00 AM, Adam Williams wrote: > > > I'm not certain at all about this, but the -e thing was not working well for > > > me. Try spitting out the RAILS_ENV value in your worker, to be sure it is > > > the environment your expect. Or, perhaps you could use 'sudo lsof -Pni -a -c > > > ruby' to see what database ruby processes are connecting to. > > > > > > > > > > > > On Apr 30, 2008, at 1:16 AM, Jason Lee wrote: > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > I'm using just the scheduler with my workers like cron jobs. I have 3 > > > > different ones that poll my db for specific object states and do work > > > > accordingly. > > > > > > > > This all works perfectly on my development machine. However, when I > > > > deploy to my staging server - with its own staging configuration - the > > > > scheduler works, the worker processes, but my ActiveRecord object > > > > never returns any data back. > > > > > > > > I explicitly include the model in the top of the worker and when the > > > > worker processes, I can see the query logging in the log file. I can > > > > even cut the query, paste into a sql window and get back the expected > > > > results. But whenever the query returns in my non-development deployed > > > > worker, the result set is empty. > > > > > > > > My backgroundrb.yml configuration is super simple: > > > > > > > > --- > > > > :backgroundrb: > > > > :ip: 0.0.0.0 > > > > :port: 11006 > > > > > > > > Again, the above is only just for scheduling and works perfectly on my > > > > local env. I also explicitly executed brb with the env specific > > > > setting: > > > > > > > > script/console backgroundrb -e stage > > > > > > > > Still no luck. > > > > > > > > Has anyone see this weird issue with AR not returning back any > > > > results, even tho the query has been executed in the db? > > > > > > > > tia, > > > > > > > > - jason > > > > _______________________________________________ > > > > Backgroundrb-devel mailing list > > > > Backgroundrb-devel at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > From stevie at slowbicycle.com Wed Apr 30 11:15:24 2008 From: stevie at slowbicycle.com (Stevie Clifton) Date: Wed, 30 Apr 2008 11:15:24 -0400 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> Message-ID: Hey Jason, To get the environment passed in correctly, I use: RAILS_ENV=staging ./script/backgroundrb start And just to make sure, you're aren't actually calling your staging environment "stage" are you? Or if you are, you have an environment intentionally named that, right? Just had to ask :). stevie On Wed, Apr 30, 2008 at 10:32 AM, Jason Lee wrote: > Ah, that's it, it's still in development env. So neither of these > commands worked for me: > > RAILS_ENV=stage rake backgroundrb:start > > script/backgroundrb start -e stage > > both go to development. Right now I'm kicking them off manually at the > command line. Maybe the rake command is not taking the environment in > to account? > > Now that I know the cause, I guess I have to try to fix it. I saw the > environment issue mentioned in previous postings on the list, but it > didn't look like there was a solution yet (I think). I'm running rev > 234 of BRb. > > If I figure this out or someone else does, please post. Thx. > > - jason > > > > On Wed, Apr 30, 2008 at 4:00 AM, Adam Williams wrote: > > I'm not certain at all about this, but the -e thing was not working well for > > me. Try spitting out the RAILS_ENV value in your worker, to be sure it is > > the environment your expect. Or, perhaps you could use 'sudo lsof -Pni -a -c > > ruby' to see what database ruby processes are connecting to. > > > > > > > > On Apr 30, 2008, at 1:16 AM, Jason Lee wrote: > > > > > > > > > > > > > > > > Hello, > > > > > > I'm using just the scheduler with my workers like cron jobs. I have 3 > > > different ones that poll my db for specific object states and do work > > > accordingly. > > > > > > This all works perfectly on my development machine. However, when I > > > deploy to my staging server - with its own staging configuration - the > > > scheduler works, the worker processes, but my ActiveRecord object > > > never returns any data back. > > > > > > I explicitly include the model in the top of the worker and when the > > > worker processes, I can see the query logging in the log file. I can > > > even cut the query, paste into a sql window and get back the expected > > > results. But whenever the query returns in my non-development deployed > > > worker, the result set is empty. > > > > > > My backgroundrb.yml configuration is super simple: > > > > > > --- > > > :backgroundrb: > > > :ip: 0.0.0.0 > > > :port: 11006 > > > > > > Again, the above is only just for scheduling and works perfectly on my > > > local env. I also explicitly executed brb with the env specific > > > setting: > > > > > > script/console backgroundrb -e stage > > > > > > Still no luck. > > > > > > Has anyone see this weird issue with AR not returning back any > > > results, even tho the query has been executed in the db? > > > > > > tia, > > > > > > - jason > > > _______________________________________________ > > > Backgroundrb-devel mailing list > > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > From jasonlee9 at gmail.com Wed Apr 30 12:49:59 2008 From: jasonlee9 at gmail.com (Jason Lee) Date: Wed, 30 Apr 2008 09:49:59 -0700 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> Message-ID: <972417190804300949s527c5fadlb7dc3d043de23bcf@mail.gmail.com> So I tried this and my debug still says the env is development.. Guess I'll have to start hacking to see why this is.. - jason On Wed, Apr 30, 2008 at 8:15 AM, Stevie Clifton wrote: > Hey Jason, > > To get the environment passed in correctly, I use: > RAILS_ENV=staging ./script/backgroundrb start > > And just to make sure, you're aren't actually calling your staging > environment "stage" are you? Or if you are, you have an environment > intentionally named that, right? Just had to ask :). > > stevie From adam at thewilliams.ws Wed Apr 30 13:35:42 2008 From: adam at thewilliams.ws (Adam Williams) Date: Wed, 30 Apr 2008 13:35:42 -0400 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: <972417190804300949s527c5fadlb7dc3d043de23bcf@mail.gmail.com> References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> <972417190804300949s527c5fadlb7dc3d043de23bcf@mail.gmail.com> Message-ID: So, there is some serious confusion about this in the code. I now know of two other folks (one you, Jason) who struggled with this besides me. I solved it by having a shared/config/backgroundrb.yml on the server, which we link into config of the app on deploy (capistrano). It looks like this: :backgroundrb: :port: 11006 :ip: 0.0.0.0 :environment: staging :lazy_load: true :debug_log: true :schedules: :notification: :deliver_event_reminders: :trigger_args: 0 */3 * * * * * As you can see, we have the environment in the config. The code in meta_worker obviously disrespects -e. Now, you ALSO need to pass -e staging, so that when Rails boots, it has the correct environment. Here is our init.d line: /var/www/apps/railsapplication/current/script/backgroundrb "$1" -e staging >> /var/log/backgroundrb I may patch this at some point, but it's already eaten a bit too much time (not you, but solving it originally). aiwilliams On Apr 30, 2008, at 12:49 PM, Jason Lee wrote: > So I tried this and my debug still says the env is development.. > > Guess I'll have to start hacking to see why this is.. > > - jason > > On Wed, Apr 30, 2008 at 8:15 AM, Stevie Clifton > wrote: >> Hey Jason, >> >> To get the environment passed in correctly, I use: >> RAILS_ENV=staging ./script/backgroundrb start >> >> And just to make sure, you're aren't actually calling your staging >> environment "stage" are you? Or if you are, you have an environment >> intentionally named that, right? Just had to ask :). >> >> stevie > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From jasonlee9 at gmail.com Wed Apr 30 17:22:15 2008 From: jasonlee9 at gmail.com (Jason Lee) Date: Wed, 30 Apr 2008 14:22:15 -0700 Subject: [Backgroundrb-devel] strange deployment issue In-Reply-To: References: <972417190804292216l5c86cb40h89560f4d23e985c1@mail.gmail.com> <109E7408-3F5C-4BA8-8812-607C0A7C62D0@thewilliams.ws> <972417190804300732s10f0e16araa1977ebe22a9288@mail.gmail.com> <972417190804300949s527c5fadlb7dc3d043de23bcf@mail.gmail.com> Message-ID: <972417190804301422s4a84b3a8m650ba880377a07e8@mail.gmail.com> Adam, Thanks for the reply. I'll have to see if I can play around with the yml file. I definitely cannot use the scheduler definition as it was very unreliable for the tasks I needed to launch. If I can just get the environments to come up without any of the other config, I'll be good. I'll post back my solution when I find it. Thanks again, - jason On Wed, Apr 30, 2008 at 10:35 AM, Adam Williams wrote: > So, there is some serious confusion about this in the code. I now know of > two other folks (one you, Jason) who struggled with this besides me. > > I solved it by having a shared/config/backgroundrb.yml on the server, which > we link into config of the app on deploy (capistrano). It looks like this: > > :backgroundrb: > :port: 11006 > > :ip: 0.0.0.0 > :environment: staging > :lazy_load: true > :debug_log: true > > :schedules: > :notification: > :deliver_event_reminders: > :trigger_args: 0 */3 * * * * * > > As you can see, we have the environment in the config. The code in > meta_worker obviously disrespects -e. Now, you ALSO need to pass -e staging, > so that when Rails boots, it has the correct environment. Here is our init.d > line: > > /var/www/apps/railsapplication/current/script/backgroundrb "$1" -e staging > >> /var/log/backgroundrb > > I may patch this at some point, but it's already eaten a bit too much time > (not you, but solving it originally). > > aiwilliams > > > > On Apr 30, 2008, at 12:49 PM, Jason Lee wrote: > > > > > > > > > > So I tried this and my debug still says the env is development.. > > > > Guess I'll have to start hacking to see why this is.. > > > > - jason > > > > On Wed, Apr 30, 2008 at 8:15 AM, Stevie Clifton > wrote: > > > > > Hey Jason, > > > > > > To get the environment passed in correctly, I use: > > > RAILS_ENV=staging ./script/backgroundrb start > > > > > > And just to make sure, you're aren't actually calling your staging > > > environment "stage" are you? Or if you are, you have an environment > > > intentionally named that, right? Just had to ask :). > > > > > > stevie > > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > >