From n.madhan at yahoo.co.in Fri Jan 2 03:57:37 2009 From: n.madhan at yahoo.co.in (madhankumar nagaraj) Date: Fri, 2 Jan 2009 14:27:37 +0530 (IST) Subject: [Backgroundrb-devel] Backgroundrb stops suddenly Message-ID: <92606.97683.qm@web95107.mail.in2.yahoo.com> hi, i have used backgroundrb to update the database with the files having some useful data, where i am getting files periodically uploaded. it works well, but suddenly it stops the worker and stopped the updation. when i checked the backgroundrb.log file it shows Mysql::Error: MySQL server has gone away: (Query used for updation)? LIMIT 1 - (ActiveRecord::StatementInvalid) but, the query is working fine in mysql. i have created one worker to do the updation with job_key. when i checked the job_key value for the particular worker is shows nil. since the worker was stopped. what would be the problem, how can i make the updation works continuously. thanks Madhan. Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Fri Jan 2 13:31:11 2009 From: gethemant at gmail.com (hemant) Date: Sat, 3 Jan 2009 00:01:11 +0530 Subject: [Backgroundrb-devel] Backgroundrb stops suddenly In-Reply-To: <92606.97683.qm@web95107.mail.in2.yahoo.com> References: <92606.97683.qm@web95107.mail.in2.yahoo.com> Message-ID: Which version of mysql gem you are using? If you use mysql C gem, you should be fine. Also, try pasting worker code, so as we can help. On Fri, Jan 2, 2009 at 2:27 PM, madhankumar nagaraj wrote: > hi, > i have used backgroundrb to update the database with the files having some > useful data, where i am getting files periodically uploaded. > it works well, but suddenly it stops the worker and stopped the updation. > when i checked the backgroundrb.log file > it shows > Mysql::Error: MySQL server has gone away: (Query used for updation) LIMIT 1 > - (ActiveRecord::StatementInvalid) > but, the query is working fine in mysql. > > i have created one worker to do the updation with job_key. > when i checked the job_key value for the particular worker is shows nil. > since the worker was stopped. > > what would be the problem, how can i make the updation works continuously. > > > > thanks > Madhan. > > ________________________________ > Add more friends to your messenger and enjoy! Invite them now. > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From raghu.srinivasan at gmail.com Sun Jan 4 03:20:59 2009 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Sun, 4 Jan 2009 00:20:59 -0800 Subject: [Backgroundrb-devel] Need help with understanding how thread_pool/pool_size work In-Reply-To: <6482c06a0812311500n142993eagf5dbeecd57a9fc38@mail.gmail.com> References: <6482c06a0812310915l33ab2f48veacf67a08446ebe6@mail.gmail.com> <6482c06a0812311443t6c4eccfbs9c1e20f2f03ac4c5@mail.gmail.com> <6482c06a0812311500n142993eagf5dbeecd57a9fc38@mail.gmail.com> Message-ID: <6482c06a0901040020t227da380va92437bd2e4c3d92@mail.gmail.com> Does anyone else have any ideas for how to fix this? If someone has sample code that they don't mind sharing, I'd really appreciate taking a look at some real code that does the pool_size/thread_pool magic. Raghu On Wed, Dec 31, 2008 at 3:00 PM, Raghu Srinivasan < raghu.srinivasan at gmail.com> wrote: > No luck Hemant. I changed my worker as follows, no error, I get the exact > same o/p as before. Where might I be messing up?Thanks! > Raghu > ============================ > class FooWorker < BackgrounDRb::MetaWorker > > set_worker_name :foo_worker > pool_size 10 > > def create > # this method is called, when worker is loaded for the > first time > end > > def run_concurrent(args) > logger.info "*** FOO_WORKER/RUN_CONCURRENT at " + > Time.now.to_s > thread_pool.defer(:my_actual_method,:x => args[:job_key]) > end > > def my_actual_method(args) > logger.info "*** FOO_WORKER/MY_ACTUAL_METHOD " + args[:x] > + " at " + Time.now.to_s > sleep 5 > end > end > ============================ > > On Wed, Dec 31, 2008 at 2:43 PM, Raghu Srinivasan < > raghu.srinivasan at gmail.com> wrote: > >> No, I don't get any errors at all. Why is it mandatory for the method that >> I want to run concurrently to receive an argument? In the real case of >> course I'll be passing some equivalent of an ID but let me try with that and >> see... >> >> >> On Wed, Dec 31, 2008 at 1:48 PM, hemant wrote: >> >>> Could it be, because method that runs in thread expects an argument >>> and you are not passing that argument when using thread_pool.defer() >>> ?? >>> >>> Did you not get any errors in log file? >>> >>> >>> >>> On Wed, Dec 31, 2008 at 10:45 PM, Raghu Srinivasan >>> wrote: >>> > I'm on BDRB v 1.0.3 and am trying to understand how thread_pool and >>> > pool_size work. I have to kick of dozens of jobs on schedule (each hour >>> or >>> > so). Other than the fact that they're all accessing the same DB there's >>> no >>> > reason for them to not run in parallel. thread_pool/pool_size should be >>> the >>> > way to go right? >>> > I've put in sample code with its results below: >>> > My Rails controller kicks off 5 jobs in a loop - each calling the >>> > run_concurrent method in my foo worker. run_concurrent then calls >>> > my_actual_method which just logger.infos a message with a time stamp >>> and >>> > sleeps for 5 seconds to simulate a long running job. I did this as >>> > per http://backgroundrb.rubyforge.org/workers/#thread_pool Since I'm >>> calling >>> > this via a defer and have a pool size of 10, I expect to see that >>> > my_actual_method actually gets called 5 times in quick succession >>> (since the >>> > pool size is greater than the # of calls). However I find that >>> > run_concurrent doesn't even call my_actual_method. Here's the output >>> from my >>> > backgroundrb log when I go to http://mysite.com/some/foobar >>> > Can someone help me understand what I'm doing wrong here? >>> > Thanks, >>> > Raghu >>> > ==================================================== >>> > Rails controller code >>> > class SomeController < ApplicationController >>> > def foobar >>> > i = 0 >>> > while i < 5 >>> > worker = MiddleMan.worker(:foo_worker) >>> > result = worker.run_concurrent(:job_key => >>> > random_string(10)) >>> > i += 1 >>> > end >>> > render :text => 'all done at ' + Time.now.to_s >>> > end >>> > end >>> > ==================================================== >>> > Worker code >>> > class FooWorker < BackgrounDRb::MetaWorker >>> > set_worker_name :foo_worker >>> > pool_size 10 >>> > def create >>> > # this method is called, when worker is loaded for the >>> first >>> > time >>> > end >>> > def run_concurrent(args) >>> > logger.info "*** FOO_WORKER/RUN_CONCURRENT at " + >>> > Time.now.to_s >>> > thread_pool.defer(:my_actual_method) >>> > end >>> > def my_actual_method(args) >>> > logger.info "*** FOO_WORKER/MY_ACTUAL_METHOD at " + >>> > Time.now.to_s >>> > sleep 5 >>> > end >>> > end >>> > ==================================================== >>> > Here's the output >>> > # Logfile created on Wed Dec 31 09:15:21 +0000 2008 by / >>> > foo_worker started (pid:5087) >>> > Schedules for worker loaded (pid:5087) >>> > run_concurrent job_keydfvq5o4m0s (pid:5087) >>> > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 09:15:26 +0000 2008 >>> (pid:5087) >>> > run_concurrent job_keyy4tascam6k (pid:5087) >>> > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 09:15:26 +0000 2008 >>> (pid:5087) >>> > run_concurrent job_key8gw2eegeqs (pid:5087) >>> > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 09:15:26 +0000 2008 >>> (pid:5087) >>> > run_concurrent job_keyywb1oop73t (pid:5087) >>> > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 09:15:26 +0000 2008 >>> (pid:5087) >>> > run_concurrent job_key17gfkqtzjh (pid:5087) >>> > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 09:15:26 +0000 2008 >>> (pid:5087) >>> > ==================================================== >>> > >>> > _______________________________________________ >>> > Backgroundrb-devel mailing list >>> > Backgroundrb-devel at rubyforge.org >>> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> > >>> >>> >>> >>> -- >>> Let them talk of their oriental summer climes of everlasting >>> conservatories; give me the privilege of making my own summer with my >>> own coals. >>> >>> http://gnufied.org >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Sun Jan 4 13:22:43 2009 From: gethemant at gmail.com (hemant kumar) Date: Sun, 04 Jan 2009 23:52:43 +0530 Subject: [Backgroundrb-devel] Need help with understanding how thread_pool/pool_size work In-Reply-To: <6482c06a0901040020t227da380va92437bd2e4c3d92@mail.gmail.com> References: <6482c06a0812310915l33ab2f48veacf67a08446ebe6@mail.gmail.com> <6482c06a0812311443t6c4eccfbs9c1e20f2f03ac4c5@mail.gmail.com> <6482c06a0812311500n142993eagf5dbeecd57a9fc38@mail.gmail.com> <6482c06a0901040020t227da380va92437bd2e4c3d92@mail.gmail.com> Message-ID: <1231093363.27670.5.camel@shire> Hi Raghu, I tried following worker: class FooWorker < BackgrounDRb::MetaWorker set_worker_name :foo_worker def create(args = nil) # this method is called, when worker is loaded for the first time logger.info "worker has started" end def accept_cmd args thread_pool.defer(:hello,args) end def hello args logger.info "running hello with #{args[:age]}" end end And code to invoke thread pool task was: MiddleMan.worker(:foo_worker).async_accept_cmd(:arg => {:age =>19},:job_key => "foo") Also, note that with tasks that you are going to add to thread pool, there is not point in invoking them through blocking call (or synchrounous like you did in your example), because it doesn't make sense to block for results when task is added to thread pool and ran at its own leisure. What you want is to collect the results in "cache" object and poll from rails. On Sun, 2009-01-04 at 00:20 -0800, Raghu Srinivasan wrote: > Does anyone else have any ideas for how to fix this? If someone has > sample code that they don't mind sharing, I'd really appreciate taking > a look at some real code that does the pool_size/thread_pool magic. > > > Raghu > > On Wed, Dec 31, 2008 at 3:00 PM, Raghu Srinivasan > wrote: > No luck Hemant. I changed my worker as follows, no error, I > get the exact same o/p as before. Where might I be messing up? > Thanks! > Raghu > ============================ > class FooWorker < BackgrounDRb::MetaWorker > > > set_worker_name :foo_worker > pool_size 10 > > > def create > # this method is called, when worker is loaded > for the first time > end > > > def run_concurrent(args) > logger.info "*** FOO_WORKER/RUN_CONCURRENT at > " + Time.now.to_s > thread_pool.defer(:my_actual_method,:x => > args[:job_key]) > end > > > def my_actual_method(args) > logger.info "*** FOO_WORKER/MY_ACTUAL_METHOD " > + args[:x] + " at " + Time.now.to_s > sleep 5 > end > end > ============================ > > > On Wed, Dec 31, 2008 at 2:43 PM, Raghu Srinivasan > wrote: > No, I don't get any errors at all. Why is it mandatory > for the method that I want to run concurrently to > receive an argument? In the real case of course I'll > be passing some equivalent of an ID but let me try > with that and see... > > > > On Wed, Dec 31, 2008 at 1:48 PM, hemant > wrote: > Could it be, because method that runs in > thread expects an argument > and you are not passing that argument when > using thread_pool.defer() > ?? > > Did you not get any errors in log file? > > > > > On Wed, Dec 31, 2008 at 10:45 PM, Raghu > Srinivasan > wrote: > > I'm on BDRB v 1.0.3 and am trying to > understand how thread_pool and > > pool_size work. I have to kick of dozens of > jobs on schedule (each hour or > > so). Other than the fact that they're all > accessing the same DB there's no > > reason for them to not run in parallel. > thread_pool/pool_size should be the > > way to go right? > > I've put in sample code with its results > below: > > My Rails controller kicks off 5 jobs in a > loop - each calling the > > run_concurrent method in my foo worker. > run_concurrent then calls > > my_actual_method which just logger.infos a > message with a time stamp and > > sleeps for 5 seconds to simulate a long > running job. I did this as > > per > http://backgroundrb.rubyforge.org/workers/#thread_pool Since I'm calling > > this via a defer and have a pool size of 10, > I expect to see that > > my_actual_method actually gets called 5 > times in quick succession (since the > > pool size is greater than the # of calls). > However I find that > > run_concurrent doesn't even call > my_actual_method. Here's the output from my > > backgroundrb log when I go to > http://mysite.com/some/foobar > > Can someone help me understand what I'm > doing wrong here? > > Thanks, > > Raghu > > > ==================================================== > > Rails controller code > > class SomeController < ApplicationController > > def foobar > > i = 0 > > while i < 5 > > worker = > MiddleMan.worker(:foo_worker) > > result = > worker.run_concurrent(:job_key => > > random_string(10)) > > i += 1 > > end > > render :text => 'all done at > ' + Time.now.to_s > > end > > end > > > ==================================================== > > Worker code > > class FooWorker < BackgrounDRb::MetaWorker > > set_worker_name :foo_worker > > pool_size 10 > > def create > > # this method is called, > when worker is loaded for the first > > time > > end > > def run_concurrent(args) > > logger.info "*** > FOO_WORKER/RUN_CONCURRENT at " + > > Time.now.to_s > > > thread_pool.defer(:my_actual_method) > > end > > def my_actual_method(args) > > logger.info "*** > FOO_WORKER/MY_ACTUAL_METHOD at " + > > Time.now.to_s > > sleep 5 > > end > > end > > > ==================================================== > > Here's the output > > # Logfile created on Wed Dec 31 09:15:21 > +0000 2008 by / > > foo_worker started (pid:5087) > > Schedules for worker loaded (pid:5087) > > run_concurrent job_keydfvq5o4m0s (pid:5087) > > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 > 09:15:26 +0000 2008 (pid:5087) > > run_concurrent job_keyy4tascam6k (pid:5087) > > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 > 09:15:26 +0000 2008 (pid:5087) > > run_concurrent job_key8gw2eegeqs (pid:5087) > > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 > 09:15:26 +0000 2008 (pid:5087) > > run_concurrent job_keyywb1oop73t (pid:5087) > > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 > 09:15:26 +0000 2008 (pid:5087) > > run_concurrent job_key17gfkqtzjh (pid:5087) > > *** FOO_WORKER/RUN_CONCURRENT at Wed Dec 31 > 09:15:26 +0000 2008 (pid:5087) > > > ==================================================== > > > > > > _______________________________________________ > > Backgroundrb-devel mailing list > > Backgroundrb-devel at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > > -- > Let them talk of their oriental summer climes > of everlasting > conservatories; give me the privilege of > making my own summer with my > own coals. > > http://gnufied.org > > > > > > From ramon.tayag at gmail.com Mon Jan 5 00:35:55 2009 From: ramon.tayag at gmail.com (Ramon Tayag) Date: Mon, 5 Jan 2009 13:35:55 +0800 Subject: [Backgroundrb-devel] Bdrb's select statements in log Message-ID: I find it a bit difficult to read the server log when backgroundrb outputs its sql activity: BdrbJobQueue Load (0.000405) SELECT * FROM `bdrb_job_queues` WHERE ( worker_name = 'mail_worker' AND taken = 0 AND scheduled_at <= '2009-01-05 05:33:51' ) LIMIT 1 FOR UPDATE SQL (0.000088) COMMIT How do I turn this off? I tried the "log: background" configuration option but it didn't do anything. Thanks, Ramon Tayag From gethemant at gmail.com Mon Jan 5 00:59:38 2009 From: gethemant at gmail.com (hemant) Date: Mon, 5 Jan 2009 11:29:38 +0530 Subject: [Backgroundrb-devel] Bdrb's select statements in log In-Reply-To: References: Message-ID: Actually this is from ActiveRecord statements inside bdrb (not exactly from bdrb). You can disable logging in AR if inside backgroundrb (check AR api) On Mon, Jan 5, 2009 at 11:05 AM, Ramon Tayag wrote: > I find it a bit difficult to read the server log when backgroundrb > outputs its sql activity: > > BdrbJobQueue Load (0.000405) SELECT * FROM `bdrb_job_queues` WHERE ( > worker_name = 'mail_worker' AND taken = 0 AND scheduled_at <= > '2009-01-05 05:33:51' ) LIMIT 1 FOR UPDATE > SQL (0.000088) COMMIT > > How do I turn this off? I tried the "log: background" configuration > option but it didn't do anything. > > Thanks, > Ramon Tayag > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From akyrho at gmail.com Mon Jan 5 05:45:45 2009 From: akyrho at gmail.com (=?ISO-8859-1?Q?Bousmanne_C=E9dric?=) Date: Mon, 5 Jan 2009 11:45:45 +0100 Subject: [Backgroundrb-devel] packet_worker_runner error Message-ID: <586195F6-27B2-4D15-8363-067628314B6C@gmail.com> Hi, I still have the following error while loading backgroundrb : /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/ packet-0.1.14/lib/packet/packet_master.rb:116:in `exec': No such file or directory - packet_worker_runner 14:11:maintenance_worker:64:/home/ antz/declix.com/production/lib/workers (Errno::ENOENT) from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_master.rb:116:in `fork_and_load' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_master.rb:80:in `start_worker' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:101:in `load_and_invoke' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:88:in `reload_workers' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:85:in `each' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:85:in `reload_workers' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:84:in `each' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:84:in `reload_workers' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:21:in `initialize' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_core.rb:141:in `call' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_core.rb:141:in `start_reactor' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_core.rb:139:in `loop' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_core.rb:139:in `start_reactor' from /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/ gems/packet-0.1.14/lib/packet/packet_master.rb:21:in `run' from /home/antz/declix.com/production/vendor/plugins/ backgroundrb/server/lib/master_proxy.rb:14:in `initialize' from /home/antz/declix.com/production/script/backgroundrb: 42:in `new' from /home/antz/declix.com/production/script/backgroundrb:42 My gem list : actionmailer (2.2.2) actionpack (2.2.2) activerecord (2.2.2) activeresource (2.2.2) activesupport (2.2.2) cached_model (1.3.1) cgi_multipart_eof_fix (2.5.0) chronic (0.2.3) daemons (1.0.10) fastthread (1.0.1) file-tail (1.0.3) gem_plugin (0.2.3) hoe (1.8.2) hpricot (0.6.164) mechanize (0.9.0) memcache-client (1.5.0) mysql (2.7) nokogiri (1.1.0) packet (0.1.14, 0.1.5) passenger (2.0.6) postgres (0.7.9.2008.01.28) rack (0.4.0) rails (2.2.2) rake (0.8.3) rmagick (2.8.0) rubyforge (1.0.1) unicode (0.1) ZenTest (3.11.0) PATH value : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/ bin:/usr/games:/opt/ruby-enterprise-1.8.6-20081215/bin Does anyone have an idea how can i properly run backgroundrb? Thanks a lot. From gethemant at gmail.com Mon Jan 5 06:13:42 2009 From: gethemant at gmail.com (hemant) Date: Mon, 5 Jan 2009 16:43:42 +0530 Subject: [Backgroundrb-devel] packet_worker_runner error In-Reply-To: <586195F6-27B2-4D15-8363-067628314B6C@gmail.com> References: <586195F6-27B2-4D15-8363-067628314B6C@gmail.com> Message-ID: packet gem when installed installs an executable called "packet_worker_runner". The location where this executable is installed can be found using: $gem env|grep "EXECUTABLE DIRECTORY"|cut -d: -f2 Make sure that, you have that directory in PATH, before starting backgroundrb. On Mon, Jan 5, 2009 at 4:15 PM, Bousmanne C?dric wrote: > Hi, > > I still have the following error while loading backgroundrb : > > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_master.rb:116:in > `exec': No such file or directory - packet_worker_runner > 14:11:maintenance_worker:64:/home/antz/declix.com/production/lib/workers > (Errno::ENOENT) > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_master.rb:116:in > `fork_and_load' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_master.rb:80:in > `start_worker' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:101:in > `load_and_invoke' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:88:in > `reload_workers' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:85:in > `each' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:85:in > `reload_workers' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:84:in > `each' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:84:in > `reload_workers' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:21:in > `initialize' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_core.rb:141:in > `call' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_core.rb:141:in > `start_reactor' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_core.rb:139:in > `loop' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_core.rb:139:in > `start_reactor' > from > /opt/ruby-enterprise-1.8.6-20081215/lib/ruby/gems/1.8/gems/packet-0.1.14/lib/packet/packet_master.rb:21:in > `run' > from > /home/antz/declix.com/production/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in > `initialize' > from /home/antz/declix.com/production/script/backgroundrb:42:in `new' > from /home/antz/declix.com/production/script/backgroundrb:42 > > My gem list : > > actionmailer (2.2.2) > actionpack (2.2.2) > activerecord (2.2.2) > activeresource (2.2.2) > activesupport (2.2.2) > cached_model (1.3.1) > cgi_multipart_eof_fix (2.5.0) > chronic (0.2.3) > daemons (1.0.10) > fastthread (1.0.1) > file-tail (1.0.3) > gem_plugin (0.2.3) > hoe (1.8.2) > hpricot (0.6.164) > mechanize (0.9.0) > memcache-client (1.5.0) > mysql (2.7) > nokogiri (1.1.0) > packet (0.1.14, 0.1.5) > passenger (2.0.6) > postgres (0.7.9.2008.01.28) > rack (0.4.0) > rails (2.2.2) > rake (0.8.3) > rmagick (2.8.0) > rubyforge (1.0.1) > unicode (0.1) > ZenTest (3.11.0) > > PATH value : > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/ruby-enterprise-1.8.6-20081215/bin > > Does anyone have an idea how can i properly run backgroundrb? Thanks a lot. > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From cameron at bigbangtechnology.com Mon Jan 5 12:22:47 2009 From: cameron at bigbangtechnology.com (Cameron Westland) Date: Mon, 5 Jan 2009 12:22:47 -0500 Subject: [Backgroundrb-devel] Problems Reading ask_result in controller Message-ID: <44B11CE2-FE84-4D3F-A3F1-6BB4D5C2D2FE@bigbangtechnology.com> Hey Folks, I am running Rails 2.1.1 and am having problems getting a result from ask_result in my controller: MiddleMan.worker(:uploader_worker, 'abc123').ask_result("percent_complete") is returning nil I am creating my worker like this: MiddleMan.new_worker(:worker => :uploader_worker, :worker_key => 'abc123') worker = MiddleMan.worker(:uploader_worker, 'abc123') worker.async_upload(:arg => files_to_upload) When I run the above command in script/console I get a return value MiddleMan.worker(:uploader_worker, 'abc123').ask_result("percent_complete") -> 14 I am using periodically_call_remote to call my controller action. Oh and I'm using memcache to store the data in cache. Really lost here why the controller can't access the data. Thanks! Cameron Big Bang Technology Inc. http://www.bigbangtechnology.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rut216 at gmail.com Mon Jan 5 19:58:34 2009 From: rut216 at gmail.com (Alex) Date: Mon, 5 Jan 2009 18:58:34 -0600 Subject: [Backgroundrb-devel] ask_status/register_status deprecated??? Message-ID: Hi, I've been using bdrb for a while and just recently found the need to use the ask_status/register_status functionality of my worker. This is the error I get whenever the register_status method is encountered: ...lib/workers/session_worker.rb:10:in `create': undefined method `register_status' for # (NoMethodError) from /home/rut216/dev/illinicourses/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:129:in `worker_init' from /var/lib/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_worker.rb:19:in `start_worker' from /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:33:in `load_worker' from /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:26:in `initialize' from /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47:in `new' from /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47 from /var/lib/gems/1.8/bin/packet_worker_runner:16:in `load' from /var/lib/gems/1.8/bin/packet_worker_runner:16 I tried the newest git and svn builds... I also greped the whole backgroundrb project directory for ask_status and register_status but found traces of neither. Is there a new way to probe a worker to check its progress? Thanks, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From buenos79 at hotmail.com Tue Jan 6 18:38:14 2009 From: buenos79 at hotmail.com (Mark D) Date: Tue, 6 Jan 2009 15:38:14 -0800 Subject: [Backgroundrb-devel] =?windows-1256?q?_Configure_multiple_workers?= =?windows-1256?q?_for_persistent_job_queue=FE?= Message-ID: Hi, I'm using a persistent job queue with a single worker thread to process the jobs. However the jobs are being created faster than the worker can handle them. Is it possible to configure backgroundrb to start multiple copies of the same worker to pick up these jobs? Thanks, Mark _________________________________________________________________ Send e-mail faster without improving your typing skills. http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_speed_122008 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raghu.srinivasan at gmail.com Tue Jan 6 19:12:33 2009 From: raghu.srinivasan at gmail.com (Raghu Srinivasan) Date: Tue, 6 Jan 2009 16:12:33 -0800 Subject: [Backgroundrb-devel] Is it possible to stop/start/restart one particular worker Message-ID: <6482c06a0901061612x11971de7pb628b262ff9b349c@mail.gmail.com> I have multiple workers in my lib/workers directory and when I start BDRB, each worker gets going. My question is: is there a way to stop and restart one particular worker? Right now I am doing a "script/background stop" followed by a "script/background start" which stops and restarts *all* the workers. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Tue Jan 6 21:40:16 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 08:10:16 +0530 Subject: [Backgroundrb-devel] Problems Reading ask_result in controller In-Reply-To: <44B11CE2-FE84-4D3F-A3F1-6BB4D5C2D2FE@bigbangtechnology.com> References: <44B11CE2-FE84-4D3F-A3F1-6BB4D5C2D2FE@bigbangtechnology.com> Message-ID: If worker results are coming from console, but not from controller, it might be because results hasn't been cached yet? If you can post worker code, it might be useful to debug further. On Mon, Jan 5, 2009 at 10:52 PM, Cameron Westland wrote: > Hey Folks, > I am running Rails 2.1.1 and am having problems getting a result from > ask_result in my controller: > MiddleMan.worker(:uploader_worker, 'abc123').ask_result("percent_complete") > is returning nil > I am creating my worker like this: > MiddleMan.new_worker(:worker => :uploader_worker, :worker_key => 'abc123') > > worker = MiddleMan.worker(:uploader_worker, 'abc123') > worker.async_upload(:arg => files_to_upload) > When I run the above command in script/console I get a return value > MiddleMan.worker(:uploader_worker, 'abc123').ask_result("percent_complete") > -> 14 > > I am using periodically_call_remote to call my controller action. > Oh and I'm using memcache to store the data in cache. > Really lost here why the controller can't access the data. > Thanks! > > > Cameron > Big Bang Technology Inc. > http://www.bigbangtechnology.com/ > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Jan 6 21:41:58 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 08:11:58 +0530 Subject: [Backgroundrb-devel] ask_status/register_status deprecated??? In-Reply-To: References: Message-ID: Hi Alex, Yup, we changed the API for results a bit. Details can be found here: http://backgroundrb.rubyforge.org/workers/#result_caching Let us know if that information is not enough. On Tue, Jan 6, 2009 at 6:28 AM, Alex wrote: > Hi, > > I've been using bdrb for a while and just recently found the need to use the > ask_status/register_status functionality of my worker. > > This is the error I get whenever the register_status method is encountered: > > ...lib/workers/session_worker.rb:10:in `create': undefined method > `register_status' for # (NoMethodError) > from > /home/rut216/dev/illinicourses/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:129:in > `worker_init' > from > /var/lib/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_worker.rb:19:in > `start_worker' > from > /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:33:in > `load_worker' > from > /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:26:in > `initialize' > from > /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47:in `new' > from > /var/lib/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47 > from /var/lib/gems/1.8/bin/packet_worker_runner:16:in `load' > from /var/lib/gems/1.8/bin/packet_worker_runner:16 > > I tried the newest git and svn builds... > > I also greped the whole backgroundrb project directory for ask_status and > register_status but found traces of neither. Is there a new way > > to probe a worker to check its progress? > > Thanks, > > Alex > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Jan 6 21:47:07 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 08:17:07 +0530 Subject: [Backgroundrb-devel] =?windows-1256?q?Configure_multiple_workers_?= =?windows-1256?q?for_persistent_job_queue=FE?= In-Reply-To: References: Message-ID: 2009/1/7 Mark D : > Hi, > > I'm using a persistent job queue with a single worker thread to process the > jobs. However the jobs are being created faster than the worker can handle > them. Is it possible to configure backgroundrb to start multiple copies of > the same worker to pick up these jobs? > 1. You can use one worker as master (the one that takes jobs out of the queue) and have that worker start multiple copies of itself (using MiddleMan.new_worker) and hand over the jobs to these workers. 2. There isn't a builtin functionality of auto-adjusting worker count if number of tasks in the queue is high. But if your tasks are inherently IO bound, you can use remove item from queue and run it inside thread pool. From gethemant at gmail.com Tue Jan 6 21:49:55 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 08:19:55 +0530 Subject: [Backgroundrb-devel] Is it possible to stop/start/restart one particular worker In-Reply-To: <6482c06a0901061612x11971de7pb628b262ff9b349c@mail.gmail.com> References: <6482c06a0901061612x11971de7pb628b262ff9b349c@mail.gmail.com> Message-ID: On Wed, Jan 7, 2009 at 5:42 AM, Raghu Srinivasan wrote: > I have multiple workers in my lib/workers directory and when I start BDRB, > each worker gets going. > > My question is: is there a way to stop and restart one particular worker? > Right now I am doing a "script/background stop" followed by a > "script/background start" which stops and restarts *all* the workers. Not currently. But I am working out support for this. There are basically three issues that will be covered in next release: 1. Error messages shouldn't be lost, no matter what. 2. Workers can be independently started/stopped. 3. Further fixes for thread pool. From jonathan at column3.net Wed Jan 7 10:45:56 2009 From: jonathan at column3.net (Jonathan Donaldson) Date: Wed, 7 Jan 2009 10:45:56 -0500 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed Message-ID: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> Sorry if this hit twice, my mailer was using the .com version of my email address not the .net one I signed up with: Using Backgroundrb pulled from github yesterday, and rails 2.2.2. In my worker, I send lots of progress back as I step through the tasks, ie: @items.each do |itm| ...stuff happens here @results[:created_items] += 1 cache[some_key] = @results end in next loop: @ results[:errors] = "Error on Item 4", cache[some_key] = @results when I call ask_result on the worker, I get nil back until after the worker is completely done, then I get my results.... ideas? This is to drive a progress bar and other 'realtime' feedback... Thanks! Jonathan Thanks! Jonathan Donaldson Partner, Column3 LLC. jonathan at column3.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Jan 7 11:02:49 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 21:32:49 +0530 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> Message-ID: On Wed, Jan 7, 2009 at 9:15 PM, Jonathan Donaldson wrote: > Sorry if this hit twice, my mailer was using the .com version of my email > address not the .net one I signed up with: > > > Using Backgroundrb pulled from github yesterday, and rails 2.2.2. > In my worker, I send lots of progress back as I step through the tasks, ie: > > @items.each do |itm| > ...stuff happens here > @results[:created_items] += 1 > cache[some_key] = @results > end > in next loop: > @ results[:errors] = "Error on Item 4", > cache[some_key] = @results > > when I call ask_result on the worker, I get nil back until after the worker > is completely done, then I get my results.... > > ideas? This is to drive a progress bar and other 'realtime' feedback... You will have to use Memcache based backend of result storage for "realtime" feedback. Inbuilt cache behavior is like what you mentioned. For more info: http://backgroundrb.rubyforge.org/workers/#result_caching From jonathan at column3.net Wed Jan 7 11:07:38 2009 From: jonathan at column3.net (Jonathan Donaldson) Date: Wed, 7 Jan 2009 11:07:38 -0500 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> Message-ID: <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> I am using memcache, with my backgroundrb.yml set like: --- :backgroundrb: :port: 11006 :ip: 127.0.0.1 :result_storage: memcache :memcache: "127.0.0.1:11211" Is there something I need to do in the worker to get it to use it? I see the same behavior with both... On Jan 7, 2009, at 11:02 AM, hemant wrote: > On Wed, Jan 7, 2009 at 9:15 PM, Jonathan Donaldson > wrote: >> Sorry if this hit twice, my mailer was using the .com version of my >> email >> address not the .net one I signed up with: >> >> >> Using Backgroundrb pulled from github yesterday, and rails 2.2.2. >> In my worker, I send lots of progress back as I step through the >> tasks, ie: >> >> @items.each do |itm| >> ...stuff happens here >> @results[:created_items] += 1 >> cache[some_key] = @results >> end >> in next loop: >> @ results[:errors] = "Error on Item 4", >> cache[some_key] = @results >> >> when I call ask_result on the worker, I get nil back until after >> the worker >> is completely done, then I get my results.... >> >> ideas? This is to drive a progress bar and other 'realtime' >> feedback... > > You will have to use Memcache based backend of result storage for > "realtime" feedback. Inbuilt cache behavior is like what you > mentioned. > For more info: > > http://backgroundrb.rubyforge.org/workers/#result_caching Thanks! Jonathan Donaldson Partner, Column3 LLC. jonathan at column3.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Jan 7 11:52:47 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 22:22:47 +0530 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> Message-ID: In that case, it would be a bug. I am looking into it, in the meanwhile please file a bug report. On Wed, Jan 7, 2009 at 9:37 PM, Jonathan Donaldson wrote: > I am using memcache, with my backgroundrb.yml set like: > --- > :backgroundrb: > :port: 11006 > :ip: 127.0.0.1 > :result_storage: memcache > :memcache: "127.0.0.1:11211" > Is there something I need to do in the worker to get it to use it? I see the > same behavior with both... > > On Jan 7, 2009, at 11:02 AM, hemant wrote: > > On Wed, Jan 7, 2009 at 9:15 PM, Jonathan Donaldson > wrote: > > Sorry if this hit twice, my mailer was using the .com version of my email > > address not the .net one I signed up with: > > > Using Backgroundrb pulled from github yesterday, and rails 2.2.2. > > In my worker, I send lots of progress back as I step through the tasks, ie: > > @items.each do |itm| > > ...stuff happens here > > @results[:created_items] += 1 > > cache[some_key] = @results > > end > > in next loop: > > @ results[:errors] = "Error on Item 4", > > cache[some_key] = @results > > when I call ask_result on the worker, I get nil back until after the worker > > is completely done, then I get my results.... > > ideas? This is to drive a progress bar and other 'realtime' feedback... > > You will have to use Memcache based backend of result storage for > "realtime" feedback. Inbuilt cache behavior is like what you > mentioned. > For more info: > > http://backgroundrb.rubyforge.org/workers/#result_caching > > > > > Thanks! > Jonathan Donaldson > Partner, Column3 LLC. > jonathan at column3.com > > > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From cameron at bigbangtechnology.com Wed Jan 7 12:58:22 2009 From: cameron at bigbangtechnology.com (Cameron Westland) Date: Wed, 7 Jan 2009 12:58:22 -0500 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> Message-ID: <4FC0D051-ADBC-4AEA-B3AD-C53AAE37A186@bigbangtechnology.com> On 7-Jan-09, at 11:52 AM, hemant wrote: > In that case, it would be a bug. I am looking into it, in the > meanwhile please file a bug report. I am having the exact same issue. Where are bug reports filed? Cameron Big Bang Technology Inc. http://www.bigbangtechnology.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Jan 7 12:59:38 2009 From: gethemant at gmail.com (hemant) Date: Wed, 7 Jan 2009 23:29:38 +0530 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: <4FC0D051-ADBC-4AEA-B3AD-C53AAE37A186@bigbangtechnology.com> References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> <4FC0D051-ADBC-4AEA-B3AD-C53AAE37A186@bigbangtechnology.com> Message-ID: On Wed, Jan 7, 2009 at 11:28 PM, Cameron Westland wrote: > > On 7-Jan-09, at 11:52 AM, hemant wrote: > > In that case, it would be a bug. I am looking into it, in the > meanwhile please file a bug report. > > I am having the exact same issue. > Where are bug reports filed? http://backgroundrb.devjavu.com/report/ From cameron at bigbangtechnology.com Wed Jan 7 13:34:51 2009 From: cameron at bigbangtechnology.com (Cameron Westland) Date: Wed, 7 Jan 2009 13:34:51 -0500 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> <4FC0D051-ADBC-4AEA-B3AD-C53AAE37A186@bigbangtechnology.com> Message-ID: <6A5ABD0C-E0AA-4813-B07A-641DB92FDA21@bigbangtechnology.com> On 7-Jan-09, at 12:59 PM, hemant wrote: > http://backgroundrb.devjavu.com/report/ I filed a bug: http://backgroundrb.devjavu.com/ticket/106 Hopefully this is an easy fix. Let me know what I can do to help. Cameron Big Bang Technology Inc. http://www.bigbangtechnology.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at column3.net Wed Jan 7 15:03:40 2009 From: jonathan at column3.net (Jonathan Donaldson) Date: Wed, 7 Jan 2009 15:03:40 -0500 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> Message-ID: Hemant, My deep apologies, it appears that this is working as expected, I apparently read the documentation on the wiki incorrectly. I set up a very basic test app and ran through your examples, and everything worked. I then went back into my code and made some minor change (not sure if I can figure out what it was ;-) ), and all is working now..... Please mark my bug under the 'dumb submission' tag... It would be great if the documentation was a bit clearer, maybe do a O'Reilly quick look or something so you can get paid for the documentation effort.... On Jan 7, 2009, at 11:52 AM, hemant wrote: > In that case, it would be a bug. I am looking into it, in the > meanwhile please file a bug report. > > > > On Wed, Jan 7, 2009 at 9:37 PM, Jonathan Donaldson > wrote: >> I am using memcache, with my backgroundrb.yml set like: >> --- >> :backgroundrb: >> :port: 11006 >> :ip: 127.0.0.1 >> :result_storage: memcache >> :memcache: "127.0.0.1:11211" >> Is there something I need to do in the worker to get it to use it? >> I see the >> same behavior with both... >> >> On Jan 7, 2009, at 11:02 AM, hemant wrote: >> >> On Wed, Jan 7, 2009 at 9:15 PM, Jonathan Donaldson > > >> wrote: >> >> Sorry if this hit twice, my mailer was using the .com version of my >> email >> >> address not the .net one I signed up with: >> >> >> Using Backgroundrb pulled from github yesterday, and rails 2.2.2. >> >> In my worker, I send lots of progress back as I step through the >> tasks, ie: >> >> @items.each do |itm| >> >> ...stuff happens here >> >> @results[:created_items] += 1 >> >> cache[some_key] = @results >> >> end >> >> in next loop: >> >> @ results[:errors] = "Error on Item 4", >> >> cache[some_key] = @results >> >> when I call ask_result on the worker, I get nil back until after >> the worker >> >> is completely done, then I get my results.... >> >> ideas? This is to drive a progress bar and other 'realtime' >> feedback... >> >> You will have to use Memcache based backend of result storage for >> "realtime" feedback. Inbuilt cache behavior is like what you >> mentioned. >> For more info: >> >> http://backgroundrb.rubyforge.org/workers/#result_caching >> >> >> >> >> Thanks! >> Jonathan Donaldson >> Partner, Column3 LLC. >> jonathan at column3.com >> >> >> >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org Thanks! Jonathan Donaldson Partner, Column3 LLC. jonathan at column3.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Jan 7 15:17:34 2009 From: gethemant at gmail.com (hemant) Date: Thu, 8 Jan 2009 01:47:34 +0530 Subject: [Backgroundrb-devel] Results from worker are not available until after worker is completed In-Reply-To: References: <5524D772-A42E-4BBD-BA2F-58D30A74B994@column3.net> <1E9F72DB-11E8-4708-9C09-65DAD139038F@column3.net> Message-ID: On Thu, Jan 8, 2009 at 1:33 AM, Jonathan Donaldson wrote: > Hemant, > My deep apologies, it appears that this is working as expected, I apparently > read the documentation on the wiki incorrectly. I set up a very basic test > app and ran through your examples, and everything worked. I then went back > into my code and made some minor change (not sure if I can figure out what > it was ;-) ), and all is working now..... > Please mark my bug under the 'dumb submission' tag... Cool. No problems. Also a note to Cameron, can you take a hard look at your app and make sure its correctly coded? Because two bug reports are related. > > It would be great if the documentation was a bit clearer, maybe do a > O'Reilly quick look or something so you can get paid for the documentation > effort.... Actually, all my efforts to get donation for my work on bdrb went in vain. So, yes I can be motivated a bit more if donations came by, otherwise we are fine and dandy. From cwdinfo at gmail.com Tue Jan 13 14:38:17 2009 From: cwdinfo at gmail.com (s.ross) Date: Tue, 13 Jan 2009 11:38:17 -0800 Subject: [Backgroundrb-devel] missing thread.so on CentOS Message-ID: I have a peculiar problem and it only happens when I start backgroundrb. I'm running Ruby 1.8.6 Ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-linux] on CentOS5 Intel(R) Xeon(R) CPU E5410 @ 2.33GHz Up until now, everything has been working just fine but recently I have been unable to restart backgroundrb. The error is /usr/lib/ruby/1.8/thread.rb:5:in `require': no such file to load -- thread.so (LoadError) So I turned to The Google, and found an article where it was suggested that I copy the thread.so from /usr/lib/ruby/1.8/yourarchitecture/ to the parent directory. I tried this and got: /usr/lib/ruby/1.8/thread.so: undefined symbol: rb_eThreadError - /usr/ lib/ruby/1.8/thread.so (LoadError) Something is obviously hosed, but I'm not seeing what. Any pointers on how to fix this problem? Thanks Steve From gethemant at gmail.com Tue Jan 13 15:12:15 2009 From: gethemant at gmail.com (hemant) Date: Wed, 14 Jan 2009 01:42:15 +0530 Subject: [Backgroundrb-devel] missing thread.so on CentOS In-Reply-To: References: Message-ID: Whats the output of: gem env and: try printing $: when backgroundrb starts. On Wed, Jan 14, 2009 at 1:08 AM, s.ross wrote: > I have a peculiar problem and it only happens when I start backgroundrb. I'm > running Ruby 1.8.6 > > Ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-linux] > > on CentOS5 > Intel(R) Xeon(R) CPU E5410 @ 2.33GHz > > Up until now, everything has been working just fine but recently I have been > unable to restart backgroundrb. The error is > > /usr/lib/ruby/1.8/thread.rb:5:in `require': no such file to load -- > thread.so (LoadError) > > So I turned to The Google, and found an article where it was suggested that > I copy the thread.so from /usr/lib/ruby/1.8/yourarchitecture/ to the parent > directory. > > I tried this and got: > > /usr/lib/ruby/1.8/thread.so: undefined symbol: rb_eThreadError - > /usr/lib/ruby/1.8/thread.so (LoadError) > > Something is obviously hosed, but I'm not seeing what. > > Any pointers on how to fix this problem? > > Thanks > > Steve > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Jan 13 15:33:20 2009 From: gethemant at gmail.com (hemant) Date: Wed, 14 Jan 2009 02:03:20 +0530 Subject: [Backgroundrb-devel] missing thread.so on CentOS In-Reply-To: References: Message-ID: Can you try executing this code and see if it works: require "thread" thr = Thread.new do sleep(2) puts "Hello world" end thr.join If it doesn't it means your ruby installation is hosed up. It it does it means Rails is munging up with $LOAD_PATH and something is screwed up. On Wed, Jan 14, 2009 at 1:59 AM, s.ross wrote: > RubyGems Environment: > - RUBYGEMS VERSION: 1.3.1 > - RUBY VERSION: 1.8.6 (2008-08-11 patchlevel 287) [i686-linux] > - INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8 > - RUBY EXECUTABLE: /usr/local/bin/ruby > - EXECUTABLE DIRECTORY: /usr/local/bin > - RUBYGEMS PLATFORMS: > - ruby > - x86-linux > - GEM PATHS: > - /usr/local/lib/ruby/gems/1.8 > - /root/.gem/ruby/1.8 > - GEM CONFIGURATION: > - :update_sources => true > - :verbose => true > - :benchmark => false > - :backtrace => false > - :bulk_threshold => 1000 > - :sources => ["http://gems.rubyforge.org/", "http://gems.github.com", > "http://gems.github.com"] > - REMOTE SOURCES: > - http://gems.rubyforge.org/ > - http://gems.github.com > - http://gems.github.com > > I can't print anything when backgroundrb starts because it errs out > immediately as far as I can tell. > > Thanks for the quick reply. > > Steve > > > On Jan 13, 2009, at 12:12 PM, hemant wrote: > >> Whats the output of: >> >> gem env >> >> and: >> >> try printing $: when backgroundrb starts. >> >> >> >> On Wed, Jan 14, 2009 at 1:08 AM, s.ross wrote: >>> >>> I have a peculiar problem and it only happens when I start backgroundrb. >>> I'm >>> running Ruby 1.8.6 >>> >>> Ruby 1.8.6 (2008-08-11 patchlevel 287) [i686-linux] >>> >>> on CentOS5 >>> Intel(R) Xeon(R) CPU E5410 @ 2.33GHz >>> >>> Up until now, everything has been working just fine but recently I have >>> been >>> unable to restart backgroundrb. The error is >>> >>> /usr/lib/ruby/1.8/thread.rb:5:in `require': no such file to load -- >>> thread.so (LoadError) >>> >>> So I turned to The Google, and found an article where it was suggested >>> that >>> I copy the thread.so from /usr/lib/ruby/1.8/yourarchitecture/ to the >>> parent >>> directory. >>> >>> I tried this and got: >>> >>> /usr/lib/ruby/1.8/thread.so: undefined symbol: rb_eThreadError - >>> /usr/lib/ruby/1.8/thread.so (LoadError) >>> >>> Something is obviously hosed, but I'm not seeing what. >>> >>> Any pointers on how to fix this problem? >>> >>> Thanks >>> >>> Steve >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> >> >> >> -- >> Let them talk of their oriental summer climes of everlasting >> conservatories; give me the privilege of making my own summer with my >> own coals. >> >> http://gnufied.org > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From carl at 2be.ca Wed Jan 14 08:57:57 2009 From: carl at 2be.ca (Carl Pelletier) Date: Wed, 14 Jan 2009 08:57:57 -0500 Subject: [Backgroundrb-devel] add scheduled task to persistent table... Message-ID: Hi everyone, little question for you guys: I a setup a scheduled task to run like this in my backgroundrb.yml: :schedules: :test_worker: :sync_ezdeal: :trigger_args: 0 15 5 * * * * This is working fine, but what I would like is that when backgroundrb run the task, ha add it to the persistent queue. this way, i will have nothing to do to follow all task run in last days/weeks. I building a admin interface on top of all the tasks that running on my server so users can start,stop a faild task or just rerun it if something go wrong. The user can restart this task in every moment he want. Can you help with this? I google it and found nothing. Thanks! P.S. sorry for sending this message twice. I used the wrong email from. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Wed Jan 14 12:11:26 2009 From: gethemant at gmail.com (hemant) Date: Wed, 14 Jan 2009 22:41:26 +0530 Subject: [Backgroundrb-devel] add scheduled task to persistent table... In-Reply-To: References: Message-ID: On Wed, Jan 14, 2009 at 7:27 PM, Carl Pelletier wrote: > Hi everyone, little question for you guys: > > I a setup a scheduled task to run like this in my backgroundrb.yml: > > :schedules: > :test_worker: > :sync_ezdeal: > :trigger_args: 0 15 5 * * * * > > This is working fine, but what I would like is that when backgroundrb run > the task, ha add it to the persistent queue. this way, i will have nothing > to do to follow all task run in last days/weeks. I building a admin > interface on top of all the tasks that running on my server so users can > start,stop a faild task or just rerun it if something go wrong. > > The user can restart this task in every moment he want. > > Can you help with this? > > I google it and found nothing. Out of box, no. However, its relatively easy to do, so i am going to implement this (patches welcome!) From carl at 2be.ca Thu Jan 15 06:15:21 2009 From: carl at 2be.ca (Carl Pelletier) Date: Thu, 15 Jan 2009 06:15:21 -0500 Subject: [Backgroundrb-devel] add scheduled task to persistent table... In-Reply-To: References: Message-ID: I will please to help and patch what needed. Can you give me some way to go? Thanks Hemant and continu your good job. Carl Pelletier ruby and rails dev at Cossette.com agency On 14-Jan-09, at 12:11, hemant wrote: > On Wed, Jan 14, 2009 at 7:27 PM, Carl Pelletier wrote: >> Hi everyone, little question for you guys: >> >> I a setup a scheduled task to run like this in my backgroundrb.yml: >> >> :schedules: >> :test_worker: >> :sync_ezdeal: >> :trigger_args: 0 15 5 * * * * >> >> This is working fine, but what I would like is that when >> backgroundrb run >> the task, ha add it to the persistent queue. this way, i will have >> nothing >> to do to follow all task run in last days/weeks. I building a admin >> interface on top of all the tasks that running on my server so >> users can >> start,stop a faild task or just rerun it if something go wrong. >> >> The user can restart this task in every moment he want. >> >> Can you help with this? >> >> I google it and found nothing. > > Out of box, no. However, its relatively easy to do, so i am going to > implement this (patches welcome!) > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From carl at 2be.ca Thu Jan 15 06:13:21 2009 From: carl at 2be.ca (Carl Pelletier) Date: Thu, 15 Jan 2009 06:13:21 -0500 Subject: [Backgroundrb-devel] add scheduled task to persistent table... In-Reply-To: References: Message-ID: <0B4F7262-0AA0-4696-8E6E-86EC049A2ED7@2be.ca> I will please to help and patch what needed. Can you give me some way to go? Thanks Hemant and continu your good job. Carl Pelletier ruby and rails dev at Cossette.com agency On 14-Jan-09, at 12:11, hemant wrote: > On Wed, Jan 14, 2009 at 7:27 PM, Carl Pelletier wrote: >> Hi everyone, little question for you guys: >> >> I a setup a scheduled task to run like this in my backgroundrb.yml: >> >> :schedules: >> :test_worker: >> :sync_ezdeal: >> :trigger_args: 0 15 5 * * * * >> >> This is working fine, but what I would like is that when >> backgroundrb run >> the task, ha add it to the persistent queue. this way, i will have >> nothing >> to do to follow all task run in last days/weeks. I building a admin >> interface on top of all the tasks that running on my server so >> users can >> start,stop a faild task or just rerun it if something go wrong. >> >> The user can restart this task in every moment he want. >> >> Can you help with this? >> >> I google it and found nothing. > > Out of box, no. However, its relatively easy to do, so i am going to > implement this (patches welcome!) > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at 2be.ca Thu Jan 15 07:58:17 2009 From: carl at 2be.ca (Carl Pelletier) Date: Thu, 15 Jan 2009 07:58:17 -0500 Subject: [Backgroundrb-devel] Something I just don't understand about the workers and job_key Message-ID: Hi everyone, I will try to be concise as possible to explain my problem. I building a admin section on my site to manage all my ruby scripts. I want to remove the script from cron and put them in backgroundrb so every admin user can start/stop those scripts when they want. So the script start first from the scheduler(yml) to be sure they run each day. The way I understand bgrb, less worker is better. So I would like to reuse the worker started from the scheduler in my rails admin to start again the task sync_ezdeal. I have a worker started from backgroundrb.yml file like this: --- :backgroundrb: :port: 11006 :ip: 0.0.0.0 :result_storage: memcache :memcache: "localhost:11211,localhost:11211" :schedules: :ezdeal_worker: :sync_ezdeal: :trigger_args: 0 30 3 * * * * ---------------- #This script run each night at 3:30. My Worker: class EzdealWorker < BackgrounDRb::MetaWorker set_worker_name :ezdeal_worker pool_size 1 def sync_ezdeal(args = nil) .... #do job cache[:filename] = xml_file # This is the filename of a generated xml file that the job sync_ezdeal build and save on disk. This filename must be unique each time the task sync_ezdeal is called. So i can look at the file from the admin site. end My call in the controller to start a new job is that: MiddleMan.worker(:ezdeal_worker).enq_sync_ezdeal(:job_key =>current_time, :arg =>{:filename=>current_time,:local =>'/Users/carl/ gm_core/log/ezdeal/ezdeal.xml'}) The job is queued correctly but when I try to get the :filename that doesnt work. This is my view: <% @tasks = BdrbJobQueue.find(:all,:order =>'id DESC') @tasks.each do |task| %> <%=h task.id%> <%=h task.worker_method %> <%=h time_ago_in_words(task.submitted_at) %> <%=h time_ago_in_words(task.started_at) unless task.started_at.blank?%> <%=h time_ago_in_words(task.finished_at) if task.finished? %> <%=h task.finished? %> <%=link_to 'log_file ',ezdeal_log_file_admin_task_path (MiddleMan.worker(task.worker_name).ask_result(:filename)) if task.finished?%> ##### This is were i retreive the filename from the worker. This should came from the database?no? like task.args[:filename] <%end%> I don't known how to fetch the filename for each job (with the job_key??). Because the ask_result is alway overright when I start a new job. The way I understand it is the cache array is the same for all workers of the same type. so the cache[:filename] is overight each time the sync_ezdeal is called. I tried by creating a new worker each time, it's working, but that look overkill to me. Is there a way to retreive the good key? I saw in the database the field args that old the data push in the cache array. Is there a way to retreive it? Because currently the data look to have strange character in it. In the database it's a binary object.. Not sure what to do with that... ex:args | \004\010{\007:\015filename"\02414012009_155548:\012local"./Users/carl/ gm_core/log/ezdeal/ezdeal.xml Thanks for any help! I hope I am clear enough! Carl Pelletier From enzodm at gmail.com Fri Jan 16 18:40:30 2009 From: enzodm at gmail.com (Samer Masry) Date: Fri, 16 Jan 2009 15:40:30 -0800 Subject: [Backgroundrb-devel] What is the best way to test if the server is up? Message-ID: This returns true even though the brdb server is down MiddleMan.backend_connections.first.connection_status => true I can just attempt to create a new worker and catch "BackgrounDRb::NoServerAvailable", but is there a better way than this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From enzodm at gmail.com Fri Jan 16 19:26:38 2009 From: enzodm at gmail.com (Samer Masry) Date: Fri, 16 Jan 2009 16:26:38 -0800 Subject: [Backgroundrb-devel] What is the best way to test if the server is up? In-Reply-To: References: Message-ID: MiddleMan.backend_connections.first.connection_status will return false when I call if from the rails instance rather than the console. On Fri, Jan 16, 2009 at 3:40 PM, Samer Masry wrote: > This returns true even though the brdb server is down > > MiddleMan.backend_connections.first.connection_status > => true > > I can just attempt to create a new worker and catch > "BackgrounDRb::NoServerAvailable", but is there a better way than this? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cwdinfo at gmail.com Sun Jan 18 13:04:43 2009 From: cwdinfo at gmail.com (s.ross) Date: Sun, 18 Jan 2009 10:04:43 -0800 Subject: [Backgroundrb-devel] [PATCH] Quiet Job Logs Message-ID: <18BF66F9-B5FB-4B70-9C60-4EB9597CCE6C@gmail.com> The job logs can get pretty noisy and obscure what's going on in the rest of the application. This patch quiets the find and delete calls on the job queue. If you want those back on, just set BdrbJobQueue::verbose_logging to true. Apologies for no tests. This is just a hack to preserve my sanity, but consider if you are having this same problem. Steve -------------- next part -------------- A non-text attachment was scrubbed... Name: bdrb_job_queue.diff Type: application/octet-stream Size: 2419 bytes Desc: not available URL: -------------- next part -------------- From diogo.silva at w20.pt Tue Jan 20 15:46:59 2009 From: diogo.silva at w20.pt (Diogo Silva) Date: Tue, 20 Jan 2009 20:46:59 +0000 Subject: [Backgroundrb-devel] Backgroundrb assistance Message-ID: Hi, Don't know if it's the right address but I'm seeking help with my workers. I have two problems related to workers and backgroundrb: 1) I have a worker that uses soap/wsdlDriver and it's set to autoload and starts called by controller. This only makes a simple SOAP communication. It works most of the times. But sometimes worker dies and the reason is related with some timeout related with httpclient gem. When it dies it's a pain to make it work again. Down it's the code of my controller: def index prepare_job @id = "prepare" render 'admin/importar/worker' end def import execute_job @id = "execute" render 'admin/importar/worker' end def prepare_job start_worker("PREPARE_IMPORT") end def execute_job start_worker("EXECUTE_IMPORT") end def start_worker(action) worker = MiddleMan.ask_status(:worker => :iif_worker) if worker != nil && worker[:progress] < 4 flash[:notice] = "J?? existe um processo de importa????o a decorrer!" else MiddleMan.ask_work(:worker => :iif_worker, :worker_method => :do_work, :data => action) session[:job_key] = :import end end def get_progress if request.xhr? worker = MiddleMan.ask_status(:worker => :iif_worker) return if worker == nil progress_percent = worker[:progress] render :update do |page| page.call('progressPercent', 'progressbar', progress_percent) if progress_percent >= 100 @info = worker[:info] @total = worker[:total] if params[:id] == "prepare" page.replace_html 'import_content', :partial => 'admin/importar/todo' elsif params[:id] == "execute" flash[:notice] = "#{@total} marcas importas com sucesso!" page.replace_html 'import_content', :partial => 'admin/importar/done' end end end else redirect_to :action => 'index' end end And below some code of worker: require 'rexml/document' require 'soap/wsdlDriver' require 'xsd/mapping' class IifWorker < BackgrounDRb::MetaWorker set_worker_name :iif_worker #set_no_auto_load(true) #attr_reader :progress, :info, :total #def create(args = nil) # register_status('Dass') # Thread.abort_on_exception = true #end def do_work(action) @progress = 0 update_status if action == "PREPARE_IMPORT" prepare_import elsif action == "EXECUTE_IMPORT" execute_import end @progress = 100 update_status end def prepare_import stuff, chave = auth_and_query results = stuff.goldMineData.record @total = results.size @progress = 15 @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f @info = findRecordsToUpdate(results) update_status @soap.reset_stream end def execute_import stuff, chave = auth_and_query results = stuff.goldMineData.record @total = results.size @progress = 15 @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f @info = updateRecords(stuff.goldMineData.record) #@soap.UpdateGoldmine(:Key=>chave) update_status @soap.reset_stream end def auth_and_query factory = SOAP::WSDLDriverFactory.new(" http://infofranchisingwsp.ife.pt:8081/infofranchisingUpdateService.asmx?WSDL ") @soap = factory.create_rpc_driver soapResponse = @soap.GetToken(nil) token = soapResponse.getTokenResult chave = authenticate(token) @progress = 5 update_status soapResponseXML = @soap.GetXML(:Key=>chave) @progress = 10 update_status return soapResponseXML.getXMLResult, chave end def update_status register_status({:progress => @progress, :info => @info, :total => @total}) end def increment_progress @progress += @increment @progress = ("%3.2f" % @progress).to_f @progress = 100 if @progress > 100 end def findRecordsToUpdate(results) (...) Can you help me solve this problem? Thanks in advance, Diogo Silva -------------- next part -------------- An HTML attachment was scrubbed... URL: From gethemant at gmail.com Tue Jan 20 16:24:43 2009 From: gethemant at gmail.com (hemant) Date: Wed, 21 Jan 2009 02:54:43 +0530 Subject: [Backgroundrb-devel] Backgroundrb assistance In-Reply-To: References: Message-ID: You can handle the exception. Http exception would be either timeout or generic Exception class and hence wrap your external service invocation code in: begin # do some work rescue Timeout::Error # timeout error rescue # something else end It should do the trick. On Wed, Jan 21, 2009 at 2:16 AM, Diogo Silva wrote: > Hi, > Don't know if it's the right address but I'm seeking help with my workers. > > I have two problems related to workers and backgroundrb: > > 1) I have a worker that uses soap/wsdlDriver and it's set to autoload and > starts called by controller. This only makes a simple SOAP communication. It > works most of the times. But sometimes worker dies and the reason is related > with some timeout related with httpclient gem. When it dies it's a pain to > make it work again. > Down it's the code of my controller: > > def index > prepare_job > @id = "prepare" > render 'admin/importar/worker' > end > > def import > execute_job > @id = "execute" > render 'admin/importar/worker' > end > > def prepare_job > start_worker("PREPARE_IMPORT") > end > > def execute_job > start_worker("EXECUTE_IMPORT") > end > > def start_worker(action) > worker = MiddleMan.ask_status(:worker => :iif_worker) > > if worker != nil && worker[:progress] < 4 > flash[:notice] = "J?? existe um processo de importa????o a decorrer!" > else > MiddleMan.ask_work(:worker => :iif_worker, :worker_method => :do_work, > :data => action) > session[:job_key] = :import > end > end > > def get_progress > if request.xhr? > worker = MiddleMan.ask_status(:worker => :iif_worker) > return if worker == nil > > progress_percent = worker[:progress] > render :update do |page| > page.call('progressPercent', 'progressbar', progress_percent) > if progress_percent >= 100 > @info = worker[:info] > @total = worker[:total] > if params[:id] == "prepare" > page.replace_html 'import_content', :partial => > 'admin/importar/todo' > elsif params[:id] == "execute" > flash[:notice] = "#{@total} marcas importas com sucesso!" > page.replace_html 'import_content', :partial => > 'admin/importar/done' > end > end > end > else > redirect_to :action => 'index' > end > end > > And below some code of worker: > > require 'rexml/document' > require 'soap/wsdlDriver' > require 'xsd/mapping' > > class IifWorker < BackgrounDRb::MetaWorker > set_worker_name :iif_worker > #set_no_auto_load(true) > #attr_reader :progress, :info, :total > > #def create(args = nil) > # register_status('Dass') > # Thread.abort_on_exception = true > #end > > def do_work(action) > @progress = 0 > update_status > if action == "PREPARE_IMPORT" > prepare_import > elsif action == "EXECUTE_IMPORT" > execute_import > end > @progress = 100 > update_status > end > > def prepare_import > stuff, chave = auth_and_query > results = stuff.goldMineData.record > > @total = results.size > @progress = 15 > @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f > @info = findRecordsToUpdate(results) > > update_status > > @soap.reset_stream > end > > def execute_import > stuff, chave = auth_and_query > results = stuff.goldMineData.record > > @total = results.size > @progress = 15 > @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f > @info = updateRecords(stuff.goldMineData.record) > #@soap.UpdateGoldmine(:Key=>chave) > > update_status > > @soap.reset_stream > end > > def auth_and_query > factory = > SOAP::WSDLDriverFactory.new("http://infofranchisingwsp.ife.pt:8081/infofranchisingUpdateService.asmx?WSDL") > @soap = factory.create_rpc_driver > soapResponse = @soap.GetToken(nil) > token = soapResponse.getTokenResult > > chave = authenticate(token) > @progress = 5 > update_status > soapResponseXML = @soap.GetXML(:Key=>chave) > @progress = 10 > update_status > > return soapResponseXML.getXMLResult, chave > end > > def update_status > register_status({:progress => @progress, :info => @info, :total => > @total}) > end > > def increment_progress > @progress += @increment > @progress = ("%3.2f" % @progress).to_f > @progress = 100 if @progress > 100 > end > > def findRecordsToUpdate(results) > (...) > > Can you help me solve this problem? > > Thanks in advance, > Diogo Silva > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Jan 20 16:25:42 2009 From: gethemant at gmail.com (hemant) Date: Wed, 21 Jan 2009 02:55:42 +0530 Subject: [Backgroundrb-devel] [PATCH] Quiet Job Logs In-Reply-To: <18BF66F9-B5FB-4B70-9C60-4EB9597CCE6C@gmail.com> References: <18BF66F9-B5FB-4B70-9C60-4EB9597CCE6C@gmail.com> Message-ID: Hey, Thanks for the patch, I have applied it to my tree and will be there in next release. On Sun, Jan 18, 2009 at 11:34 PM, s.ross wrote: > The job logs can get pretty noisy and obscure what's going on in the rest of > the application. This patch quiets the find and delete calls on the job > queue. If you want those back on, just set BdrbJobQueue::verbose_logging to > true. > > Apologies for no tests. This is just a hack to preserve my sanity, but > consider if you are having this same problem. > > Steve > > > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From gethemant at gmail.com Tue Jan 20 16:40:00 2009 From: gethemant at gmail.com (hemant) Date: Wed, 21 Jan 2009 03:10:00 +0530 Subject: [Backgroundrb-devel] Backgroundrb assistance In-Reply-To: References: Message-ID: If my memory you mean - result cache, you should definitely use memcache in production. But i think you are using rather old version of the plugin and hence that option may not be available. On Wed, Jan 21, 2009 at 3:03 AM, Diogo Silva wrote: > Hi, > > Thanks for support and sorry for annoyance. I'll try the solution tomorrow > morning and let you know if worked. > > Another question. I'm having also problems related to failure in allocating > memory with backgroundrb. Which is the best solution while configurating > backgroudrb.yml? > > Best Regards, > Diogo Silva > > 2009/1/20 hemant >> >> You can handle the exception. Http exception would be either timeout >> or generic Exception class and hence wrap your external service >> invocation code in: >> >> begin >> # do some work >> rescue Timeout::Error >> # timeout error >> rescue >> # something else >> end >> >> It should do the trick. >> >> >> >> On Wed, Jan 21, 2009 at 2:16 AM, Diogo Silva wrote: >> > Hi, >> > Don't know if it's the right address but I'm seeking help with my >> > workers. >> > >> > I have two problems related to workers and backgroundrb: >> > >> > 1) I have a worker that uses soap/wsdlDriver and it's set to autoload >> > and >> > starts called by controller. This only makes a simple SOAP >> > communication. It >> > works most of the times. But sometimes worker dies and the reason is >> > related >> > with some timeout related with httpclient gem. When it dies it's a pain >> > to >> > make it work again. >> > Down it's the code of my controller: >> > >> > def index >> > prepare_job >> > @id = "prepare" >> > render 'admin/importar/worker' >> > end >> > >> > def import >> > execute_job >> > @id = "execute" >> > render 'admin/importar/worker' >> > end >> > >> > def prepare_job >> > start_worker("PREPARE_IMPORT") >> > end >> > >> > def execute_job >> > start_worker("EXECUTE_IMPORT") >> > end >> > >> > def start_worker(action) >> > worker = MiddleMan.ask_status(:worker => :iif_worker) >> > >> > if worker != nil && worker[:progress] < 4 >> > flash[:notice] = "J?? existe um processo de importa????o a >> > decorrer!" >> > else >> > MiddleMan.ask_work(:worker => :iif_worker, :worker_method => >> > :do_work, >> > :data => action) >> > session[:job_key] = :import >> > end >> > end >> > >> > def get_progress >> > if request.xhr? >> > worker = MiddleMan.ask_status(:worker => :iif_worker) >> > return if worker == nil >> > >> > progress_percent = worker[:progress] >> > render :update do |page| >> > page.call('progressPercent', 'progressbar', progress_percent) >> > if progress_percent >= 100 >> > @info = worker[:info] >> > @total = worker[:total] >> > if params[:id] == "prepare" >> > page.replace_html 'import_content', :partial => >> > 'admin/importar/todo' >> > elsif params[:id] == "execute" >> > flash[:notice] = "#{@total} marcas importas com sucesso!" >> > page.replace_html 'import_content', :partial => >> > 'admin/importar/done' >> > end >> > end >> > end >> > else >> > redirect_to :action => 'index' >> > end >> > end >> > >> > And below some code of worker: >> > >> > require 'rexml/document' >> > require 'soap/wsdlDriver' >> > require 'xsd/mapping' >> > >> > class IifWorker < BackgrounDRb::MetaWorker >> > set_worker_name :iif_worker >> > #set_no_auto_load(true) >> > #attr_reader :progress, :info, :total >> > >> > #def create(args = nil) >> > # register_status('Dass') >> > # Thread.abort_on_exception = true >> > #end >> > >> > def do_work(action) >> > @progress = 0 >> > update_status >> > if action == "PREPARE_IMPORT" >> > prepare_import >> > elsif action == "EXECUTE_IMPORT" >> > execute_import >> > end >> > @progress = 100 >> > update_status >> > end >> > >> > def prepare_import >> > stuff, chave = auth_and_query >> > results = stuff.goldMineData.record >> > >> > @total = results.size >> > @progress = 15 >> > @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f >> > @info = findRecordsToUpdate(results) >> > >> > update_status >> > >> > @soap.reset_stream >> > end >> > >> > def execute_import >> > stuff, chave = auth_and_query >> > results = stuff.goldMineData.record >> > >> > @total = results.size >> > @progress = 15 >> > @increment = ("%4.2f" % ((100.0 - @progress) / results.size)).to_f >> > @info = updateRecords(stuff.goldMineData.record) >> > #@soap.UpdateGoldmine(:Key=>chave) >> > >> > update_status >> > >> > @soap.reset_stream >> > end >> > >> > def auth_and_query >> > factory = >> > >> > SOAP::WSDLDriverFactory.new("http://infofranchisingwsp.ife.pt:8081/infofranchisingUpdateService.asmx?WSDL") >> > @soap = factory.create_rpc_driver >> > soapResponse = @soap.GetToken(nil) >> > token = soapResponse.getTokenResult >> > >> > chave = authenticate(token) >> > @progress = 5 >> > update_status >> > soapResponseXML = @soap.GetXML(:Key=>chave) >> > @progress = 10 >> > update_status >> > >> > return soapResponseXML.getXMLResult, chave >> > end >> > >> > def update_status >> > register_status({:progress => @progress, :info => @info, :total => >> > @total}) >> > end >> > >> > def increment_progress >> > @progress += @increment >> > @progress = ("%3.2f" % @progress).to_f >> > @progress = 100 if @progress > 100 >> > end >> > >> > def findRecordsToUpdate(results) >> > (...) >> > >> > Can you help me solve this problem? >> > >> > Thanks in advance, >> > Diogo Silva >> > >> > _______________________________________________ >> > Backgroundrb-devel mailing list >> > Backgroundrb-devel at rubyforge.org >> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > >> >> >> >> -- >> Let them talk of their oriental summer climes of everlasting >> conservatories; give me the privilege of making my own summer with my >> own coals. >> >> http://gnufied.org > > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From chad.thatcher at gmail.com Fri Jan 23 08:05:41 2009 From: chad.thatcher at gmail.com (Chad Thatcher) Date: Fri, 23 Jan 2009 13:05:41 +0000 Subject: [Backgroundrb-devel] Hello to the group Message-ID: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> Hello everyone. I am making several asynchronous calls to one particular method in my worker like this: MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) If I call this several times in a row, all subsequent calls are put in a queue and I would like to be able to keep track of this. The behavior is perfect because I only ever want reindex() running once at any one time but I would like to be able to show my users where in the queue their request is. Is there any way to view what async calls are lined up? The worker_info stuff doesn't seem to give away any details like this. I also thought about using "enq_" to schedule but is there a way to specify that the trigger is the completion of the previous scheduled instance and so on? Or is the scheduling purely time based? Kind regards, Chad. From gethemant at gmail.com Fri Jan 23 13:57:49 2009 From: gethemant at gmail.com (hemant) Date: Sat, 24 Jan 2009 00:27:49 +0530 Subject: [Backgroundrb-devel] Hello to the group In-Reply-To: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> References: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> Message-ID: That data is in write Queue of master worker. It can't be meaningfully extracted to provide how many tasks are in the queue. Put it other way, its not even in a Queue in traditional sense. Data is just there in a buffer so that master will write it to the socket when socket is ready for write. If you want Queue. You will have either use enq_xxx stuff or provide a sugar around asyc_index() method so as actual method is not directly called but is added to the thread pool (thread_pool.defer), this way you can query how many tasks are there in thread pool queue. However word "thread" comes with its own meaning and associated crapiness in ruby, so beware. On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher wrote: > Hello everyone. > > I am making several asynchronous calls to one particular method in my worker > like this: > > MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) > > If I call this several times in a row, all subsequent calls are put in a > queue and I would like to be able to keep track of this. The behavior is > perfect because I only ever want reindex() running once at any one time but > I would like to be able to show my users where in the queue their request > is. Is there any way to view what async calls are lined up? The > worker_info stuff doesn't seem to give away any details like this. > > I also thought about using "enq_" to schedule but is there a way to specify > that the trigger is the completion of the previous scheduled instance and so > on? Or is the scheduling purely time based? > > Kind regards, > > Chad. > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From chad.thatcher at gmail.com Fri Jan 23 15:16:40 2009 From: chad.thatcher at gmail.com (Chad Thatcher) Date: Fri, 23 Jan 2009 20:16:40 +0000 Subject: [Backgroundrb-devel] Hello to the group In-Reply-To: References: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> Message-ID: Thanks Hemant, Yes I was specifically trying to avoid threads. However, I am guessing I could configure it so that only one thread ran at one time right? That would cut down on the complexity. Is the enq_xxx stuff only time based? If so I wonder if it would be difficult to add a feature that allows you to add a whole bunch of tasks to the queue and then they just run one after the other in the order in which they were added. Maybe I could intercept persistent_job#finished! to find the next (grouped) task in the queue and update it to start a few seconds in the future so the bgrb picks it up and runs it etc. But that seems evil somehow :). I realise this might not be the target usage for backgroundrb but I look forward to hearing your thoughts because I'm hoping it won't be too difficult for me to do. The ultimate fall-back, of course, is to roll my own task queue that uses a table similar table to bdrb_job_queues but that seems wasteful when there is so much there for it already. All the best, Chad. On 23 Jan 2009, at 18:57, hemant wrote: > That data is in write Queue of master worker. It can't be meaningfully > extracted to provide how many tasks are in the queue. Put it other > way, its not even in a Queue in traditional sense. Data is just there > in a buffer so that master will write it to the socket when socket is > ready for write. > > If you want Queue. You will have either use enq_xxx stuff or provide a > sugar around asyc_index() method so as actual method is not directly > called but is added to the thread pool (thread_pool.defer), this way > you can query how many tasks are there in thread pool queue. > > However word "thread" comes with its own meaning and associated > crapiness in ruby, so beware. > > > On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher > wrote: >> Hello everyone. >> >> I am making several asynchronous calls to one particular method in >> my worker >> like this: >> >> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >> >> If I call this several times in a row, all subsequent calls are put >> in a >> queue and I would like to be able to keep track of this. The >> behavior is >> perfect because I only ever want reindex() running once at any one >> time but >> I would like to be able to show my users where in the queue their >> request >> is. Is there any way to view what async calls are lined up? The >> worker_info stuff doesn't seem to give away any details like this. >> >> I also thought about using "enq_" to schedule but is there a way to >> specify >> that the trigger is the completion of the previous scheduled >> instance and so >> on? Or is the scheduling purely time based? >> >> Kind regards, >> >> Chad. >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org From gethemant at gmail.com Fri Jan 23 20:45:07 2009 From: gethemant at gmail.com (hemant) Date: Sat, 24 Jan 2009 07:15:07 +0530 Subject: [Backgroundrb-devel] Hello to the group In-Reply-To: References: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> Message-ID: On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher wrote: > Thanks Hemant, > > Yes I was specifically trying to avoid threads. However, I am guessing I > could configure it so that only one thread ran at one time right? That > would cut down on the complexity. Yes, changing thread pool size to 1 usually helps. Libraries like mechanize (even Net/HTTP) crap out under concurrent threads but if your pool size is 1, you won't have these problems, thread pool will be just a queue. But if you don't want any concurrency, why not just do something like this: class FooWorker < BackgrounDRb::MetaWorker def create(args = nil) @task_array = [] # check task array for pending tasks and run them add_periodic_timer(2) { run_from_task_array() } end def accept_indexing(args = nil) klass = Struct.new(:cmd_args) # add task to the task array @task_array << klass.new(args) end def run_from_task_array task = @task_array.shift # run the task end def check_count return @task_array.count end end if you don't want threads and above code will work too. Specially since thread pool uses Queue data structure from ruby stdlib which is quite heavyweight and thread safe, but you don't want any thread safety here, so why pay additional penalty for that? All your tasks run in same main thread. > > Is the enq_xxx stuff only time based? If so I wonder if it would be > difficult to add a feature that allows you to add a whole bunch of tasks to > the queue and then they just run one after the other in the order in which > they were added. Maybe I could intercept persistent_job#finished! to find > the next (grouped) task in the queue and update it to start a few seconds in > the future so the bgrb picks it up and runs it etc. But that seems evil > somehow :). You can totally do that! Check add_timer or add_periodic_timer > > I realise this might not be the target usage for backgroundrb but I look > forward to hearing your thoughts because I'm hoping it won't be too > difficult for me to do. The ultimate fall-back, of course, is to roll my > own task queue that uses a table similar table to bdrb_job_queues but that > seems wasteful when there is so much there for it already. > Right. > > On 23 Jan 2009, at 18:57, hemant wrote: > >> That data is in write Queue of master worker. It can't be meaningfully >> extracted to provide how many tasks are in the queue. Put it other >> way, its not even in a Queue in traditional sense. Data is just there >> in a buffer so that master will write it to the socket when socket is >> ready for write. >> >> If you want Queue. You will have either use enq_xxx stuff or provide a >> sugar around asyc_index() method so as actual method is not directly >> called but is added to the thread pool (thread_pool.defer), this way >> you can query how many tasks are there in thread pool queue. >> >> However word "thread" comes with its own meaning and associated >> crapiness in ruby, so beware. >> >> >> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher >> wrote: >>> >>> Hello everyone. >>> >>> I am making several asynchronous calls to one particular method in my >>> worker >>> like this: >>> >>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >>> >>> If I call this several times in a row, all subsequent calls are put in a >>> queue and I would like to be able to keep track of this. The behavior is >>> perfect because I only ever want reindex() running once at any one time >>> but >>> I would like to be able to show my users where in the queue their request >>> is. Is there any way to view what async calls are lined up? The >>> worker_info stuff doesn't seem to give away any details like this. >>> >>> I also thought about using "enq_" to schedule but is there a way to >>> specify >>> that the trigger is the completion of the previous scheduled instance and >>> so >>> on? Or is the scheduling purely time based? >>> >>> Kind regards, >>> >>> Chad. >>> >>> _______________________________________________ >>> Backgroundrb-devel mailing list >>> Backgroundrb-devel at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>> >> >> >> >> -- >> Let them talk of their oriental summer climes of everlasting >> conservatories; give me the privilege of making my own summer with my >> own coals. >> >> http://gnufied.org > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -- Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals. http://gnufied.org From chad.thatcher at gmail.com Sat Jan 24 07:35:05 2009 From: chad.thatcher at gmail.com (Chad Thatcher) Date: Sat, 24 Jan 2009 12:35:05 +0000 Subject: [Backgroundrb-devel] Hello to the group In-Reply-To: References: <7196D609-6D3F-414C-94D7-E4B35EF6CA29@gmail.com> Message-ID: Thanks Hemant, you are absolutely correct about threads being too much overhead for what I need so this looks like a perfect solution! Cheers, Chad. On 24 Jan 2009, at 01:45, hemant wrote: > On Sat, Jan 24, 2009 at 1:46 AM, Chad Thatcher > wrote: >> Thanks Hemant, >> >> Yes I was specifically trying to avoid threads. However, I am >> guessing I >> could configure it so that only one thread ran at one time right? >> That >> would cut down on the complexity. > > Yes, changing thread pool size to 1 usually helps. Libraries like > mechanize (even Net/HTTP) crap out under concurrent threads but if > your pool size is 1, you won't have these problems, thread pool will > be just a queue. But if you don't want any concurrency, why not just > do something like this: > > class FooWorker < BackgrounDRb::MetaWorker > def create(args = nil) > @task_array = [] > # check task array for pending tasks and run them > add_periodic_timer(2) { run_from_task_array() } > end > > def accept_indexing(args = nil) > klass = Struct.new(:cmd_args) > # add task to the task array > @task_array << klass.new(args) > end > > def run_from_task_array > task = @task_array.shift > # run the task > end > > def check_count > return @task_array.count > end > end > > if you don't want threads and above code will work too. Specially > since thread pool uses Queue data structure from ruby stdlib which is > quite heavyweight and thread safe, but you don't want any thread > safety here, so why pay additional penalty for that? All your tasks > run in same main thread. > > > >> >> Is the enq_xxx stuff only time based? If so I wonder if it would be >> difficult to add a feature that allows you to add a whole bunch of >> tasks to >> the queue and then they just run one after the other in the order >> in which >> they were added. Maybe I could intercept persistent_job#finished! >> to find >> the next (grouped) task in the queue and update it to start a few >> seconds in >> the future so the bgrb picks it up and runs it etc. But that seems >> evil >> somehow :). > > You can totally do that! Check add_timer or add_periodic_timer > >> >> I realise this might not be the target usage for backgroundrb but I >> look >> forward to hearing your thoughts because I'm hoping it won't be too >> difficult for me to do. The ultimate fall-back, of course, is to >> roll my >> own task queue that uses a table similar table to bdrb_job_queues >> but that >> seems wasteful when there is so much there for it already. >> > > Right. > >> >> On 23 Jan 2009, at 18:57, hemant wrote: >> >>> That data is in write Queue of master worker. It can't be >>> meaningfully >>> extracted to provide how many tasks are in the queue. Put it other >>> way, its not even in a Queue in traditional sense. Data is just >>> there >>> in a buffer so that master will write it to the socket when socket >>> is >>> ready for write. >>> >>> If you want Queue. You will have either use enq_xxx stuff or >>> provide a >>> sugar around asyc_index() method so as actual method is not directly >>> called but is added to the thread pool (thread_pool.defer), this way >>> you can query how many tasks are there in thread pool queue. >>> >>> However word "thread" comes with its own meaning and associated >>> crapiness in ruby, so beware. >>> >>> >>> On Fri, Jan 23, 2009 at 6:35 PM, Chad Thatcher >> > >>> wrote: >>>> >>>> Hello everyone. >>>> >>>> I am making several asynchronous calls to one particular method >>>> in my >>>> worker >>>> like this: >>>> >>>> MiddleMan.worker(:reindex_worker).async_reindex(:arg => options) >>>> >>>> If I call this several times in a row, all subsequent calls are >>>> put in a >>>> queue and I would like to be able to keep track of this. The >>>> behavior is >>>> perfect because I only ever want reindex() running once at any >>>> one time >>>> but >>>> I would like to be able to show my users where in the queue their >>>> request >>>> is. Is there any way to view what async calls are lined up? The >>>> worker_info stuff doesn't seem to give away any details like this. >>>> >>>> I also thought about using "enq_" to schedule but is there a way to >>>> specify >>>> that the trigger is the completion of the previous scheduled >>>> instance and >>>> so >>>> on? Or is the scheduling purely time based? >>>> >>>> Kind regards, >>>> >>>> Chad. >>>> >>>> _______________________________________________ >>>> Backgroundrb-devel mailing list >>>> Backgroundrb-devel at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >>>> >>> >>> >>> >>> -- >>> Let them talk of their oriental summer climes of everlasting >>> conservatories; give me the privilege of making my own summer with >>> my >>> own coals. >>> >>> http://gnufied.org >> >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel >> > > > > -- > Let them talk of their oriental summer climes of everlasting > conservatories; give me the privilege of making my own summer with my > own coals. > > http://gnufied.org From ramon.tayag at gmail.com Tue Jan 27 04:14:03 2009 From: ramon.tayag at gmail.com (Ramon Tayag) Date: Tue, 27 Jan 2009 17:14:03 +0800 Subject: [Backgroundrb-devel] Doing max of N tasks per given time Message-ID: Hey everyone, PROBLEM I need some backgroundrb help. I have a Rails app that connects to an SMTP server that can only send up to 250 emails per hour. If I try to send the 251st email, it will just ignore it. SENDMAIL? I almost bashed my head trying to setup sendmail so I can send my own emails, but not all emails were being sent. FANCY BDRB STUFF Looking at http://backgroundrb.rubyforge.org/scheduling/ , I can see that there are many things I can do with backgroundrb, but I can't seem to figure out how to do what's written on the subject. I basically want to send up to 250 emails immediately, and queue the 251st email til after the next hour since the max limit of 250 will be reset. Is this possible with backgroundrb? Thanks, Ramon Tayag From chad.thatcher at gmail.com Tue Jan 27 07:12:49 2009 From: chad.thatcher at gmail.com (Chad Thatcher) Date: Tue, 27 Jan 2009 12:12:49 +0000 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: Hi Ramon, I'm also new to BGRB but I had a similar problem only very recently and I think a variation of my solution might help you. Here is my idea for you (thanks mostly to a even guidance of Hemant himself): class MailWorker < BackgrounDRb::MetaWorker set_worker_name :mail_worker def create(args = nil) # this method is called, when worker is loaded for the first time end def create(args = nil) @mail_queue = [] # double check that add_periodic_timer() accepts seconds - if so, 3600 is one hour. add_periodic_timer(3600) { push_mail() } end def add_mail(args = nil) # PS. this method submitted by Hemant, I haven't looked into the Struct stuff. klass = Struct.new(:cmd_args) # add mail to the queue @mail_queue << klass.new(args) end def push_mail sleep 10 # I think this is important to make sure that we have definitely gone over an hour since the last push! # loop through the first 250 !NOTE: must specify a concrete range as it is possible that new mails are being queued while this methods runs. task = @mail_queue.shift # send mail here # end loop end def check_count return @mail_queue.count end end You can even add a "is_busy" flag of some sort to make absolutely sure this doesn't get run twice at the same time. One scenario (if its at all possible) is that there is a huge delay in the sending of mail, meaning the process will lap over the hour. All sorts of bad things could go wrong I suspect. Anyway, thanks primarily to Hemants' help I hope this helps you too. Cheers, Chad. On 27 Jan 2009, at 09:14, Ramon Tayag wrote: > Hey everyone, > > PROBLEM > I need some backgroundrb help. I have a Rails app that connects to an > SMTP server that can only send up to 250 emails per hour. If I try to > send the 251st email, it will just ignore it. > > SENDMAIL? > I almost bashed my head trying to setup sendmail so I can send my own > emails, but not all emails were being sent. > > FANCY BDRB STUFF > Looking at http://backgroundrb.rubyforge.org/scheduling/ , I can see > that there are many things I can do with backgroundrb, but I can't > seem to figure out how to do what's written on the subject. > > I basically want to send up to 250 emails immediately, and queue the > 251st email til after the next hour since the max limit of 250 will be > reset. Is this possible with backgroundrb? > > Thanks, > Ramon Tayag > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel From petesalty at gmail.com Thu Jan 29 21:13:38 2009 From: petesalty at gmail.com (Dale Cook) Date: Thu, 29 Jan 2009 18:13:38 -0800 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> Ramon, What I would do is push all the records that need to be mailed to a table (or flag them in a table they already reside in). Then you can have BDRB set up to run once an hour - it picks up the most recent (or the oldest, or some other combination, your choice) and mails them, 250 each hour until there are none left. You can add in records whenever you want, confident that they will go out eventually. Trying to do it another way, i.e. sending out 250, then setting up a schedule to send out the rest, could be done but it's a lot harder and probably doesn't get you much more, especially for the 251st and onward email recipients, plus you then have to deal with the issue of that happens if you need to send another batch of 250+ within an hour of the first being sent - what to do then? You can read more about BDRB cron scheduling at http://backgroundrb.rubyforge.org/scheduling/ (this isn't one of my strengths unfortunately). Hope this helped some. Dale On Tue, Jan 27, 2009 at 1:14 AM, Ramon Tayag wrote: > Hey everyone, > > PROBLEM > I need some backgroundrb help. I have a Rails app that connects to an > SMTP server that can only send up to 250 emails per hour. If I try to > send the 251st email, it will just ignore it. > > SENDMAIL? > I almost bashed my head trying to setup sendmail so I can send my own > emails, but not all emails were being sent. > > FANCY BDRB STUFF > Looking at http://backgroundrb.rubyforge.org/scheduling/ , I can see > that there are many things I can do with backgroundrb, but I can't > seem to figure out how to do what's written on the subject. > > I basically want to send up to 250 emails immediately, and queue the > 251st email til after the next hour since the max limit of 250 will be > reset. Is this possible with backgroundrb? > > Thanks, > Ramon Tayag > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramon.tayag at gmail.com Thu Jan 29 21:25:38 2009 From: ramon.tayag at gmail.com (Ramon Tayag) Date: Fri, 30 Jan 2009 10:25:38 +0800 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> References: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> Message-ID: Hi Dale! I suppose by table you mean something like ar_mailer? Thanks, Ramon Tayag On Fri, Jan 30, 2009 at 10:13 AM, Dale Cook wrote: > Ramon, > > What I would do is push all the records that need to be mailed to a table > (or flag them in a table they already reside in). Then you can have BDRB set > up to run once an hour - it picks up the most recent (or the oldest, or some > other combination, your choice) and mails them, 250 each hour until there > are none left. You can add in records whenever you want, confident that they > will go out eventually. > Trying to do it another way, i.e. sending out 250, then setting up a > schedule to send out the rest, could be done but it's a lot harder and > probably doesn't get you much more, especially for the 251st and onward > email recipients, plus you then have to deal with the issue of that happens > if you need to send another batch of 250+ within an hour of the first being > sent - what to do then? > > You can read more about BDRB cron scheduling at > http://backgroundrb.rubyforge.org/scheduling/ (this isn't one of my > strengths unfortunately). > > Hope this helped some. > Dale From jonathan.wallace at gmail.com Thu Jan 29 23:39:23 2009 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Thu, 29 Jan 2009 23:39:23 -0500 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> Message-ID: Hi Ramon, I think he means persisting emails to a database table that acts as a queue for your emails. Make the logic in the worker only process 250 elements in your database table. For example, #db table create_table "queued_emails", :force => true do |t| t.string "subject", t.string "body", t.string "to", t.timestamps end #model class QueuedEmail < ActiveRecord::Base end #worker class EmailProcessingWorker < BackgrounDRb::MetaWorker set_worker_name :email_processing_worker def create(args = nil); end def send_emails emails = QueuedEmail.find(:all, :order => "created_on", :limit => 250) emails.each {|e| Notifier.send_email(e) } emails.destroy! end #bdrb config file :backgroundrb: :ip: 0.0.0.0 :port: 1234 :environment: production :schedules: :email_processing_worker: :send_emails: :trigger_args: 0 * */1 * * * You'd need to write the Notifier class which inherits from AR::Mailer and use at your own risk, etc. This code is just a toss off and most definitely contains deficiencies but gets the idea across. Hope this helps, Jonathan On Thu, Jan 29, 2009 at 9:25 PM, Ramon Tayag wrote: > Hi Dale! I suppose by table you mean something like ar_mailer? > > Thanks, > Ramon Tayag > > > > On Fri, Jan 30, 2009 at 10:13 AM, Dale Cook wrote: > > Ramon, > > > > What I would do is push all the records that need to be mailed to a table > > (or flag them in a table they already reside in). Then you can have BDRB > set > > up to run once an hour - it picks up the most recent (or the oldest, or > some > > other combination, your choice) and mails them, 250 each hour until there > > are none left. You can add in records whenever you want, confident that > they > > will go out eventually. > > Trying to do it another way, i.e. sending out 250, then setting up a > > schedule to send out the rest, could be done but it's a lot harder and > > probably doesn't get you much more, especially for the 251st and onward > > email recipients, plus you then have to deal with the issue of that > happens > > if you need to send another batch of 250+ within an hour of the first > being > > sent - what to do then? > > > > You can read more about BDRB cron scheduling at > > http://backgroundrb.rubyforge.org/scheduling/ (this isn't one of my > > strengths unfortunately). > > > > Hope this helped some. > > Dale > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ramon.tayag at gmail.com Fri Jan 30 00:56:43 2009 From: ramon.tayag at gmail.com (Ramon Tayag) Date: Fri, 30 Jan 2009 13:56:43 +0800 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> Message-ID: Ahh, yes I understand more. I might not need to use ARMailer? I'll just save things into the QueuedEmails table, let backgroundrb read that and if it finds something, email it. Thanks! Much clearer now. Ramon Tayag On Fri, Jan 30, 2009 at 12:39 PM, Jonathan Wallace wrote: > Hi Ramon, > > I think he means persisting emails to a database table that acts as a queue > for your emails. Make the logic in the worker only process 250 elements in > your database table. > > For example, > #db table > create_table "queued_emails", :force => true do |t| > t.string "subject", > t.string "body", > t.string "to", > t.timestamps > end > > #model > class QueuedEmail < ActiveRecord::Base > end > > #worker > class EmailProcessingWorker < BackgrounDRb::MetaWorker > set_worker_name :email_processing_worker > def create(args = nil); end > def send_emails > emails = QueuedEmail.find(:all, :order => "created_on", :limit => 250) > emails.each {|e| Notifier.send_email(e) } > emails.destroy! > end > > #bdrb config file > :backgroundrb: > :ip: 0.0.0.0 > :port: 1234 > :environment: production > :schedules: > :email_processing_worker: > :send_emails: > :trigger_args: 0 * */1 * * * > > You'd need to write the Notifier class which inherits from AR::Mailer and > use at your own risk, etc. This code is just a toss off and most definitely > contains deficiencies but gets the idea across. > > Hope this helps, > > Jonathan > > On Thu, Jan 29, 2009 at 9:25 PM, Ramon Tayag wrote: >> >> Hi Dale! I suppose by table you mean something like ar_mailer? >> >> Thanks, >> Ramon Tayag >> >> >> >> On Fri, Jan 30, 2009 at 10:13 AM, Dale Cook wrote: >> > Ramon, >> > >> > What I would do is push all the records that need to be mailed to a >> > table >> > (or flag them in a table they already reside in). Then you can have BDRB >> > set >> > up to run once an hour - it picks up the most recent (or the oldest, or >> > some >> > other combination, your choice) and mails them, 250 each hour until >> > there >> > are none left. You can add in records whenever you want, confident that >> > they >> > will go out eventually. >> > Trying to do it another way, i.e. sending out 250, then setting up a >> > schedule to send out the rest, could be done but it's a lot harder and >> > probably doesn't get you much more, especially for the 251st and onward >> > email recipients, plus you then have to deal with the issue of that >> > happens >> > if you need to send another batch of 250+ within an hour of the first >> > being >> > sent - what to do then? >> > >> > You can read more about BDRB cron scheduling at >> > http://backgroundrb.rubyforge.org/scheduling/ (this isn't one of my >> > strengths unfortunately). >> > >> > Hope this helped some. >> > Dale >> _______________________________________________ >> Backgroundrb-devel mailing list >> Backgroundrb-devel at rubyforge.org >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > From jonathan.wallace at gmail.com Fri Jan 30 01:28:37 2009 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Fri, 30 Jan 2009 01:28:37 -0500 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: <3b4d152e0901291813t5995a608j394a917950025b74@mail.gmail.com> Message-ID: "Notifier" in my code is a reference to ARMailer. You'll still need to use that to send an email message from within rails or backgroundrb. Of course, persisting the jobs to a database table means that you could use something much lighter than backgroundrb (due to it loading a large portion of rails). For example, you could have a plain ruby script that accesses the database and just uses TMail or an equivalent perl script. And then you could use cron to schedule the script. Jonathan On Fri, Jan 30, 2009 at 12:56 AM, Ramon Tayag wrote: > Ahh, yes I understand more. I might not need to use ARMailer? I'll > just save things into the QueuedEmails table, let backgroundrb read > that and if it finds something, email it. > > Thanks! Much clearer now. > Ramon Tayag > > > > On Fri, Jan 30, 2009 at 12:39 PM, Jonathan Wallace > wrote: > > Hi Ramon, > > > > I think he means persisting emails to a database table that acts as a > queue > > for your emails. Make the logic in the worker only process 250 elements > in > > your database table. > > > > For example, > > #db table > > create_table "queued_emails", :force => true do |t| > > t.string "subject", > > t.string "body", > > t.string "to", > > t.timestamps > > end > > > > #model > > class QueuedEmail < ActiveRecord::Base > > end > > > > #worker > > class EmailProcessingWorker < BackgrounDRb::MetaWorker > > set_worker_name :email_processing_worker > > def create(args = nil); end > > def send_emails > > emails = QueuedEmail.find(:all, :order => "created_on", :limit => > 250) > > emails.each {|e| Notifier.send_email(e) } > > emails.destroy! > > end > > > > #bdrb config file > > :backgroundrb: > > :ip: 0.0.0.0 > > :port: 1234 > > :environment: production > > :schedules: > > :email_processing_worker: > > :send_emails: > > :trigger_args: 0 * */1 * * * > > > > You'd need to write the Notifier class which inherits from AR::Mailer and > > use at your own risk, etc. This code is just a toss off and most > definitely > > contains deficiencies but gets the idea across. > > > > Hope this helps, > > > > Jonathan > > > > On Thu, Jan 29, 2009 at 9:25 PM, Ramon Tayag > wrote: > >> > >> Hi Dale! I suppose by table you mean something like ar_mailer? > >> > >> Thanks, > >> Ramon Tayag > >> > >> > >> > >> On Fri, Jan 30, 2009 at 10:13 AM, Dale Cook > wrote: > >> > Ramon, > >> > > >> > What I would do is push all the records that need to be mailed to a > >> > table > >> > (or flag them in a table they already reside in). Then you can have > BDRB > >> > set > >> > up to run once an hour - it picks up the most recent (or the oldest, > or > >> > some > >> > other combination, your choice) and mails them, 250 each hour until > >> > there > >> > are none left. You can add in records whenever you want, confident > that > >> > they > >> > will go out eventually. > >> > Trying to do it another way, i.e. sending out 250, then setting up a > >> > schedule to send out the rest, could be done but it's a lot harder and > >> > probably doesn't get you much more, especially for the 251st and > onward > >> > email recipients, plus you then have to deal with the issue of that > >> > happens > >> > if you need to send another batch of 250+ within an hour of the first > >> > being > >> > sent - what to do then? > >> > > >> > You can read more about BDRB cron scheduling at > >> > http://backgroundrb.rubyforge.org/scheduling/ (this isn't one of my > >> > strengths unfortunately). > >> > > >> > Hope this helped some. > >> > Dale > >> _______________________________________________ > >> Backgroundrb-devel mailing list > >> Backgroundrb-devel at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnutting at gmail.com Fri Jan 30 07:50:51 2009 From: jnutting at gmail.com (Jack Nutting) Date: Fri, 30 Jan 2009 13:50:51 +0100 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: On Tue, Jan 27, 2009 at 10:14 AM, Ramon Tayag wrote: > PROBLEM > I need some backgroundrb help. I have a Rails app that connects to an > SMTP server that can only send up to 250 emails per hour. If I try to > send the 251st email, it will just ignore it. > > SENDMAIL? > I almost bashed my head trying to setup sendmail so I can send my own > emails, but not all emails were being sent. An outbound mail queue seems like a "solved problem", i.e. every piece of mail server software already does this. Adding your own queue before passing it off to your ISP's SMTP seems like a waste of effort on your part, and will lead to delays whenever you need to send more than 250 mails/hour. I think a better solution might be to run your own SMTP server. Sendmail is torturous (I maintained the sendmail installation for a former employer, so I know only too well), but it's not the only choice. Both qmail and postfix have good reputations as sendmail replacements. Take a look at qmail. -- // jack // http://www.nuthole.com From jonathan.wallace at gmail.com Fri Jan 30 09:52:13 2009 From: jonathan.wallace at gmail.com (Jonathan Wallace) Date: Fri, 30 Jan 2009 09:52:13 -0500 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: Jack, That's a great point. In a 5 minute google search, I couldn't find anything about qmail or postfix being able to rate limit out of the box but I did find that exim could. http://www.exim.org/exim-html-4.63/doc/html/spec_html/ch39.html#SECTratelimiting The only question in my mind is, does the database email queue provide persistence that an email server would not? And how important is such things to the OP. I'm not familiar enough with SMTP servers to know what happens during a server reboot what happens to queued mail. Cheers, Jonathan On Fri, Jan 30, 2009 at 7:50 AM, Jack Nutting wrote: > On Tue, Jan 27, 2009 at 10:14 AM, Ramon Tayag > wrote: > > PROBLEM > > I need some backgroundrb help. I have a Rails app that connects to an > > SMTP server that can only send up to 250 emails per hour. If I try to > > send the 251st email, it will just ignore it. > > > > SENDMAIL? > > I almost bashed my head trying to setup sendmail so I can send my own > > emails, but not all emails were being sent. > > An outbound mail queue seems like a "solved problem", i.e. every piece > of mail server software already does this. Adding your own queue > before passing it off to your ISP's SMTP seems like a waste of effort > on your part, and will lead to delays whenever you need to send more > than 250 mails/hour. > > I think a better solution might be to run your own SMTP server. > Sendmail is torturous (I maintained the sendmail installation for a > former employer, so I know only too well), but it's not the only > choice. Both qmail and postfix have good reputations as sendmail > replacements. Take a look at qmail. > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chad.thatcher at gmail.com Fri Jan 30 16:14:30 2009 From: chad.thatcher at gmail.com (Chad Thatcher) Date: Fri, 30 Jan 2009 21:14:30 +0000 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: <7799B961-FF2A-409F-B2D7-B6EF6B0CD2DF@gmail.com> Exim is an excellent choice imho. I've used it for years and its solid and very simple to configure (well, by comparison to sendmail anyway). In fact it was based originally on sendmail source code, if I remember correctly, and one of the main goals of the Exim creator(s) was to eliminate the awful sendmail configuration. Once you have passed the message to exim it enters the mail queue and that has been impervious to sudden server crashes/reboots etc. since the dawn of email :). In any event, troubleshooting/diagnosing/ queuing mail with your own homegrown solution is reinventing the wheel as someone already said. A homegrown solution isn't going to be any more resilient then the tried an tested mailq. IMHO the best way to ensure that the mail got there is to use receipts. Anyway thats my 2c. :) Chad. On 30 Jan 2009, at 14:52, Jonathan Wallace wrote: > Jack, > > That's a great point. In a 5 minute google search, I couldn't find > anything about qmail or postfix being able to rate limit out of the > box but I did find that exim could. http://www.exim.org/exim-html-4.63/doc/html/spec_html/ch39.html#SECTratelimiting > > The only question in my mind is, does the database email queue > provide persistence that an email server would not? And how > important is such things to the OP. I'm not familiar enough with > SMTP servers to know what happens during a server reboot what > happens to queued mail. > > Cheers, > Jonathan > > On Fri, Jan 30, 2009 at 7:50 AM, Jack Nutting > wrote: > On Tue, Jan 27, 2009 at 10:14 AM, Ramon Tayag > wrote: > > PROBLEM > > I need some backgroundrb help. I have a Rails app that connects > to an > > SMTP server that can only send up to 250 emails per hour. If I > try to > > send the 251st email, it will just ignore it. > > > > SENDMAIL? > > I almost bashed my head trying to setup sendmail so I can send my > own > > emails, but not all emails were being sent. > > An outbound mail queue seems like a "solved problem", i.e. every piece > of mail server software already does this. Adding your own queue > before passing it off to your ISP's SMTP seems like a waste of effort > on your part, and will lead to delays whenever you need to send more > than 250 mails/hour. > > I think a better solution might be to run your own SMTP server. > Sendmail is torturous (I maintained the sendmail installation for a > former employer, so I know only too well), but it's not the only > choice. Both qmail and postfix have good reputations as sendmail > replacements. Take a look at qmail. > > -- > // jack > // http://www.nuthole.com > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel > > _______________________________________________ > Backgroundrb-devel mailing list > Backgroundrb-devel at rubyforge.org > http://rubyforge.org/mailman/listinfo/backgroundrb-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnutting at gmail.com Fri Jan 30 18:26:55 2009 From: jnutting at gmail.com (Jack Nutting) Date: Sat, 31 Jan 2009 00:26:55 +0100 Subject: [Backgroundrb-devel] Doing max of N tasks per given time In-Reply-To: References: Message-ID: On Fri, Jan 30, 2009 at 3:52 PM, Jonathan Wallace wrote: > That's a great point. In a 5 minute google search, I couldn't find anything > about qmail or postfix being able to rate limit out of the box but I did > find that exim could. > http://www.exim.org/exim-html-4.63/doc/html/spec_html/ch39.html#SECTratelimiting Actually, one of the benefits of the running his own mail server is that he should be able to skip his ISP's mail server entirely, thereby eliminating the rate limit entirely (unless his ISP is blocking outbound SMTP traffic, which is possible). -- // jack // http://www.nuthole.com