From dwlist at stellarmadness.org Mon Aug 11 22:28:11 2008 From: dwlist at stellarmadness.org (Darin Wilson) Date: Mon, 11 Aug 2008 19:28:11 -0700 (PDT) Subject: [Activewarehouse-discuss] Patch: use vendor/rails feature with aw-etl In-Reply-To: <1264E761-3B2E-4100-8D12-44D4C5821717@gmail.com> References: <48830D28.4070008@vdomck.org> <1264E761-3B2E-4100-8D12-44D4C5821717@gmail.com> Message-ID: <51337.67.188.223.148.1218508091.squirrel@stellarmadness.org> Hi- Below is a patch I used for a project I'm working that I thought might be useful to others. Our situation is that we have an aw-etl project running on a machine (with some other services) that needs to be upgraded to Rails 2.1. Since aw-etl does not work with Rails 2.1 yet, I thought I would use the handy vendor/rails feature in Rails that allows you to freeze a specific version of Rails with your project. The trouble is that the --rails-root option of aw-etl prevents this from working properly because the option is not processed until after the "require rubygems" statement is executed - this locks you into using whatever version is installed at the system level. The hack below looks for the --rails-root option at the top of the execution process, and imports the Rails environment at that point, rather than at engine initialization time. I couldn't figure out a reasonable way of including a test with this patch, but it is definitely working for us. Perhaps if someone else needs this, they can try it out and confirm that it works? Thanks! Darin Wilson (patch was created from the etl/trunk directory of the activewarehouse source) Index: lib/etl/engine.rb =================================================================== --- lib/etl/engine.rb (revision 908) +++ lib/etl/engine.rb (working copy) @@ -25,7 +25,8 @@ @log_write_mode = 'w' if options[:newlog] @skip_bulk_import = options[:skip_bulk_import] @read_locally = options[:read_locally] - @rails_root = options[:rails_root] + # the --rails_root option is handled in lib/etl.rb, before any other require statements + @rails_root = options[:rails_root] require File.join(@rails_root, 'config/environment') if @rails_root Index: lib/etl.rb =================================================================== --- lib/etl.rb (revision 908) +++ lib/etl.rb (working copy) @@ -29,6 +29,17 @@ require 'yaml' require 'erb' +# we don't want to parse all of the args just yet, but we want to +# initialize the Rails environment if the --rails-root option was +# supplied - this will let us use the Rails version in vendor/rails +# if it's there +ARGV.each_with_index do |arg, i| + if arg == '--rails-root' && (i + 1) < ARGV.length + require File.join(ARGV[i + 1], 'config/environment') + break + end +end + require 'rubygems' unless Kernel.respond_to?(:gem) From brad.cantin at gmail.com Thu Aug 14 10:47:49 2008 From: brad.cantin at gmail.com (Brad Cantin) Date: Thu, 14 Aug 2008 10:47:49 -0400 Subject: [Activewarehouse-discuss] Hey there Message-ID: <7140a9950808140747k68261003xfbb9d8d85e5e748@mail.gmail.com> I have been looking into building a data warehouse for a project and I found the activewarehouse gems. I was looking for a tutorial and the link and the blog for http://anthonyeden.com seem to be down. Is there another tutorial around ? Thanks for any help and I hope to be able to contribute to this project once I get up to speed. ~Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From thibaut.barrere at gmail.com Thu Aug 14 11:00:50 2008 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 14 Aug 2008 17:00:50 +0200 Subject: [Activewarehouse-discuss] Hey there In-Reply-To: <4a68b8cf0808140800h5c6db973ob2ef7c0728e0ec69@mail.gmail.com> References: <7140a9950808140747k68261003xfbb9d8d85e5e748@mail.gmail.com> <4a68b8cf0808140800h5c6db973ob2ef7c0728e0ec69@mail.gmail.com> Message-ID: <4a68b8cf0808140800s4567603an7e9ee47ba0989760@mail.gmail.com> Hi Brad, Unfortunately I'm not sure this tutorial is still available somewhere - anyone else ? If you need the ETL part, there is a pretty good user manual here: http://activewarehouse.rubyforge.org/docs/activewarehouse-etl.html Otherwise be sure to check out http://activewarehouse.rubyforge.org/rdoc/ hope this helps! Thibaut Barr?re -- LoGeek [blog] http://evolvingworker.com - tools for a better day [blog] http://blog.logeek.fr - about writing software From mghaught at gmail.com Thu Aug 14 11:11:52 2008 From: mghaught at gmail.com (Marty Haught) Date: Thu, 14 Aug 2008 09:11:52 -0600 Subject: [Activewarehouse-discuss] Hey there In-Reply-To: <4a68b8cf0808140800s4567603an7e9ee47ba0989760@mail.gmail.com> References: <7140a9950808140747k68261003xfbb9d8d85e5e748@mail.gmail.com> <4a68b8cf0808140800h5c6db973ob2ef7c0728e0ec69@mail.gmail.com> <4a68b8cf0808140800s4567603an7e9ee47ba0989760@mail.gmail.com> Message-ID: <57f29e620808140811y2764eaa8i3003fb3cfb4a4d34@mail.gmail.com> Bob Cotton made a github repo for the demo. Not sure if that helps any of you but it's worth looking at. I also posted a pdf of my slides of an ActiveWarehouse presentation I did earlier in the year. Again, there may be some useful bits in there too. http://github.com/bcotton/activewarehouse-demo/tree/master http://martyhaught.com/articles/2008/03/20/active-warehouse-slides/ Cheers, Marty On Thu, Aug 14, 2008 at 9:00 AM, Thibaut Barr?re wrote: > Hi Brad, > > Unfortunately I'm not sure this tutorial is still available somewhere > - anyone else ? > > If you need the ETL part, there is a pretty good user manual here: > http://activewarehouse.rubyforge.org/docs/activewarehouse-etl.html > > Otherwise be sure to check out http://activewarehouse.rubyforge.org/rdoc/ > > hope this helps! > > Thibaut Barr?re > -- > LoGeek > [blog] http://evolvingworker.com - tools for a better day > [blog] http://blog.logeek.fr - about writing software > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From dwlist at stellarmadness.org Thu Aug 14 13:39:49 2008 From: dwlist at stellarmadness.org (Darin Wilson) Date: Thu, 14 Aug 2008 10:39:49 -0700 Subject: [Activewarehouse-discuss] Hey there In-Reply-To: <4a68b8cf0808140800s4567603an7e9ee47ba0989760@mail.gmail.com> Message-ID: I was able to dig it up at the internet archive: http://web.archive.org/web/20070103003836/anthonyeden.com/2006/12/20/activew arehouse-example-with-rails-svn-logs Some parts of the docs are out of date. I found I sort of had to triangulate between the tutorial, the user's guide and the source itself to get things completely up and running, but it's working really well for us right now. Good luck! Darin > From: Thibaut Barr?re > Date: Thu, 14 Aug 2008 17:00:50 +0200 > To: > Subject: Re: [Activewarehouse-discuss] Hey there > > Hi Brad, > > Unfortunately I'm not sure this tutorial is still available somewhere > - anyone else ? > > If you need the ETL part, there is a pretty good user manual here: > http://activewarehouse.rubyforge.org/docs/activewarehouse-etl.html > > Otherwise be sure to check out http://activewarehouse.rubyforge.org/rdoc/ > > hope this helps! > > Thibaut Barr?re > -- > LoGeek > [blog] http://evolvingworker.com - tools for a better day > [blog] http://blog.logeek.fr - about writing software > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss From jmaine at blurb.com Thu Aug 14 14:38:20 2008 From: jmaine at blurb.com (Jacob Maine) Date: Thu, 14 Aug 2008 11:38:20 -0700 (PDT) Subject: [Activewarehouse-discuss] ActiveWarehouse patches Message-ID: <31177298.199351218739100960.JavaMail.root@kilgore.blurb.com> Hi all, Over the last few months, I've done some work on activewarehouse and activewarehouse-etl. You can review the changes or get the code at http://github.com/mainej. My main focus was on slowly changing dimensions, although I ended up patching lots of other things. I've extended the SCD specs and fixed a number of bugs, including some pointed out on this list by Martin Honermeyer. Anthony's github repository has most of the same updates, although I haven't asked him to pull from me recently. If you plan on using the activewarehouse patches, you'll also need the patches in adapter_extensions. We've been running a nightly incremental load for about a month, with type 1 and 2 slowly changing dimensions, and one accumulating snapshot fact. The ETL seems to be going smoothly. If you end up using my patches and notice problems, let me know and I'll see if I can help. It would be great if other people could contribute to AW via github. It's a convenient way to share patches. Darin Wilson - you mentioned a patch you put together recently - would you be willing to post it on github? Best, Jacob Maine From dwlist at stellarmadness.org Fri Aug 15 01:24:46 2008 From: dwlist at stellarmadness.org (Darin Wilson) Date: Thu, 14 Aug 2008 22:24:46 -0700 Subject: [Activewarehouse-discuss] ActiveWarehouse patches In-Reply-To: <31177298.199351218739100960.JavaMail.root@kilgore.blurb.com> Message-ID: Our company is still pretty deeply entrenched in svn, so I haven't ventured too far into the world of git just yet. However, this may provide me with the kick in the pants that I've needed. :) I'll check out github and start adding patches there. Thanks! Darin > From: Jacob Maine > Date: Thu, 14 Aug 2008 11:38:20 -0700 (PDT) > To: > Subject: [Activewarehouse-discuss] ActiveWarehouse patches > > Hi all, > > Over the last few months, I've done some work on activewarehouse and > activewarehouse-etl. You can review the changes or get the code at > http://github.com/mainej. > > My main focus was on slowly changing dimensions, although I ended up patching > lots of other things. I've extended the SCD specs and fixed a number of bugs, > including some pointed out on this list by Martin Honermeyer. > > Anthony's github repository has most of the same updates, although I haven't > asked him to pull from me recently. If you plan on using the activewarehouse > patches, you'll also need the patches in adapter_extensions. > > We've been running a nightly incremental load for about a month, with type 1 > and 2 slowly changing dimensions, and one accumulating snapshot fact. The > ETL seems to be going smoothly. If you end up using my patches and notice > problems, let me know and I'll see if I can help. > > It would be great if other people could contribute to AW via github. It's a > convenient way to share patches. Darin Wilson - you mentioned a patch you put > together recently - would you be willing to post it on github? > > Best, > Jacob Maine > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss