From anthonyeden at gmail.com Wed Aug 1 10:30:21 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 1 Aug 2007 10:30:21 -0400 Subject: [Activewarehouse-discuss] Error with demo app In-Reply-To: References: Message-ID: Did you ever find a solution to this issue? V/r Anthony On 6/22/07, Alex Wolfe wrote: > I'm gettng the same error as Andrew O'Brien: > > > Using AdapterExtensions > ./db/../config/../vendor/plugins/activewarehouse/lib/active_warehouse/aggregate_field.rb:17:in > `initialize': undefined method `scale' for > # > (NoMethodError) > from > ./db/../config/../vendor/plugins/activewarehouse/lib/active_warehouse/fact.rb:123:in > > > I'm on Rails 1.2.3 (ActiveRecord 1.15.3), Mac OS X with MySQL 5.0.21. > > In ActiveRecord 1.15.3 scale (and precision) are defined on > ActiveRecord::ConnectionAdapters::Column, which is extended > by MysqlColumn. So I don't understand the NoMethodError. Anyone have any > bright ideas? > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From steve.eichert at gmail.com Wed Aug 1 13:37:51 2007 From: steve.eichert at gmail.com (Steve Eichert) Date: Wed, 1 Aug 2007 13:37:51 -0400 Subject: [Activewarehouse-discuss] Demo of activewarehouse front end? Message-ID: Is there a demo site up anywhere that has reports and such created with activewarehouse? I'm planning on installing it locally but, was wondering if anyone has anything up that's publicly accessible for me to checkout, as well as to send to some others for their review. Thanks, Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070801/7f615f9f/attachment.html From chris.d.williams at gmail.com Sat Aug 4 11:44:08 2007 From: chris.d.williams at gmail.com (Chris Williams) Date: Sat, 4 Aug 2007 11:44:08 -0400 Subject: [Activewarehouse-discuss] Having problems with demo app... Message-ID: I retrieved the demo app from SVN and having some issues with loading the fact table. Below is the error I am seeing in the etl.log file. I thought I had downloaded the latest software from SVN. Any suggestions on how to correct this would be great. I checked the author_dimension table and aeden is in there. Thanks CW Error transforming from input/aw_log.xml on line 1: Unable to resolve aeden to foreign key for author_id in row 1 c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/transform/foreign_key_lookup_transform.rb:30:in `transform' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:233:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:231:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:231:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:230:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:230:in `process' c:/ruby/lib/ruby/1.8/benchmark.rb:293:in `measure' c:/ruby/lib/ruby/1.8/benchmark.rb:307:in `realtime' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:226:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1309:in `each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/control/source/file_source.rb:49:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/parser/sax_parser.rb:118:in `call' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/parser/sax_parser.rb:118:in `end_element' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:138:in `parse' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:137:in `each' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:137:in `parse' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/parser/sax_parser.rb:28:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/parser/sax_parser.rb:24:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/control/source/file_source.rb:44:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:187:in `each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:187:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:181:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:181:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/engine.rb:37:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/commands/etl.rb:73 c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/commands/etl.rb:70:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/../lib/etl/commands/etl.rb:70 c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.8.4/bin/etl:28 c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/bin/etl:18 From anthonyeden at gmail.com Sat Aug 4 14:27:52 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Sat, 4 Aug 2007 14:27:52 -0400 Subject: [Activewarehouse-discuss] Having problems with demo app... In-Reply-To: References: Message-ID: On 8/4/07, Chris Williams wrote: > I retrieved the demo app from SVN and having some issues with loading > the fact table. Below is the error I am seeing in the etl.log file. > I thought I had downloaded the latest software from SVN. Any > suggestions on how to correct this would be great. I checked the > author_dimension table and aeden is in there. Chris, I've updated the rails_warehouse demo code to use AW ETL 0.9, and I can now run it. In order to get AW ETL 0.9 though you'll have to check out the etl/trunk directory and then run rake reinstall. If you install AW ETL 0.9 and the updated code from rails_warehouse/trunk and it still won't run, then please let me know. V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From ysriram at strongmail.com Tue Aug 14 18:27:09 2007 From: ysriram at strongmail.com (Yeshwanth Sriram) Date: Tue, 14 Aug 2007 15:27:09 -0700 Subject: [Activewarehouse-discuss] Get rows out of the query results Message-ID: How to get rows out of the query results. Need to construct a chart. Are there any users who have used activewarehouse for charting purposes. I'm using activewarehouse from trunk. Y Yeshwanth Sriram PS Engineer (650) 421-7111 Direct (650) 421-4201 Fax Unlock the Business Potential of Digital Messaging www.strongmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070814/a3e4e111/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 2640 bytes Desc: image001.gif Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070814/a3e4e111/attachment.gif From thibaut.barrere at gmail.com Wed Aug 15 03:57:09 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Wed, 15 Aug 2007 09:57:09 +0200 Subject: [Activewarehouse-discuss] Anyone freezing its AW-ETL gems ? Message-ID: <4a68b8cf0708150057m35e24549sdb97584ac941e5f3@mail.gmail.com> Hi! Just curious, and willing to simplify my deployments as I just started a new ETL consulting mission which I'll carry out with AW-ETL. Is there anyone freezing all the gems required for AW-ETL to work, in order to have "svn checkout" deployment and nothing to gem install ? btw: any estimate of the release date for AW-ETL 0.9 ? Don't want to hurry or anything - just to help me choose between going trunk or waiting for the release. cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070815/ff71e9e1/attachment.html From anthonyeden at gmail.com Wed Aug 15 07:03:02 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 15 Aug 2007 07:03:02 -0400 Subject: [Activewarehouse-discuss] Anyone freezing its AW-ETL gems ? In-Reply-To: <4a68b8cf0708150057m35e24549sdb97584ac941e5f3@mail.gmail.com> References: <4a68b8cf0708150057m35e24549sdb97584ac941e5f3@mail.gmail.com> Message-ID: On 8/15/07, Thibaut Barr?re wrote: > Hi! > > Just curious, and willing to simplify my deployments as I just started a new > ETL consulting mission which I'll carry out with AW-ETL. Cool. > Is there anyone freezing all the gems required for AW-ETL to work, in order > to have "svn checkout" deployment and nothing to gem install ? I do it on one of my projects. The main difference between freezing and not freezing is that you have to use the bin/etl script in the frozen version directly. > btw: any estimate of the release date for AW-ETL 0.9 ? Don't want to hurry > or anything - just to help me choose between going trunk or waiting for the > release. AW ETL 0.9 is already out, I just didn't do an announcement on it. :-) V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Wed Aug 15 08:29:38 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Wed, 15 Aug 2007 14:29:38 +0200 Subject: [Activewarehouse-discuss] Anyone freezing its AW-ETL gems ? In-Reply-To: References: <4a68b8cf0708150057m35e24549sdb97584ac941e5f3@mail.gmail.com> Message-ID: <4a68b8cf0708150529l446a2336r66187b2d1441ee16@mail.gmail.com> Hi, > > I do it on one of my projects. The main difference between freezing > and not freezing is that you have to use the bin/etl script in the > frozen version directly. Is there anything else required apart from unpacking the gem ? (ie: do you add a LOAD_PATH somewhere or some tweaking to the Rails environment ?) > btw: any estimate of the release date for AW-ETL 0.9 ? Don't want to hurry > > or anything - just to help me choose between going trunk or waiting for > the > > release. > > AW ETL 0.9 is already out, I just didn't do an announcement on it. :-) Yup - just noticed as I gem installed on a cruisecontrol server! Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070815/d30d32f0/attachment.html From chris.d.williams at gmail.com Thu Aug 16 20:06:45 2007 From: chris.d.williams at gmail.com (Chris Williams) Date: Thu, 16 Aug 2007 20:06:45 -0400 Subject: [Activewarehouse-discuss] Another demo question... Message-ID: I am trying the demo again out of svn now that etl 0.9 is out. I am still getting the following error. Any other suggestions? I am running on windows if that matters.. Thanks CW Error transforming from input/aw_log.xml on line 1: Unable to resolve aeden to foreign key for author_id in row 1 c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/transform/foreign_key_lookup_transform.rb:30:in `transform' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:357:in `process_control' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:355:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:355:in `process_control' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:354:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:354:in `process_control' c:/ruby/lib/ruby/1.8/benchmark.rb:293:in `measure' c:/ruby/lib/ruby/1.8/benchmark.rb:307:in `realtime' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:351:in `process_control' c:/ruby/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:1309:in `each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/source/file_source.rb:51:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/parser/sax_parser.rb:118:in `call' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/parser/sax_parser.rb:118:in `end_element' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:138:in `parse' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:137:in `each' c:/ruby/lib/ruby/1.8/rexml/parsers/sax2parser.rb:137:in `parse' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/parser/sax_parser.rb:28:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/parser/sax_parser.rb:24:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/source/file_source.rb:45:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:314:in `each_with_index' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:314:in `process_control' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:308:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:308:in `process_control' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:261:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:259:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:55:in `process' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:83:in `execute' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:80:in `each' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:80:in `execute' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:90 c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:488:in `load' c:/ruby/bin/etl:18 From thibaut.barrere at gmail.com Fri Aug 17 10:12:51 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 17 Aug 2007 16:12:51 +0200 Subject: [Activewarehouse-discuss] etl binary does not return an error when a fatal screen doesn't pass - expected behaviour ? Message-ID: <4a68b8cf0708170712r39df2d55re033447d314e17b1@mail.gmail.com> Hello ETLers, I've just tried the screen feature on a dimension import, and first I want to say it's a great feature ! (having used home-baked approaches with SSIS before) Here's my first use-case, very basic but very useful to me: screen(:fatal) { assert_equal 300, ETL::Engine.connection(CONFIG).select_value("SELECT COUNT(*) FROM date_dimension").to_i } (I've noticed the RowCounterScreen also, will use it later) As i've set up a continuous integration server for this datawarehouse, I've also noticed that the "etl" process will not return an error code if the screen doesn't pass: eg: calling result = system("etl fill_datedimension.ctl") will return true. Is it the expected behaviour ? Wouldn't it be practical to return an error code (or just non-zero) so that the caller know there was a fatal issue ? cheers ps: sidenote, but what do you think about moving to a google group for the mailing list hosting ? (I love the post monitor option, for instance) -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/d322fe5d/attachment.html From thibaut.barrere at gmail.com Fri Aug 17 10:44:20 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 17 Aug 2007 16:44:20 +0200 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) Message-ID: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> Hi, as my database is using latin1 encoding, but I want to keep my code in utf-8, and use a lot of accents in my dimensions (janvier, f?vrier, etc for months in french), I wrote the attached encode processor. I'm using it this way: post_process :encode, { :source_file => 'date_dimension.txt', :source_encoding => 'utf-8', :target_file => 'date_dimension_latin1.txt', :target_encoding => 'latin1'} Very basic and not tested on big loads: it reads the source line by line and calls iconv on it (using the ruby library, not a system call, to be windows compatible - although the system call will definitely be faster...). feel free to use it or include it in the core, if that's its destiny. cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/cbd4d59a/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: encode_processor.rb Type: application/octet-stream Size: 2041 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/cbd4d59a/attachment.obj From anthonyeden at gmail.com Fri Aug 17 10:50:21 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 17 Aug 2007 10:50:21 -0400 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> Message-ID: Thibaut, Any chance you can provide a unit test for it? I'd like to include it. V/r Anthony On 8/17/07, Thibaut Barr?re wrote: > Hi, > > as my database is using latin1 encoding, but I want to keep my code in > utf-8, and use a lot of accents in my dimensions (janvier, f?vrier, etc for > months in french), I wrote the attached encode processor. I'm using it this > way: > > post_process :encode, { :source_file => 'date_dimension.txt', > :source_encoding => 'utf-8', :target_file => 'date_dimension_latin1.txt', > :target_encoding => 'latin1'} > > Very basic and not tested on big loads: it reads the source line by line and > calls iconv on it (using the ruby library, not a system call, to be windows > compatible - although the system call will definitely be faster...). > > feel free to use it or include it in the core, if that's its destiny. > > cheers > > Thibaut > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Fri Aug 17 10:59:28 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 17 Aug 2007 16:59:28 +0200 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> Message-ID: <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> > Any chance you can provide a unit test for it? I'd like to include it. Yes, absolutely - I'm going to have a further look at AW-ETL trunk to see how to package this properly (Rspec with mock ?? or Test::unit ?) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/9fef6584/attachment.html From anthonyeden at gmail.com Fri Aug 17 11:20:43 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 17 Aug 2007 11:20:43 -0400 Subject: [Activewarehouse-discuss] etl binary does not return an error when a fatal screen doesn't pass - expected behaviour ? In-Reply-To: <4a68b8cf0708170712r39df2d55re033447d314e17b1@mail.gmail.com> References: <4a68b8cf0708170712r39df2d55re033447d314e17b1@mail.gmail.com> Message-ID: On 8/17/07, Thibaut Barr?re wrote: > Hello ETLers, > > I've just tried the screen feature on a dimension import, and first I want > to say it's a great feature ! (having used home-baked approaches with SSIS > before) Here's my first use-case, very basic but very useful to me: > > screen(:fatal) { > assert_equal 300, ETL::Engine.connection(CONFIG).select_value("SELECT > COUNT(*) FROM date_dimension").to_i > } > > (I've noticed the RowCounterScreen also, will use it later) Good idea. Hopefully I'll have the screens in a little bit of a better state by next release. > As i've set up a continuous integration server for this datawarehouse, I've > also noticed that the "etl" process will not return an error code if the > screen doesn't pass: > > eg: calling result = system("etl fill_datedimension.ctl") will return true. > > Is it the expected behaviour ? Wouldn't it be practical to return an error > code (or just non-zero) so that the caller know there was a fatal issue ? I actually hadn't considered returning an error code, but given your use case I think it's a reasonable expectation. I'll see what I can do. > ps: sidenote, but what do you think about moving to a google group for the > mailing list hosting ? (I love the post monitor option, for instance) Yes, I wouldn't mind doing that. I've set up the list, I just need to announce it. V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Fri Aug 17 15:51:09 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 17 Aug 2007 21:51:09 +0200 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> Message-ID: <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> Here's a more complete release, with tests and documentation patch. Notes: - encode_source_latin1.txt should go under tests/data. - the doc will probably need to be refactored a bit, as this processor could be used as both a pre/post-processor... cheers, -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/668e7b9d/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: encode_processor.rb Type: application/octet-stream Size: 2339 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/668e7b9d/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: encode_processor_test.rb Type: application/octet-stream Size: 1590 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/668e7b9d/attachment-0001.obj -------------- next part -------------- ?ph?m?re has accents. let's encode them. -------------- next part -------------- A non-text attachment was scrubbed... Name: encode_processor_documentation.diff Type: application/octet-stream Size: 988 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070817/668e7b9d/attachment-0002.obj From anthonyeden at gmail.com Fri Aug 17 15:57:48 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 17 Aug 2007 15:57:48 -0400 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> Message-ID: Excellent, I'll apply this weekend. V/r Anthony On 8/17/07, Thibaut Barr?re wrote: > Here's a more complete release, with tests and documentation patch. Notes: > - encode_source_latin1.txt should go under tests/data. > - the doc will probably need to be refactored a bit, as this processor could > be used as both a pre/post-processor... > > cheers, > > -- Thibaut > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From chris.d.williams at gmail.com Sat Aug 18 16:56:44 2007 From: chris.d.williams at gmail.com (Chris Williams) Date: Sat, 18 Aug 2007 16:56:44 -0400 Subject: [Activewarehouse-discuss] Figured out the issues I was having with the demo app Message-ID: For some reason, certain string values in the dimension tables had an extra control character at the end of the value. This was causing the foreign key lookup to fail. I don't know if I was getting these extra control characters because I am on windows or maybe a MySQL setup issue. I used MySQL - Front to view the tables and had to remove the characters by hand. Since it was only a few columns, it was no big deal. Here are the columns I saw that had the extra characters. I only changed the author and change_type since it was effecting the data load. author_dimension - name change_type_dimension - change_type_description file_dimension - framework I plan on looking at this problem some more to see if I can figure out why this is happening. Thanks CW From anthonyeden at gmail.com Sat Aug 18 16:58:45 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Sat, 18 Aug 2007 16:58:45 -0400 Subject: [Activewarehouse-discuss] Figured out the issues I was having with the demo app In-Reply-To: References: Message-ID: On 8/18/07, Chris Williams wrote: > For some reason, certain string values in the dimension tables had an > extra control character at the end of the value. This was causing the > foreign key lookup to fail. I don't know if I was getting these extra > control characters because I am on windows or maybe a MySQL setup > issue. I used MySQL - Front to view the tables and had to remove the > characters by hand. Since it was only a few columns, it was no big > deal. Here are the columns I saw that had the extra characters. I > only changed the author and change_type since it was effecting the > data load. > > author_dimension > - name > > change_type_dimension > - change_type_description > > file_dimension > - framework > > I plan on looking at this problem some more to see if I can figure out > why this is happening. Glad to hear you're making progress. I'm interested to see what the ultimate cause was. V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From chris.d.williams at gmail.com Sat Aug 18 18:12:01 2007 From: chris.d.williams at gmail.com (Chris Williams) Date: Sat, 18 Aug 2007 18:12:01 -0400 Subject: [Activewarehouse-discuss] Figured out the issues I was having with the demo app In-Reply-To: References: Message-ID: It looks like it is adding the extra control character on items that are strings and are the last item on a line in the csv files. I am assuming this is a windows/unix. UNIX does just a LF while windows does CR+LF. I am poking around delimited_parser.rb since I think a modification is needed there. Time to learn how to debug ruby... Thanks CW On 8/18/07, Anthony Eden wrote: > On 8/18/07, Chris Williams wrote: > > For some reason, certain string values in the dimension tables had an > > extra control character at the end of the value. This was causing the > > foreign key lookup to fail. I don't know if I was getting these extra > > control characters because I am on windows or maybe a MySQL setup > > issue. I used MySQL - Front to view the tables and had to remove the > > characters by hand. Since it was only a few columns, it was no big > > deal. Here are the columns I saw that had the extra characters. I > > only changed the author and change_type since it was effecting the > > data load. > > > > author_dimension > > - name > > > > change_type_dimension > > - change_type_description > > > > file_dimension > > - framework > > > > I plan on looking at this problem some more to see if I can figure out > > why this is happening. > > Glad to hear you're making progress. I'm interested to see what the > ultimate cause was. > > V/r > Anthony > > -- > Cell: 808 782-5046 > Current Location: Melbourne, FL > From chris.d.williams at gmail.com Sun Aug 19 15:45:44 2007 From: chris.d.williams at gmail.com (Chris Williams) Date: Sun, 19 Aug 2007 15:45:44 -0400 Subject: [Activewarehouse-discuss] Figured out the issues I was having with the demo app In-Reply-To: References: Message-ID: Anthony, here are some patches I came up with for the activewarehouse-etl and adapter_extensions. With these patches and adding the :line_separator => '\r\n' to the bulk_import portion of the etl scripts, it fixed the problem on windows with mysql. From what I could tell, Postgres doesn't have a similar line_separator option and I made an attempt at a fix for sqlserver. I am still working on creating some real test cases to pass on. Either way, he is a description of the changes. activewarehouse-etl/lib/etl/processor/bulk_import_processor.rb I created a new lines item in the hash. I did this since the that option is line based not field based. adapter_extensions/lib/adapter_extensions/connection_adapters/mysql_adapter.rb Added the additional commands to the MySQL copy command to define the line_separator. adapter_extensions/lib/adapter_extensions/connection_adapters/sqlserver_adapter.rb I added a -r to the bcp command to define the line_separator (not tested) author_dimension.txt Here is an example of file that has the \r\n for the end of line. Let me know if you have any questions. Thanks CW On 8/19/07, Anthony Eden wrote: > On 8/19/07, Chris Williams wrote: > > I think I have narrowed it down to the bulk importer but I have hit a > > wall. I dumped the authors table from my MySQL database and the extra > > character is \r. I tried changing the author_dimension.ctl to include > > the :line_separator => '\r\n' to the post_process item but that didn't > > help. Where is the conn.bulk_load defined? Is that outside the ETL > > code base? I searched the code and didn't see bulk_load referenced. > > It's defined in the adapter_extensions library, which is another > library under the ActiveWarehouse umbrella project. > > V/r > Anthony > > -- > Cell: 808 782-5046 > Current Location: Melbourne, FL > -------------- next part -------------- A non-text attachment was scrubbed... Name: activewarehouse-etl.patch Type: application/octet-stream Size: 677 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070819/dcbf6af6/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: adapter_extensions.patch Type: application/octet-stream Size: 2113 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070819/dcbf6af6/attachment-0003.obj -------------- next part -------------- 1,aeden 2,sethladd 3,wenyizhou 4,stouset From thibaut.barrere at gmail.com Mon Aug 20 15:44:40 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Mon, 20 Aug 2007 21:44:40 +0200 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? Message-ID: <4a68b8cf0708201244ne7e9fdelfdf4065abd974a5a@mail.gmail.com> Hi I've noticed a very strange thing when deploying my first .ebf/.ctl to a new machine. When launching etl datawarehouse.ebf, I just get: ================ About to launch the ETL process... Running etl fill_datawarehouse.ebf Using AdapterExtensions Starting ETL process initializing ETL engine Processing fill_datawarehouse.ebf Executing batch Build finished. ================ When launching the .ctl files separately, I also get the same almost blank output. (on my development machine I get lot of output about data being processed etc). any idea before I go deeper to put traces ? (I'm using AW-ETL 0.9.0 from gems, on mac os x) -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070820/2cee5e17/attachment.html From anthonyeden at gmail.com Tue Aug 21 13:06:45 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 21 Aug 2007 13:06:45 -0400 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? In-Reply-To: <4a68b8cf0708201244ne7e9fdelfdf4065abd974a5a@mail.gmail.com> References: <4a68b8cf0708201244ne7e9fdelfdf4065abd974a5a@mail.gmail.com> Message-ID: On 8/20/07, Thibaut Barr?re wrote: > Hi > > I've noticed a very strange thing when deploying my first .ebf/.ctl to a new > machine. > > When launching etl datawarehouse.ebf, I just get: > > ================ > About to launch the ETL process... > Running etl fill_datawarehouse.ebf > Using AdapterExtensions > Starting ETL process > initializing ETL engine > Processing fill_datawarehouse.ebf > Executing batch > Build finished. > ================ > > When launching the .ctl files separately, I also get the same almost blank > output. > > (on my development machine I get lot of output about data being processed > etc). > > > any idea before I go deeper to put traces ? (I'm using AW-ETL 0.9.0 from > gems, on mac os x) The only thing I can suggest is to take a look at etl.log and see if that shows errors occurring. Additionally you may want to set the logger level to DEBUG. V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Wed Aug 22 07:49:01 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Wed, 22 Aug 2007 13:49:01 +0200 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? In-Reply-To: References: <4a68b8cf0708201244ne7e9fdelfdf4065abd974a5a@mail.gmail.com> Message-ID: <4a68b8cf0708220449rffa3db6h84c396e725dcd408@mail.gmail.com> > The only thing I can suggest is to take a look at etl.log and see if > that shows errors occurring. Additionally you may want to set the > logger level to DEBUG. Nothing shows up in etl.log so far. Then I have added : ETL::Engine.logger = Logger.new(STDOUT) ETL::Engine.logger.level = Logger::DEBUG to my .ctl file but I get nothing on the output either. Guess I'm qualifying to go deeper in the code :) thanks for your answer - in case you get more ideas please chime in; I'll try to see what happens and report back. Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070822/652d111e/attachment.html From thibaut.barrere at gmail.com Thu Aug 23 11:26:51 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 23 Aug 2007 17:26:51 +0200 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? [SOLVED] Message-ID: <4a68b8cf0708230826i44482006s54882851c81ce7a7@mail.gmail.com> Hi, I solved the issue (but did not totally understood) - my CI machine was running ruby 1.8.4 - I upgraded to 1.8.6 and everything runs smoothly. Maybe worth putting a warning somewhere in the documentation - the behaviour was indeed very strange. I may take some time to launch the test suite on an older ruby to see if anything pops up. cheers -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/7a9b0ac8/attachment.html From anthonyeden at gmail.com Thu Aug 23 11:31:11 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 23 Aug 2007 11:31:11 -0400 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? [SOLVED] In-Reply-To: <4a68b8cf0708230826i44482006s54882851c81ce7a7@mail.gmail.com> References: <4a68b8cf0708230826i44482006s54882851c81ce7a7@mail.gmail.com> Message-ID: Good to know. I bet there's an exception being caught and not reported somewhere. I've put a little requirements section in the README to indicate the need for Ruby 1.8.6. V/r Anthony On 8/23/07, Thibaut Barr?re wrote: > Hi, > > I solved the issue (but did not totally understood) - my CI machine was > running ruby 1.8.4 - I upgraded to 1.8.6 and everything runs smoothly. > > Maybe worth putting a warning somewhere in the documentation - the behaviour > was indeed very strange. > > I may take some time to launch the test suite on an older ruby to see if > anything pops up. > > cheers > > -- Thibaut > -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Thu Aug 23 11:33:23 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 23 Aug 2007 17:33:23 +0200 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? [SOLVED] In-Reply-To: References: <4a68b8cf0708230826i44482006s54882851c81ce7a7@mail.gmail.com> Message-ID: <4a68b8cf0708230833h3a1d3536wdffa92e99bacf744@mail.gmail.com> > Good to know. I bet there's an exception being caught and not reported > somewhere. I've put a little requirements section in the README to > indicate the need for Ruby 1.8.6. One more detail: everything seems to work fine on 1.8.5 too (my dev machine), but not on 1.8.4. -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/aee1b477/attachment.html From anthonyeden at gmail.com Thu Aug 23 11:34:34 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 23 Aug 2007 11:34:34 -0400 Subject: [Activewarehouse-discuss] Strange behaviour - any hint before going further ? [SOLVED] In-Reply-To: <4a68b8cf0708230833h3a1d3536wdffa92e99bacf744@mail.gmail.com> References: <4a68b8cf0708230826i44482006s54882851c81ce7a7@mail.gmail.com> <4a68b8cf0708230833h3a1d3536wdffa92e99bacf744@mail.gmail.com> Message-ID: README updated. V/r Anthony On 8/23/07, Thibaut Barr?re wrote: > > > > Good to know. I bet there's an exception being caught and not reported > > somewhere. I've put a little requirements section in the README to > > indicate the need for Ruby 1.8.6. > > One more detail: everything seems to work fine on 1.8.5 too (my dev > machine), but not on 1.8.4. > > -- Thibaut > -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Thu Aug 23 16:52:08 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 23 Aug 2007 22:52:08 +0200 Subject: [Activewarehouse-discuss] How do you use screens with today's implementation ? Message-ID: <4a68b8cf0708231352t1ad88b37s86bdb236b8004f76@mail.gmail.com> Hi! I work on files at the source and destination level, then do a bulk import, which means that the screens (currently running before the post-processing) cannot see anything in the database (or they have to parse the output files or anything like that). I'm curious here - how other people (and most likely Anthony) use the screen feature as implemented today ? ps: because I'll most likely have to tune the way AW-ETL work before the end of my current contract to get the behaviour I need (like returning error results on etl screen failure, or maybe screen processing too), I have used Piston to import the trunk of AW-ETL and its dependencies, and modified the gems loading accordingly, before patching anything. If anyone is interested in a more detailed write-up on following the trunk with piston, drop a line. regards, Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/8ee65202/attachment.html From anthonyeden at gmail.com Thu Aug 23 17:05:33 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 23 Aug 2007 17:05:33 -0400 Subject: [Activewarehouse-discuss] How do you use screens with today's implementation ? In-Reply-To: <4a68b8cf0708231352t1ad88b37s86bdb236b8004f76@mail.gmail.com> References: <4a68b8cf0708231352t1ad88b37s86bdb236b8004f76@mail.gmail.com> Message-ID: Currently I don't use them very well. ;-) The screen implementation that's there is more of a starting point, providing an idea of how screens might eventually work. I need to do a lot more on making them actually work with the processing stream, which means capturing more meta data about the runtime results. For now I mostly use them to check things like whether or not the destination flat file has the appropriate number of columns throughout. What I would like to use them for is to check the results *after* the warehouse has been loaded...but I'll need to tweak things a bit to make that happen. V/r Anthony On 8/23/07, Thibaut Barr?re wrote: > Hi! > > I work on files at the source and destination level, then do a bulk import, > which means that the screens (currently running before the post-processing) > cannot see anything in the database (or they have to parse the output files > or anything like that). > > I'm curious here - how other people (and most likely Anthony) use the screen > feature as implemented today ? > > ps: because I'll most likely have to tune the way AW-ETL work before the end > of my current contract to get the behaviour I need (like returning error > results on etl screen failure, or maybe screen processing too), I have used > Piston to import the trunk of AW-ETL and its dependencies, and modified the > gems loading accordingly, before patching anything. If anyone is interested > in a more detailed write-up on following the trunk with piston, drop a line. > > regards, > > Thibaut > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Thu Aug 23 17:31:19 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 23 Aug 2007 23:31:19 +0200 Subject: [Activewarehouse-discuss] etl binary does not return an error when a fatal screen doesn't pass - expected behaviour ? In-Reply-To: References: <4a68b8cf0708170712r39df2d55re033447d314e17b1@mail.gmail.com> Message-ID: <4a68b8cf0708231431w896768brc5be0ddc6f43eb71@mail.gmail.com> > I actually hadn't considered returning an error code, but given your > use case I think it's a reasonable expectation. I'll see what I can > do. Here's a patch which seems to work for me - pretty basic: return non-zero error code if a fatal screen fails. Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/1fb19b1d/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: engine_non_zero_exit_code_on_fatal_screen_failure.diff Type: application/octet-stream Size: 550 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/1fb19b1d/attachment-0001.obj From thibaut.barrere at gmail.com Thu Aug 23 17:54:50 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 23 Aug 2007 23:54:50 +0200 Subject: [Activewarehouse-discuss] How do you use screens with today's implementation ? In-Reply-To: References: <4a68b8cf0708231352t1ad88b37s86bdb236b8004f76@mail.gmail.com> Message-ID: <4a68b8cf0708231454s58099281qdb8c0348f02e0208@mail.gmail.com> Very interesting to see how different people can use the same software differently :) In my case, I've ended up moving engine.rb's post_process(control) call just above the scren processing, and I do simple things like: screen(:fatal) { expected_number_of_days = (END_DATE - START_DATE).to_i + 1 assert_equal(expected_number_of_days, ETL::Engine.connection(CONFIG).select_value("SELECT count(*) from date_dimension").to_i, "Number of days in database should match") assert_equal START_DATE.to_s, ETL::Engine.connection(CONFIG).select_value("SELECT sql_date_stamp FROM date_dimension LIMIT 1") } very basic check but there is room for more. Just a very raw idea: maybe the screen could be considered to be a more general kind of Processor which just happens to take a block ? This way we could write things like: after_read(:screen,:fatal) { assert ... } or place it as the first or last post_process, depending on the required behaviour... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/d07baed5/attachment.html From anthonyeden at gmail.com Thu Aug 23 20:28:30 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 23 Aug 2007 20:28:30 -0400 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> Message-ID: I've applied your changes to the trunk. Thanks! V/r Anthony On 8/17/07, Thibaut Barr?re wrote: > Here's a more complete release, with tests and documentation patch. Notes: > - encode_source_latin1.txt should go under tests/data. > - the doc will probably need to be refactored a bit, as this processor could > be used as both a pre/post-processor... > > cheers, > > -- Thibaut > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From anthonyeden at gmail.com Thu Aug 23 20:33:43 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 23 Aug 2007 20:33:43 -0400 Subject: [Activewarehouse-discuss] Get rows out of the query results In-Reply-To: References: Message-ID: You can either use the each method in CubeQueryResult with a block or one of the values methods in that same class. V/r Anthony On 8/14/07, Yeshwanth Sriram wrote: > > How to get rows out of the query results. Need to construct a chart. Are > there any users who have used activewarehouse for charting purposes. I'm > using activewarehouse from trunk. > > Y > > > > > > Yeshwanth Sriram > PS Engineer > (650) 421-7111 Direct > (650) 421-4201 Fax > > *Unlock the Business Potential of Digital Messaging* > www.strongmail.com > > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Cell: 808 782-5046 Current Location: Melbourne, FL -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/70c603f6/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 2640 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070823/70c603f6/attachment.gif From thibaut.barrere at gmail.com Fri Aug 24 05:55:12 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 24 Aug 2007 11:55:12 +0200 Subject: [Activewarehouse-discuss] [littl'contribution] File encode processor (built upon iconv) In-Reply-To: References: <4a68b8cf0708170744k5fefff8aj643c20652474de2c@mail.gmail.com> <4a68b8cf0708170759w7951d54es4cbac049647542ab@mail.gmail.com> <4a68b8cf0708171251u2af44cf5n66b71f67579bc762@mail.gmail.com> Message-ID: <4a68b8cf0708240255t39951b93r1cd8a5295ebab413@mail.gmail.com> Great! Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070824/cadc53b9/attachment.html From thibaut.barrere at gmail.com Fri Aug 24 10:46:50 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 24 Aug 2007 16:46:50 +0200 Subject: [Activewarehouse-discuss] Batch behaviour - is it expected ? Message-ID: <4a68b8cf0708240746k17b74e1fgef839def4c444003@mail.gmail.com> Hi, I've noticed a couple of things with ebf processing: the lines count (read/wrote) which is reported at the end of each .ctl call seems to be cumulative. The --limit behaviour also follows this cumulative counter. Is it the intented behaviour ? Or would it may make sense to display the .ctl line count followed by the .ebf line count (cumulative) ? cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070824/c37875c2/attachment.html From anthonyeden at gmail.com Fri Aug 24 11:00:13 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 24 Aug 2007 11:00:13 -0400 Subject: [Activewarehouse-discuss] Batch behaviour - is it expected ? In-Reply-To: <4a68b8cf0708240746k17b74e1fgef839def4c444003@mail.gmail.com> References: <4a68b8cf0708240746k17b74e1fgef839def4c444003@mail.gmail.com> Message-ID: On 8/24/07, Thibaut Barr?re wrote: > Hi, > > I've noticed a couple of things with ebf processing: the lines count > (read/wrote) which is reported at the end of each .ctl call seems to be > cumulative. The --limit behaviour also follows this cumulative counter. > > Is it the intented behaviour ? > > Or would it may make sense to display the .ctl line count followed by the > .ebf line count (cumulative) ? It would make sense to do this. V/r Anthony -- Cell: 808 782-5046 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Sat Aug 25 18:30:38 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Sun, 26 Aug 2007 00:30:38 +0200 Subject: [Activewarehouse-discuss] Handling missing records in foreign key lookup transform (and an ActiveRecordResolver patch) Message-ID: <4a68b8cf0708251530m2ffe84ddo6fa69bc416551a8a@mail.gmail.com> Hi, The doc states that "[the Enumerable source] is often used to inject a default record into the head of the ETL pipeline when populating a dimension. For example, if you had records in your operational database tied to people and if it is possible for the record to have a null reference in it's person_id foreign key, then you might want an 'Unknown' person." I have to say this feature is *very* practical, and it got my need covered as far as dimension filling is concerned (FWIW, here's an extract in case the Hash[*] trick is useful to someone else): (..) available_fields = CsvHelper.get_fields_names(preprocessed_file) required_fields = [ :"Login Name",:"Full Name", :Company ] default_record = Hash[*required_fields.zip(['unknown','unknown','My Company']).flatten] (..) source :default, { :type => :enumerable, :enumerable => [default_record] } source :in, { :file => preprocessed_file, :parser => :delimited, :skip_lines => 1 }, available_fields In my case though, facts records can happen to have a person_id which does not match any available people record, which makes the following transform fails: transform :creator, :foreign_key_lookup, {:resolver => ActiveRecordResolver.new(PeopleDimension, :find_by_login)} I patched (attached) ActiveRecordResolver to allow the optional passing of a fallback key: transform :creator, :foreign_key_lookup, {:resolver => ActiveRecordResolver.new(PeopleDimension, :find_by_login, 'unknown')} Basically if the resolve fails, it tries to resolve again with find_by_login('unknown'). I'd be curious to know how other people handle this case, if ever needed. Anthony: if there is no other easy way to achieve this, maybe it's worth including in AW-ETL ? (either in this form, or as a separate new resolver). cheers -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070826/503f3167/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: foreign_key_lookup_with_default_record.diff Type: application/octet-stream Size: 1806 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070826/503f3167/attachment-0001.obj From activewarehouse at munkyboy.com Mon Aug 27 19:45:30 2007 From: activewarehouse at munkyboy.com (Michael Luu) Date: Mon, 27 Aug 2007 16:45:30 -0700 Subject: [Activewarehouse-discuss] non-cubed reports Message-ID: <184C79ED-D13B-4796-BBEA-294D1962ED8B@munkyboy.com> Hello all, First off, thanks to all the contributors to activewarehouse. You kind souls who realized that data warehousing code doesn't have to be a bloated Java implementation... I'm wondering if anyone has thought about adding a report that only supports hierarchical navigation on rows. I basically want to display a list of aggregate columns for each row. For example: CarDimension has a hierarchy of Country -> Make -> Model CustomerDimension has a hierarchy of name (aka no real hierarchy) SalesFact includes Car and Customer Dimension and the following aggregates base price tax I would want to show a report table like this: Product Hierarcy | Avg Base Price | Total Tax | Total Customers | etc... USA | 32 | 22 | 31 | ... Japan | 35 | 39 | 31 | ... Korea | 31 | 87 | 31 | ... and allow the user to navigate to... Product Hierarcy | Avg Base Price | Total Tax | Total Customers | etc... Ford | 32 | 22 | 12 | ... Chevy | 35 | 39 | 21 | ... Saturn | 31 | 87 | 9 | ... Cheers, Mike From ysriram at strongmail.com Tue Aug 28 00:18:26 2007 From: ysriram at strongmail.com (Yeshwanth Sriram) Date: Mon, 27 Aug 2007 21:18:26 -0700 Subject: [Activewarehouse-discuss] non-cubed reports In-Reply-To: <184C79ED-D13B-4796-BBEA-294D1962ED8B@munkyboy.com> References: <184C79ED-D13B-4796-BBEA-294D1962ED8B@munkyboy.com> Message-ID: Sort of like perl version of pivot. It would be nice to have sub-totals/grand-totals/row-totals. -----Original Message----- From: activewarehouse-discuss-bounces at rubyforge.org [mailto:activewarehouse-discuss-bounces at rubyforge.org] On Behalf Of Michael Luu Sent: Monday, August 27, 2007 4:46 PM To: activewarehouse-discuss at rubyforge.org Subject: [Activewarehouse-discuss] non-cubed reports Hello all, First off, thanks to all the contributors to activewarehouse. You kind souls who realized that data warehousing code doesn't have to be a bloated Java implementation... I'm wondering if anyone has thought about adding a report that only supports hierarchical navigation on rows. I basically want to display a list of aggregate columns for each row. For example: CarDimension has a hierarchy of Country -> Make -> Model CustomerDimension has a hierarchy of name (aka no real hierarchy) SalesFact includes Car and Customer Dimension and the following aggregates base price tax I would want to show a report table like this: Product Hierarcy | Avg Base Price | Total Tax | Total Customers | etc... USA | 32 | 22 | 31 | ... Japan | 35 | 39 | 31 | ... Korea | 31 | 87 | 31 | ... and allow the user to navigate to... Product Hierarcy | Avg Base Price | Total Tax | Total Customers | etc... Ford | 32 | 22 | 12 | ... Chevy | 35 | 39 | 21 | ... Saturn | 31 | 87 | 9 | ... Cheers, Mike _______________________________________________ Activewarehouse-discuss mailing list Activewarehouse-discuss at rubyforge.org http://rubyforge.org/mailman/listinfo/activewarehouse-discuss From thibaut.barrere at gmail.com Wed Aug 29 15:13:45 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Wed, 29 Aug 2007 21:13:45 +0200 Subject: [Activewarehouse-discuss] Feature suggestion - block processor Message-ID: <4a68b8cf0708291213ge8e2006obf345a39fbf79fc8@mail.gmail.com> Hi, what do you think about having a block processor, like we have a block transform today ? after_read do |row| ... end if it's useful to more than just myself, I could have a look at implementing it. cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070829/7b689cfc/attachment.html From anthonyeden at gmail.com Wed Aug 29 16:08:54 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 29 Aug 2007 16:08:54 -0400 Subject: [Activewarehouse-discuss] Feature suggestion - block processor In-Reply-To: <4a68b8cf0708291213ge8e2006obf345a39fbf79fc8@mail.gmail.com> References: <4a68b8cf0708291213ge8e2006obf345a39fbf79fc8@mail.gmail.com> Message-ID: Yes, it is a very useful idea. V/r Anthony On 8/29/07, Thibaut Barr?re wrote: > Hi, > > what do you think about having a block processor, like we have a block > transform today ? > > after_read do |row| > ... > end > > if it's useful to more than just myself, I could have a look at implementing > it. > > cheers > > Thibaut > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Cell: 808 782-5046 Current Location: Melbourne, FL From endersonmaia at gmail.com Thu Aug 30 19:34:15 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: Thu, 30 Aug 2007 20:34:15 -0300 Subject: [Activewarehouse-discuss] Can't run etl with demo app Message-ID: <228840e70708301634m7ad7a2e0r643877c14fef2788@mail.gmail.com> I get this error, running etl from base direcotry for the app, and I have the same database.yml in config/ and db/etl/ $ etl author_dimension.ctl Using AdapterExtensions Starting ETL process initializing ETL engine /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:33:in `read': No such file or directory - database.yml (Errno::ENOENT) from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:33:in `init' from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:79:in `execute' from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 /bin/../lib/etl/commands/etl.rb:90 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 /bin/etl:28 from /usr/bin/etl:16:in `load' from /usr/bin/etl:16 If I run etl com the lib/etl/ dir, I get a diferente error $ etl author_dimension.ctl Using AdapterExtensions Starting ETL process initializing ETL engine /usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/connection_specification.rb:204:in `establish_connection': etl_execution database is not configured (ActiveRecord::AdapterNotSpecified) from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 /bin/../lib/etl/execution/base.rb:6 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in `new_constants_in' from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 /bin/../lib/etl/execution.rb:16 from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require' ... 11 levels... from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in `require' from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 /bin/etl:28 from /usr/bin/etl:16:in `load' from /usr/bin/etl:16 I'm not understanding what's happening! -- Enderson Maia -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070830/ba5b91bd/attachment.html From endersonmaia at gmail.com Thu Aug 30 19:42:40 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: Thu, 30 Aug 2007 20:42:40 -0300 Subject: [Activewarehouse-discuss] Can't run etl with demo app In-Reply-To: <228840e70708301634m7ad7a2e0r643877c14fef2788@mail.gmail.com> References: <228840e70708301634m7ad7a2e0r643877c14fef2788@mail.gmail.com> Message-ID: <228840e70708301642x744c8152rb0e1d957becb36ec@mail.gmail.com> Ok, ok! I found :) I used the database.yml.example and changed ! Now it runs fine! On 8/30/07, Enderson Maia wrote: > > I get this error, running etl from base direcotry for the app, and I have > the same database.yml in config/ and db/etl/ > > $ etl author_dimension.ctl > Using AdapterExtensions > Starting ETL process > initializing ETL engine > /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:33:in > `read': No such file or directory - database.yml (Errno::ENOENT) > from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:33:in `init' > from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/commands/etl.rb:79:in > `execute' > from /usr/lib/ruby/gems/1.8/gems/activewarehouse- etl-0.9.0 > /bin/../lib/etl/commands/etl.rb:90 > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `gem_original_require' > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in > `new_constants_in' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 > /bin/etl:28 > from /usr/bin/etl:16:in `load' > from /usr/bin/etl:16 > > > If I run etl com the lib/etl/ dir, I get a diferente error > > $ etl author_dimension.ctl > Using AdapterExtensions > Starting ETL process > initializing ETL engine > /usr/lib/ruby/gems/1.8/gems/activerecord- 1.15.3/lib/active_record/connection_adapters/abstract/connection_specification.rb:204:in > `establish_connection': etl_execution database is not configured > (ActiveRecord::AdapterNotSpecified) > from /usr/lib/ruby/gems/1.8/gems/activewarehouse- etl-0.9.0 > /bin/../lib/etl/execution/base.rb:6 > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `gem_original_require' > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:342:in > `new_constants_in' > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/execution.rb:16 > > from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:27:in > `gem_original_require' > ... 11 levels... > from /usr/lib/ruby/gems/1.8/gems/activesupport-1.4.2/lib/active_support/dependencies.rb:495:in > `require' > from /usr/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0 > /bin/etl:28 > from /usr/bin/etl:16:in `load' > from /usr/bin/etl:16 > > > I'm not understanding what's happening! > > > > -- > Enderson Maia -- Enderson Maia -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070830/05362076/attachment-0001.html From endersonmaia at gmail.com Fri Aug 31 13:43:42 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: Fri, 31 Aug 2007 14:43:42 -0300 Subject: [Activewarehouse-discuss] Hierarchy Dimension Message-ID: <228840e70708311043g2e372089ke352e7d602ae30df@mail.gmail.com> Can someone give me an example of using a dimension as hierarchy. I tryed the documentation, but it's confuse (for me) I have something here: ProductDimension ProductGroupDimension Where Product belongs_to :group, :foreign_key => :group_id, :class_name => "ProductGroupDimension" How can I use this in my report, so I can first show the groups, and after the products in that group. Am I clear ? PS: I tryed to translate my models names, 'cause I'm doing it in portuguese. -- Enderson Maia -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070831/15608537/attachment.html From jbateman0769 at gmail.com Fri Aug 31 14:09:42 2007 From: jbateman0769 at gmail.com (John Bateman) Date: Fri, 31 Aug 2007 14:09:42 -0400 Subject: [Activewarehouse-discuss] Hierarchy Dimension In-Reply-To: <228840e70708311043g2e372089ke352e7d602ae30df@mail.gmail.com> References: <228840e70708311043g2e372089ke352e7d602ae30df@mail.gmail.com> Message-ID: <8d39cd3b0708311109qcb027dcm5ff65acf121a99c0@mail.gmail.com> You should actually try and NOT create this kind of hierarchy in a warehouse / star schema if you can avoid it. Relational models that are normalized are traditionally for OLTP (Transactional) systems where as DE-NORMALIZED setups are for OLAP (analytical) systems. What you can do it create Cartesian of your group / products in a single dimension and reference that in your fact. Table: ProductGroupDimension ID: 1 PRODUCT: Apple GROUP: Fruit ID: 2 PRODUCT: Banana GROUP: Fruit ID: 3 PRODUCT: Carrot GROUP: Veggie FactTable: ID: 1 product_group_id: 2 sold: 3 price: $0.50 total: 1.50 A Good idea when getting into warehousing is to check out the Richard Kimball book "Data Warhousing Toolkit. There are some interesting talks about this sort of thing and avoiding a 'snow flake' schema and creating a 'star' instead. Ons of the biggest things to remember, warehouses are DE-NORMALIZED wherever possible. Hope this helps. On 8/31/07, Enderson Maia wrote: > > Can someone give me an example of using a dimension as hierarchy. > > I tryed the documentation, but it's confuse (for me) > > I have something here: > > ProductDimension > ProductGroupDimension > > Where Product belongs_to :group, :foreign_key => :group_id, :class_name => > "ProductGroupDimension" > > > How can I use this in my report, so I can first show the groups, and after > the products in that group. > > Am I clear ? > > > PS: I tryed to translate my models names, 'cause I'm doing it in > portuguese. > -- > Enderson Maia > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070831/76bcc91f/attachment.html From endersonmaia at gmail.com Fri Aug 31 14:48:02 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: Fri, 31 Aug 2007 15:48:02 -0300 Subject: [Activewarehouse-discuss] Hierarchy Dimension In-Reply-To: <8d39cd3b0708311109qcb027dcm5ff65acf121a99c0@mail.gmail.com> References: <228840e70708311043g2e372089ke352e7d602ae30df@mail.gmail.com> <8d39cd3b0708311109qcb027dcm5ff65acf121a99c0@mail.gmail.com> Message-ID: <228840e70708311148y82fc1e0kea99b2a274eb83ea@mail.gmail.com> Thanks, but acts_as_hierarchy_dimension isn't for this ? I will need to have a fact table for GroupSales and ProductSales, but I'd like to make a drill-down with groups and products in a group. And still have the database for GroupSales, so I don't need to group by ProductGoup. Did u got the point ? I tested what you said, it works. :) On 8/31/07, John Bateman wrote: > > You should actually try and NOT create this kind of hierarchy in a > warehouse / star schema if you can avoid it. > > Relational models that are normalized are traditionally for OLTP > (Transactional) systems where as DE-NORMALIZED setups are for OLAP > (analytical) systems. > > What you can do it create Cartesian of your group / products in a single > dimension and reference that in your fact. > > Table: ProductGroupDimension > ID: 1 > PRODUCT: Apple > GROUP: Fruit > > ID: 2 > PRODUCT: Banana > GROUP: Fruit > > ID: 3 > PRODUCT: Carrot > GROUP: Veggie > > FactTable: > > ID: 1 > product_group_id: 2 > sold: 3 > price: $0.50 > total: 1.50 > > A Good idea when getting into warehousing is to check out the Richard > Kimball book "Data Warhousing Toolkit. There are some interesting talks > about this sort of thing and avoiding a 'snow flake' schema and creating a > 'star' instead. > > Ons of the biggest things to remember, warehouses are DE-NORMALIZED > wherever possible. > > Hope this helps. > > On 8/31/07, Enderson Maia wrote: > > > Can someone give me an example of using a dimension as hierarchy. > > > > I tryed the documentation, but it's confuse (for me) > > > > I have something here: > > > > ProductDimension > > ProductGroupDimension > > > > Where Product belongs_to :group, :foreign_key => :group_id, :class_name > > => "ProductGroupDimension" > > > > > > How can I use this in my report, so I can first show the groups, and > > after the products in that group. > > > > Am I clear ? > > > > > > PS: I tryed to translate my models names, 'cause I'm doing it in > > portuguese. > > -- > > Enderson Maia > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Enderson Maia -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20070831/e57d1e26/attachment.html