From thibaut.barrere at gmail.com Mon Oct 1 17:16:09 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Mon, 1 Oct 2007 23:16:09 +0200 Subject: [Activewarehouse-discuss] [rails_sql_views/patch] support for implicit columns Message-ID: <4a68b8cf0710011416t7530fc15vfa94b85f28fc7797@mail.gmail.com> Hi, today rails_sql_views requires explicit view fields declaration, ie: create_view(:view_name,"select first_name,last_name from my_table") do |view| view.column :first_name view.column :last_name end I've patched my local adapter_extensions to support fields declarations implicit to allow this: create_view(:view_name,"select first_name,last_name from my_table") FWIW, my own need is to ease my work while I'm working around today's lack of support for subqueries in mysql views. Should this be committed to the trunk ? (patch attached, doesn't add support for create_mapping_view though). cheers, Thibaut Barr?re -- LoGeek [blog] http://www.dotnetguru2.org/tbarrere -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071001/724de369/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: implicit_columns.patch Type: application/octet-stream Size: 1187 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071001/724de369/attachment.obj From anthonyeden at gmail.com Tue Oct 2 07:55:33 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 2 Oct 2007 07:55:33 -0400 Subject: [Activewarehouse-discuss] [rails_sql_views/patch] support for implicit columns In-Reply-To: <4a68b8cf0710011416t7530fc15vfa94b85f28fc7797@mail.gmail.com> References: <4a68b8cf0710011416t7530fc15vfa94b85f28fc7797@mail.gmail.com> Message-ID: Any ideas on whether or not this patch works in PostgreSQL? V/r Anthony On 10/1/07, Thibaut Barr?re wrote: > Hi, > > today rails_sql_views requires explicit view fields declaration, ie: > > create_view(:view_name,"select first_name,last_name from my_table") do > |view| > view.column :first_name > view.column :last_name > end > > I've patched my local adapter_extensions to support fields declarations > implicit to allow this: > > create_view(:view_name,"select first_name,last_name from my_table") > > FWIW, my own need is to ease my work while I'm working around today's lack > of support for subqueries in mysql views. > > Should this be committed to the trunk ? (patch attached, doesn't add support > for create_mapping_view though). > > cheers, > > Thibaut Barr?re > -- > LoGeek > [blog] http://www.dotnetguru2.org/tbarrere > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > -- Cell: 321 473-4966 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Tue Oct 2 08:13:30 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 2 Oct 2007 14:13:30 +0200 Subject: [Activewarehouse-discuss] [rails_sql_views/patch] support for implicit columns In-Reply-To: References: <4a68b8cf0710011416t7530fc15vfa94b85f28fc7797@mail.gmail.com> Message-ID: <4a68b8cf0710020513q19ec9711p4d1271987c6e6dd5@mail.gmail.com> > > Any ideas on whether or not this patch works in PostgreSQL? > Nope - only tested it on MySQL so far. I'm not sure if it's standard sql or not at all. Btw, I met issues (only when running tests) with db:schema:dump/load connected to views creations; I had to work-around it by setting the schema dumper to :sql in environment.rb. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071002/1428d940/attachment.html From thibaut.barrere at gmail.com Tue Oct 2 09:42:22 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 2 Oct 2007 15:42:22 +0200 Subject: [Activewarehouse-discuss] Any way to retrieve command line arguments from inside the .ctl ? Message-ID: <4a68b8cf0710020642p2b5bbdd4u1735d79c281fba4d@mail.gmail.com> Hi, is there a clean way to retrieve custom command line arguments from inside the .ctl ? After looking at commands/etl.rb, I'd say it's not possible (or maybe there is a Getoptlong trick ?). I'm pretty sure setting up ENV['MYVAR'] = "mydata" should also do the trick. My use case is Rake tasks called with parameters in various environments. Any hint ? cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071002/529feeae/attachment.html From thibaut.barrere at gmail.com Fri Oct 5 03:11:09 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 5 Oct 2007 09:11:09 +0200 Subject: [Activewarehouse-discuss] Behaviour differences betwen after_read and before_write Message-ID: <4a68b8cf0710050011x39816f38r83eed56d29aeaf55@mail.gmail.com> Hi, it seems that when a RowProcessor used as a before_write returns nil, the row is removed from the pipeline, but when the same processor is used as an after_read, the nil row remains in the pipeline (and the next processor will most likely raise a nil exception at soon as it interacts with the row). The difference seems to lie in engine.rb (process_control): after_read: rows = processed_rows.flatten before_write: rows = processed_rows.flatten.compact Is there a specific reason for not having compact called as well on after_read ? My feeling is that it may be worth DRYing process_control - for the moment I didn't analyse further to evaluate the cost and possible side-effects on existing installations though... cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071005/ae8da52d/attachment.html From anthonyeden at gmail.com Fri Oct 5 11:41:46 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 5 Oct 2007 11:41:46 -0400 Subject: [Activewarehouse-discuss] Behaviour differences betwen after_read and before_write In-Reply-To: <4a68b8cf0710050011x39816f38r83eed56d29aeaf55@mail.gmail.com> References: <4a68b8cf0710050011x39816f38r83eed56d29aeaf55@mail.gmail.com> Message-ID: On 10/5/07, Thibaut Barr?re wrote: > Hi, > > it seems that when a RowProcessor used as a before_write returns nil, the > row is removed from the pipeline, but when the same processor is used as an > after_read, the nil row remains in the pipeline (and the next processor will > most likely raise a nil exception at soon as it interacts with the row). > > The difference seems to lie in engine.rb (process_control): > > after_read: rows = processed_rows.flatten > before_write: rows = processed_rows.flatten.compact > > Is there a specific reason for not having compact called as well on > after_read ? I assumed that you would want to let your row go through your transform pipeline before compacting it to remove nils. > My feeling is that it may be worth DRYing process_control - for the moment I > didn't analyse further to evaluate the cost and possible side-effects on > existing installations though... It is probably worth it, and I have tried to refactor some of it, however I have not completely refactored it. Ideally I think that the process_control would basically be a bunch of calls to internal methods to express each of the processing steps. V/r Anthony -- Cell: 321 473-4966 Current Location: Melbourne, FL From anthonyeden at gmail.com Fri Oct 5 13:03:40 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 5 Oct 2007 13:03:40 -0400 Subject: [Activewarehouse-discuss] Any way to retrieve command line arguments from inside the .ctl ? In-Reply-To: <4a68b8cf0710020642p2b5bbdd4u1735d79c281fba4d@mail.gmail.com> References: <4a68b8cf0710020642p2b5bbdd4u1735d79c281fba4d@mail.gmail.com> Message-ID: Well you could probably still access ARGV directly. Personally I just use ENV, however if we need to make it easier to get the CL args we could. V/r Anthony On 10/2/07, Thibaut Barr?re wrote: > Hi, > > is there a clean way to retrieve custom command line arguments from inside > the .ctl ? After looking at commands/etl.rb, I'd say it's not possible (or > maybe there is a Getoptlong trick ?). I'm pretty sure setting up > ENV['MYVAR'] = "mydata" should also do the trick. > > My use case is Rake tasks called with parameters in various environments. > > Any hint ? > > cheers > > Thibaut > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > -- Cell: 321 473-4966 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Fri Oct 5 14:38:59 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 5 Oct 2007 20:38:59 +0200 Subject: [Activewarehouse-discuss] Any way to retrieve command line arguments from inside the .ctl ? In-Reply-To: References: <4a68b8cf0710020642p2b5bbdd4u1735d79c281fba4d@mail.gmail.com> Message-ID: <4a68b8cf0710051138p1e2d4ac4he637aa440b55a8ee@mail.gmail.com> > Well you could probably still access ARGV directly. Personally I just > use ENV, however if we need to make it easier to get the CL args we > could. I finally picked up ENV as well, because it's really simple. I just prefix the ENV vars with something specific enough to avoid conflicts. Good enough for me. Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071005/706823a9/attachment.html From thibaut.barrere at gmail.com Tue Oct 9 05:10:47 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 9 Oct 2007 11:10:47 +0200 Subject: [Activewarehouse-discuss] I need to avoid the copy of my source files to the source_data folder, can you help ? Message-ID: <4a68b8cf0710090210ma5ad4d3w6574413e9f7053d5@mail.gmail.com> Hi, I'd need to accelerate the processing of my web logs with AW-ETL. I noticed that the copy of these logs to the "source_data" folder takes a significant amount of time in this case of mine. Is there a simple way to disable this automatic copy, and use the files in-place instead ? (before I dive a bit further into the code). thanks! Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071009/8f64b36f/attachment-0001.html From anthonyeden at gmail.com Tue Oct 9 12:36:26 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 9 Oct 2007 12:36:26 -0400 Subject: [Activewarehouse-discuss] I need to avoid the copy of my source files to the source_data folder, can you help ? In-Reply-To: <4a68b8cf0710090210ma5ad4d3w6574413e9f7053d5@mail.gmail.com> References: <4a68b8cf0710090210ma5ad4d3w6574413e9f7053d5@mail.gmail.com> Message-ID: On 10/9/07, Thibaut Barr?re wrote: > Hi, > > I'd need to accelerate the processing of my web logs with AW-ETL. I noticed > that the copy of these logs to the "source_data" folder takes a significant > amount of time in this case of mine. > > Is there a simple way to disable this automatic copy, and use the files > in-place instead ? (before I dive a bit further into the code). There is no way to disable this at the moment. V/r Anthony -- Cell: 321 473-4966 Current Location: Melbourne, FL From thibaut.barrere at gmail.com Tue Oct 9 15:30:30 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 9 Oct 2007 21:30:30 +0200 Subject: [Activewarehouse-discuss] I need to avoid the copy of my source files to the source_data folder, can you help ? In-Reply-To: References: <4a68b8cf0710090210ma5ad4d3w6574413e9f7053d5@mail.gmail.com> Message-ID: <4a68b8cf0710091230t5c4e5ba9s4ae1fb8a08b0247e@mail.gmail.com> > There is no way to disable this at the moment. Thanks for the answer. -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071009/f9d84011/attachment.html From maze at strahlungsfrei.de Thu Oct 11 12:52:49 2007 From: maze at strahlungsfrei.de (Martin Honermeyer) Date: Thu, 11 Oct 2007 18:52:49 +0200 Subject: [Activewarehouse-discuss] Multiple Sources for one dimension? Message-ID: <200710111852.49248.maze@strahlungsfrei.de> Hello guys, I followed this project and the mailing list for a while. For me as a keen Rails developer, this sounds really interesting! Now it's time to get active - I am starting my first data warehousing project, building upon our business. I managed to get ActiveWarehouse and the ETL tool (both trunk versions) to work with Edge Rails and our MSSQL server. Since we have two operational systems with product and order data, I wonder if and how I can extract data from multiple differing sources into the same dimension, preferably in one .ctl file? Just setting up two sources works, but it seems they must have the same table structure (field names, values etc.). As I can have only one destination in my .ctl and I have to define the fields in there, I can't differentiate between sources. Is that right? Or is it possible to have different transforms, depending on the source, in order to unify operational source data? I'd really like to help get this going! Thanks in advance Martin Honermeyer From anthonyeden at gmail.com Sun Oct 14 17:14:01 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Sun, 14 Oct 2007 11:14:01 -1000 Subject: [Activewarehouse-discuss] Multiple Sources for one dimension? In-Reply-To: <200710111852.49248.maze@strahlungsfrei.de> References: <200710111852.49248.maze@strahlungsfrei.de> Message-ID: On 10/11/07, Martin Honermeyer wrote: > Hello guys, > > I followed this project and the mailing list for a while. For me as a keen > Rails developer, this sounds really interesting! Now it's time to get > active - I am starting my first data warehousing project, building upon our > business. > > I managed to get ActiveWarehouse and the ETL tool (both trunk versions) to > work with Edge Rails and our MSSQL server. Since we have two operational > systems with product and order data, I wonder if and how I can extract data > from multiple differing sources into the same dimension, preferably in > one .ctl file? > > Just setting up two sources works, but it seems they must have the same table > structure (field names, values etc.). As I can have only one destination in > my .ctl and I have to define the fields in there, I can't differentiate > between sources. Is that right? Or is it possible to have different > transforms, depending on the source, in order to unify operational source > data? You can have multiple sources *and* multiple destinations, and you can mix and match as needed. All the sources do is essentially stick values into a row that is represented as a Hash of name/value pairs. Therefore you could have one source put the keys :name, :age, :sex and then a second source could put the keys :favorite_color, :favorite_animal into the row as well. At this point you can then process as you need and ultimately write the values you want back to the destination(s). You do this by specifying the fields to be written using the :order option in the third argument of the destination configuration. So for example, given the sources described above, your destination might look something like this: destination :out, { :type => :database, :target => :data_warehouse, :database => 'etl_unittest', :table => 'people', }, { :order => [:name, :age, :sex, :favorite_color, :favorite_animal] } HTH. V/r Anthony -- Cell: 321 473-4966 Current Location: Honolulu, HI Home Location: Melbourne, FL From endersonmaia at gmail.com Mon Oct 15 14:20:08 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: 15 Oct 2007 11:20:08 -0700 Subject: [Activewarehouse-discuss] Do we like the same books? Message-ID: <20071015183131.3180018585B4@rubyforge.org> I just joined Shelfari to connect with other book lovers. Come see the books I love and see if we have any in common. Then pick my next book so I can keep on reading. Click below to join my group of friends on Shelfari! http://www.shelfari.com/Register.aspx?ActivityId=26625145&InvitationCode=e2416071-ac91-40bd-aac9-7ef1b3fc81b6 Enderson Maia Shelfari is a free site that lets you share book ratings and reviews with friends and meet people who have similar tastes in books. It also lets you build an online bookshelf, join book clubs, and get good book recommendations from friends. You should check it out. -------- You have received this email because Enderson Maia (endersonmaia at gmail.com) directly invited you to join his/her community on Shelfari. It is against Shelfari's policies to invite people who you don't know directly. Follow this link (http://www.shelfari.com/actions/emailoptout.aspx?email=activewarehouse-discuss at rubyforge.org&activityid=26625145) to prevent future invitations to this address. If you believe you do not know this person, you may view (http://www.shelfari.com/o1516326391) his/her Shelfari page or report him/her in our feedback (http://www.shelfari.com/Feedback.aspx) section. Shelfari, 616 1st Ave #300, Seattle, WA 98104 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071015/498f867c/attachment.html From akappen at gmail.com Tue Oct 16 17:42:15 2007 From: akappen at gmail.com (Andy Kappen) Date: Tue, 16 Oct 2007 16:42:15 -0500 Subject: [Activewarehouse-discuss] TableReport :format and :fact_attributes field name ambiguity Message-ID: Perhaps this is not the proper way to approach this problem, so please correct me if I am doing something silly. I have a fact model that performs two aggregates on the same field. Basically my motivation is to calculate the mean and standard deviation of one fact field. Here's an example of the fact table aggregates: class SomeFact < ActiveWarehouse::Fact aggregate :hours_duration, :type => :avg, :label => "Mean (Hours)" aggregate :hours_duration, :type => :stddev, :label => "StdDev (Hours)" end When I try to create a TableReport that uses this fact table, I am having some problems specifying which aggregate I want when defining :format or :fact_attributes. For instance: @report = ActiveWarehouse::Report::TableReport.new( . . . :format => { :hours_duration => Proc.new {|value| sprintf("%.2f", value)} }, :fact_attributes => [:hours_duration] ) The problem with :format is obvious. . . if I wanted to display the mean with two decimal places and the standard deviation with four decimal places, I can't find a way to specify which 'fact field' to apply which format to. The :fact_attribute has a similar problem. . . if I want to see the mean and standard deviation fields, there is no way to list both in the :fact_attributes because they share a 'field name'. Am I overlooking something obvious here? I looked at the AW code and it appears to always fetch the aggregate field by its field name in the table. Is there a way to apply another identifier to an aggregate so that I can discriminate between them? Cheers, Andy K. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071016/403fd392/attachment.html From endersonmaia at gmail.com Thu Oct 18 04:45:07 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: 18 Oct 2007 01:45:07 -0700 Subject: [Activewarehouse-discuss] Do we read the same books? Message-ID: <20071018084500.4F1B818585DD@rubyforge.org> Just a friendly reminder that I invited you to Shelfari. Come see the books I love and see if we have any in common. Then pick my next book so I can keep on reading. Click below to join my group of friends on Shelfari! http://www.shelfari.com/Register.aspx?ActivityId=27981978&InvitationCode=e2416071-ac91-40bd-aac9-7ef1b3fc81b6 Enderson Maia Shelfari is a free site that lets you share book ratings and reviews with friends and meet people who have similar tastes in books. It also lets you build an online bookshelf, join book clubs, and get good book recommendations from friends. You should check it out. -------- You have received this email because Enderson Maia (endersonmaia at gmail.com) directly invited you to join his/her community on Shelfari. It is against Shelfari's policies to invite people who you don't know directly. Follow this link (http://www.shelfari.com/actions/emailoptout.aspx?email=activewarehouse-discuss at rubyforge.org&activityid=27981978) to prevent future invitations to this address. If you believe you do not know this person, you may view (http://www.shelfari.com/o1516326391) his/her Shelfari page or report him/her in our feedback (http://www.shelfari.com/Feedback.aspx) section. Shelfari, 616 1st Ave #300, Seattle, WA 98104 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071018/0826b2ab/attachment.html From colin.dmz at gmail.com Mon Oct 29 12:45:33 2007 From: colin.dmz at gmail.com (Colin Surprenant) Date: Mon, 29 Oct 2007 12:45:33 -0400 Subject: [Activewarehouse-discuss] creating custom transform Message-ID: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> Hi, I am trying to create a custom transform but I am having a hard time with dependencies. As a simple test I just created my custom transform like this directly in my control file: module ETL #:nodoc: module Transform #:nodoc: class TestTransform < ETL::Transform::Transform def transform(name, value, row) # do transform value end end end end I'd appreciate if I could get help figuring the required dependencies to enable the definition of a custom transform such a this in my project. Thanks, Colin. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071029/c4d7612e/attachment-0001.html From thibaut.barrere at gmail.com Mon Oct 29 13:35:33 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Mon, 29 Oct 2007 18:35:33 +0100 Subject: [Activewarehouse-discuss] creating custom transform In-Reply-To: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> Message-ID: <4a68b8cf0710291035y7488ecb4md3304773efe6d7cb@mail.gmail.com> Hi Colin, could you copy paste your exception stacktrace here ? It will help us figure things out. cheers Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071029/20de0405/attachment.html From colin.dmz at gmail.com Mon Oct 29 13:46:46 2007 From: colin.dmz at gmail.com (Colin Surprenant) Date: Mon, 29 Oct 2007 13:46:46 -0400 Subject: [Activewarehouse-discuss] creating custom transform In-Reply-To: <4a68b8cf0710291035y7488ecb4md3304773efe6d7cb@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> <4a68b8cf0710291035y7488ecb4md3304773efe6d7cb@mail.gmail.com> Message-ID: <8a6d52260710291046t7a016c0bge880316dab27ad8c@mail.gmail.com> Hi, Sure, thanks. I also tried adding the following requires, same result: require 'etl/transform/transform' require 'etl/control/control' c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:266:in `load_missing_constant': uninitialized constant ETL::Control::Context::ETL::Transform::Transform (NameError) from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:452:in `const_missing' from control/ibegin_model.ctl:67:in `get_binding' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/control.rb:232:in `get_binding' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/control.rb:12:in `create' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/control.rb:262:in `parse' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/control/control.rb:286:in `resolve' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:292:in `process_control' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:261:in `process' ... 10 levels... from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-1.4.4/lib/active_support/dependencies.rb:495:in `require' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 from c:/ruby/bin/etl:16:in `load' from c:/ruby/bin/etl:16 On 10/29/07, Thibaut Barr?re wrote: > > Hi Colin, > > could you copy paste your exception stacktrace here ? It will help us > figure things out. > > cheers > > Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071029/85a44ee0/attachment.html From sleight42 at gmail.com Mon Oct 29 16:28:25 2007 From: sleight42 at gmail.com (Evan David Light) Date: Mon, 29 Oct 2007 16:28:25 -0400 Subject: [Activewarehouse-discuss] CSV Report? In-Reply-To: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> Message-ID: I don't suppose that anyone has written a CSV report capability? It looks like it would likely be a simple matter to refactor the current report_helper.rb to barf out CSV instead of XHTML. However, I'd rather not go reinventing the wheel. Thanks, Evan From evan at tiggerpalace.com Mon Oct 29 18:43:31 2007 From: evan at tiggerpalace.com (Evan David Light) Date: Mon, 29 Oct 2007 18:43:31 -0400 Subject: [Activewarehouse-discuss] CSV report Message-ID: Howdy, all. I've been using ActiveWarehouse for about a month now. After having used MicroStrategy Intelligence Server briefly about 4 years ago, ActiveWarehouse is a breath of fresh air. I'm loving it. That said, I don't suppose that anyone has written a CSV report capability? It looks like it would likely be a simple matter to refactor the current report_helper.rb to barf out CSV instead of XHTML. However, I'd rather not go reinventing the wheel. I imagine that this feature would make more sense as a report that could be launched/generated as a link from a particular XHTML view of a cube. Thanks, Evan P.S. Apologies if you get a duplicate of this. RubyForge's mailing lists may be running slowly. From evan at tiggerpalace.com Mon Oct 29 19:19:27 2007 From: evan at tiggerpalace.com (Evan David Light) Date: Mon, 29 Oct 2007 19:19:27 -0400 Subject: [Activewarehouse-discuss] Bug in ETL::Control::DatabaseSource? In-Reply-To: References: Message-ID: <53849E92-9B1C-49E8-9A9B-B9A0C3E34CC8@tiggerpalace.com> Currently reads: conditions << configuration[:conditions] if configuration[:conditions] However, because configuration[:conditions] appears to expect an Array of Strings representing conditions, shouldn't this read something more like: conditions |= configuration[:conditions] if configuration[:conditions] Evan From thibaut.barrere at gmail.com Tue Oct 30 02:03:56 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 30 Oct 2007 07:03:56 +0100 Subject: [Activewarehouse-discuss] creating custom transform In-Reply-To: <8a6d52260710291046t7a016c0bge880316dab27ad8c@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> <4a68b8cf0710291035y7488ecb4md3304773efe6d7cb@mail.gmail.com> <8a6d52260710291046t7a016c0bge880316dab27ad8c@mail.gmail.com> Message-ID: <4a68b8cf0710292303j6cd3d767ve4fec5edcbaddeb5@mail.gmail.com> can you try this ? - put the transform code into test_transform.rb (under the same folder where your .ctl is) - add require 'test_transform' a the top of the .ctl file cheers Thibaut From thibaut.barrere at gmail.com Tue Oct 30 06:09:08 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Tue, 30 Oct 2007 11:09:08 +0100 Subject: [Activewarehouse-discuss] CSV Report? In-Reply-To: References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> Message-ID: <4a68b8cf0710300309v42112bdes5a982c1d32d5dd86@mail.gmail.com> Hi Evan, I'm using AW-ETL with Ruports [1] for reporting purposes. I'm not generating CSV specifically, but I think it's in there. cheers Thibaut [1] http://groups.google.com/group/ruby-reports From anthonyeden at gmail.com Tue Oct 30 07:45:07 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 30 Oct 2007 07:45:07 -0400 Subject: [Activewarehouse-discuss] CSV report In-Reply-To: References: Message-ID: On 10/29/07, Evan David Light wrote: > Howdy, all. I've been using ActiveWarehouse for about a month now. > After having used MicroStrategy Intelligence Server briefly about 4 > years ago, ActiveWarehouse is a breath of fresh air. I'm loving it. > > That said, I don't suppose that anyone has written a CSV report > capability? It looks like it would likely be a simple matter to > refactor the current report_helper.rb to barf out CSV instead of > XHTML. However, I'd rather not go reinventing the wheel. The front end in the plugin is extremely bare right now. Bottom line is that there isn't much there and that includes CSV reporting. That being said, it wouldn't be too hard to implement basic CSV reporting in Rails using the model classes that are provided with AW. > I imagine that this feature would make more sense as a report that > could be launched/generated as a link from a particular XHTML view of > a cube. I agree. V/r Anthony Eden -- Cell: 321 473-4966 Current Location: Melbourne, FL Home Location: Melbourne, FL From anthonyeden at gmail.com Tue Oct 30 07:53:21 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 30 Oct 2007 07:53:21 -0400 Subject: [Activewarehouse-discuss] Bug in ETL::Control::DatabaseSource? In-Reply-To: <53849E92-9B1C-49E8-9A9B-B9A0C3E34CC8@tiggerpalace.com> References: <53849E92-9B1C-49E8-9A9B-B9A0C3E34CC8@tiggerpalace.com> Message-ID: On 10/29/07, Evan David Light wrote: > Currently reads: > > conditions << configuration[:conditions] if configuration[:conditions] > > However, because configuration[:conditions] appears to expect an Array > of Strings representing conditions, shouldn't this read something more > like: > > conditions |= configuration[:conditions] if configuration[:conditions] It definitely needs to remain <<. The previous chunk of code creates the conditions array and will add a condition if the new_records_only configuration returns true. After this configuration[:conditions] is appended to the conditions array which is then used in the query. Now, while ActiveRecord conditions does support the use of an array where the first element is the conditions SQL string and the remaining arguments are used as replacement variables, I have not used this pattern inside AW ETL, so perhaps that's why you were thinking it should work differently? What I could do is allow it to accept an array in configuration[:conditions] and then flatten the whole array before executing the query, would that work for you? V/r Anthony -- Cell: 321 473-4966 Current Location: Melbourne, FL Home Location: Melbourne, FL From evan at tiggerpalace.com Tue Oct 30 09:59:36 2007 From: evan at tiggerpalace.com (Evan David Light) Date: Tue, 30 Oct 2007 09:59:36 -0400 Subject: [Activewarehouse-discuss] Bug in ETL::Control::DatabaseSource? In-Reply-To: References: <53849E92-9B1C-49E8-9A9B-B9A0C3E34CC8@tiggerpalace.com> Message-ID: Aw nuts. No, the mistake that I made was more elementary than that. See, I was reading the code (while looking at the docs) but had not executed the code. Double dumbass on me for not realizing that ["a", "b", ["c"]].join(' AND ') results in ' a AND b AND c '. :-( '|=' would have resulted in a flat array mergining the unique elements of the two arrays but that's unnecessary. My bad. On Oct 30, 2007, at 7:53 AM, Anthony Eden wrote: > On 10/29/07, Evan David Light wrote: >> Currently reads: >> >> conditions << configuration[:conditions] if >> configuration[:conditions] >> >> However, because configuration[:conditions] appears to expect an >> Array >> of Strings representing conditions, shouldn't this read something >> more >> like: >> >> conditions |= configuration[:conditions] if >> configuration[:conditions] > > It definitely needs to remain <<. The previous chunk of code creates > the conditions array and will add a condition if the new_records_only > configuration returns true. After this configuration[:conditions] is > appended to the conditions array which is then used in the query. > > Now, while ActiveRecord conditions does support the use of an array > where the first element is the conditions SQL string and the remaining > arguments are used as replacement variables, I have not used this > pattern inside AW ETL, so perhaps that's why you were thinking it > should work differently? What I could do is allow it to accept an > array in configuration[:conditions] and then flatten the whole array > before executing the query, would that work for you? > > V/r > Anthony > > -- > Cell: 321 473-4966 > Current Location: Melbourne, FL > Home Location: Melbourne, FL From evan at tiggerpalace.com Tue Oct 30 10:01:34 2007 From: evan at tiggerpalace.com (Evan David Light) Date: Tue, 30 Oct 2007 10:01:34 -0400 Subject: [Activewarehouse-discuss] CSV Report? In-Reply-To: <4a68b8cf0710300309v42112bdes5a982c1d32d5dd86@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> <4a68b8cf0710300309v42112bdes5a982c1d32d5dd86@mail.gmail.com> Message-ID: CSV sure is in Ruport (heh, listened to Greg Brown talk about it at RubyEast just a few weeks ago). Although it looks like it would be stupid easy to build a similar (although perhaps less versatile) capability into AW with some tweaks to the report_helper and perhaps completed with a customer report type. I'll look at both options. Thanks for the tip. On Oct 30, 2007, at 6:09 AM, Thibaut Barr?re wrote: > Hi Evan, > > I'm using AW-ETL with Ruports [1] for reporting purposes. > > I'm not generating CSV specifically, but I think it's in there. > > cheers > > Thibaut > > [1] http://groups.google.com/group/ruby-reports From anthonyeden at gmail.com Tue Oct 30 10:04:45 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 30 Oct 2007 10:04:45 -0400 Subject: [Activewarehouse-discuss] Bug in ETL::Control::DatabaseSource? In-Reply-To: References: <53849E92-9B1C-49E8-9A9B-B9A0C3E34CC8@tiggerpalace.com> Message-ID: No worries, as long as it works for you then that's good. V/r Anthony On 10/30/07, Evan David Light wrote: > Aw nuts. No, the mistake that I made was more elementary than that. > > See, I was reading the code (while looking at the docs) but had not > executed the code. Double dumbass on me for not realizing that ["a", > "b", ["c"]].join(' AND ') results in ' a AND b AND c '. :-( '|=' > would have resulted in a flat array mergining the unique elements of > the two arrays but that's unnecessary. My bad. > > On Oct 30, 2007, at 7:53 AM, Anthony Eden wrote: > > > On 10/29/07, Evan David Light wrote: > >> Currently reads: > >> > >> conditions << configuration[:conditions] if > >> configuration[:conditions] > >> > >> However, because configuration[:conditions] appears to expect an > >> Array > >> of Strings representing conditions, shouldn't this read something > >> more > >> like: > >> > >> conditions |= configuration[:conditions] if > >> configuration[:conditions] > > > > It definitely needs to remain <<. The previous chunk of code creates > > the conditions array and will add a condition if the new_records_only > > configuration returns true. After this configuration[:conditions] is > > appended to the conditions array which is then used in the query. > > > > Now, while ActiveRecord conditions does support the use of an array > > where the first element is the conditions SQL string and the remaining > > arguments are used as replacement variables, I have not used this > > pattern inside AW ETL, so perhaps that's why you were thinking it > > should work differently? What I could do is allow it to accept an > > array in configuration[:conditions] and then flatten the whole array > > before executing the query, would that work for you? > > > > V/r > > Anthony > > > > -- > > Cell: 321 473-4966 > > Current Location: Melbourne, FL > > Home Location: Melbourne, FL > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Cell: 321 473-4966 Current Location: Melbourne, FL Home Location: Melbourne, FL From colin.dmz at gmail.com Tue Oct 30 12:13:41 2007 From: colin.dmz at gmail.com (Colin Surprenant) Date: Tue, 30 Oct 2007 12:13:41 -0400 Subject: [Activewarehouse-discuss] creating custom transform In-Reply-To: <4a68b8cf0710292303j6cd3d767ve4fec5edcbaddeb5@mail.gmail.com> References: <8a6d52260710290945n22dfb70am342972cfc6f739d7@mail.gmail.com> <4a68b8cf0710291035y7488ecb4md3304773efe6d7cb@mail.gmail.com> <8a6d52260710291046t7a016c0bge880316dab27ad8c@mail.gmail.com> <4a68b8cf0710292303j6cd3d767ve4fec5edcbaddeb5@mail.gmail.com> Message-ID: <8a6d52260710300913w16080df7m9e57236d991d644b@mail.gmail.com> Yep - that works! Great, thanks! Colin. On 10/30/07, Thibaut Barr?re wrote: > > can you try this ? > - put the transform code into test_transform.rb (under the same folder > where your .ctl is) > - add require 'test_transform' a the top of the .ctl file > > cheers > > Thibaut > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071030/77d3921c/attachment.html From colin.dmz at gmail.com Tue Oct 30 13:49:36 2007 From: colin.dmz at gmail.com (Colin Surprenant) Date: Tue, 30 Oct 2007 13:49:36 -0400 Subject: [Activewarehouse-discuss] processor class name lookup Message-ID: <8a6d52260710301049k6080f3dh7bd334462a8b3f67@mail.gmail.com> Hi, Is there a particular reason why the processor lookup code in ETL::Control::Context::transform uses the classify method instead of camelize ?? The code in question is: (line 123 in etl/control/control.rb) class_name = "#{transformer.to_s.classify}Transform" The problem is that if you define a custom transformer like let's say "TagsTransform", when you try to invoke it using transform :thefield, :tags the controller will try to invoke the "TagTransform" class instead of "TagsTransform". Thanks, Colin. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071030/66a78fd7/attachment.html From anthonyeden at gmail.com Tue Oct 30 13:51:28 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Tue, 30 Oct 2007 13:51:28 -0400 Subject: [Activewarehouse-discuss] Project Status and What's Next Message-ID: Hello all. I've decided that it's time to put a little bit of effort into deciding where to go next with ActiveWarehouse so here goes. First, let's look at where we are today: * The ActiveWarehouse Rails plugin does a decent job of defining classes that can be used to model a dimensional data warehouse. This includes classes such as Fact, Dimension and Bridge. These classes, in conjunction with their migration generators can be used to construct a basic dimensional model. Additionally we have mix-ins for SCDs, Hierarchical dimensions and classes for dimensional views (role playing) and date dimensions. * The ActiveWarehouse ETL library is fairly full-featured. It has support for file source data including delimited, fixed-width and XML as well as inline data produced by a Ruby Enumerable, data from relational databases and data from Rails model objects. It supports a variety of built-in transforms as well as custom transforms and inline block transforms. Finally it supports output to flat files as well as databases and supports bulk upload through the adapter extensions library for MySQL, PostgreSQL and SQL Server. These are the bright spots. The areas which need improvement are: * ActiveWarehouse Rails plugin reporting tools. Right now AW doesn't give you much in terms of a reporting and query front end. The report_helper.rb code is horrendous and is basically all there is for reporting. * Aggregation in AW is all over the place right now with none of the aggregation implementations delivering much in the way of high-performance aggregation schemes. You can still execute queries through the NoAggregate implementation however for anything substantial you're talking about potentially slow queries. So what's next then: * Release ActiveWarehouse ETL 1.0. I think it's time to freeze this thing. It has documentation, a reasonable number of tests and is essentially usable right now so I think it's time for a release. * Focus on the AW front end. AW needs to come ready to go with at least some basic reporting tools on the front. * Clean out all of the old aggregation techniques and settle on something specific, perhaps based on aggregation techniques in other open source data warehouse libraries? What other items do you think should part of upcoming AW development? If you have time and are interested in contributing please let me know as AW can always benefit from more developers with a passion for data warehousing. V/r Anthony Eden -- Cell: 321 473-4966 Current Location: Melbourne, FL Home Location: Melbourne, FL From marty at logicleaf.com Tue Oct 30 17:42:34 2007 From: marty at logicleaf.com (Marty Haught) Date: Tue, 30 Oct 2007 15:42:34 -0600 Subject: [Activewarehouse-discuss] Top level roll-up Message-ID: I have a question on doing a top level roll-up programmatically. I would like to have a top level called 'All' for several of my hierarchies. Is there an easy way to do this using the TableReport? I can fake it out by adding a column into the desired dimension with every row having the string 'All' in it but this feels wrong. What I'd love to do is this: define_hierarchy :my_hierarchy, [:all, :category, :name] The define_hierarchy method would understand that :all is reserved and this means to show a single row roll-up at the top level. I am new to data warehousing so perhaps what I'm asking for is accomplished in another way. Thanks in advance for your replies. Cheers, Marty