From thibaut.barrere at gmail.com Thu May 1 05:25:12 2008 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Thu, 1 May 2008 11:25:12 +0200 Subject: [Activewarehouse-discuss] activewarehouse-etl and sql server from leopard In-Reply-To: References: Message-ID: <4a68b8cf0805010225l78d9e2a2mde5d4af4ea4a2867@mail.gmail.com> Hi Gunner, Welcome to ActiveWarehouse! Just a quick guess - the missing sqlserver_connection message could indicate that the sqlserver adapter is not installed. Since Rails 2.0, adapters for most of the databases (except mysql, postgres and sqllite I believe) are separate download. In theory you should be able to install this adapter using the following command line: gem install activerecord-sqlserver-adapter --source=http://gems.rubyonrails.org I read some guys stating it wouldn't work these days. If it doesn't, head over the sqlserver-adapter group : http://groups.google.com/group/rails-sqlserver-adapter/ cheers, Thibaut Barr?re -- LoGeek [blog] http://evolvingworker.com - tools for a better day [blog] http://blog.logeek.fr - about writing software -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikedc55 at gmail.com Sun May 4 12:40:09 2008 From: mikedc55 at gmail.com (Mike Yard) Date: Sun, 4 May 2008 18:40:09 +0200 Subject: [Activewarehouse-discuss] AW reports question Message-ID: <1ef730060805040940ob6ad150g25f3225f8b5008a0@mail.gmail.com> Hello, I have two questions about reports i AW. Hope you can help me out here. Q1) I'm trying to figure out if it's at all possible to show a stringvalue where you'd normally aggregate. For example: The following report has a row_dimension_name with persons and a column_dimension_name with information about the person. Sex Beard Date of Birth -------------| ------- |--------- | ----------------- Person1 | Male | Yes | xxxx-xx-xx Person2 | Male | No | xxxx-xx-xx Person3 | Male | No | xxxx-xx-xx Is this possible? Q2) Also, is it possible to show just 2 row_dimension names without displaying a column_dimension_name? E.g. Person 1 | address_of_person_1 Person 2 | address_of_person_2 Thanks. Mike Yard From bob.cotton at rallydev.com Wed May 7 07:06:33 2008 From: bob.cotton at rallydev.com (Bob Cotton) Date: Wed, 07 May 2008 05:06:33 -0600 Subject: [Activewarehouse-discuss] Reusing .ctl files Message-ID: I have an ETL process that needs to run in two modes: - Loading files - Streaming My destination needs to be bulk loaded when reading from files, and writing directly to the DB when streaming. I've managed to subclass the SAX parser to read from STDIN for streaming. I tried splitting the source and destination into different .ctl files and wrapping them in a batch file, but the pipeline was not propagated between the control files. What's the best way to factor out and reuse portions of the ETL pipeline? Thanks -Bob From dwlist at stellarmadness.org Thu May 8 20:41:24 2008 From: dwlist at stellarmadness.org (Darin Wilson) Date: Thu, 8 May 2008 17:41:24 -0700 (PDT) Subject: [Activewarehouse-discuss] Reusing .ctl files Message-ID: <2545.70.231.232.250.1210293684.squirrel@stellarmadness.org> I tried a couple of times to respond to this, but it seems like my messages weren't going through. Trying again here with a different account - my apologies to all if this lands on the list multiple times. :S Anyway, once more with feeling... --------------------------- Hi Bob- The .ctl files are just plain old Ruby code, so you can insert conditionals right into the .ctl file to control the flow. In your case, you might try something like: if mode == 'streaming' destination :out { :database => 'my_db', etc... }, { } else destination :out, { :file => 'outfile.txt', :separator => "," }, { :order => [ ... ] } post_process :bulk_import, { :file => 'outfile.txt', :target => 'target_db', etc... } end If this logic needs to be shared across different .ctl files, you can move the code into helper methods in a separate .rb file, then call the methods from within the .ctl file HTH, Darin > From: Bob Cotton > Date: May 7, 2008 4:06:33 AM PDT > To: activewarehouse-discuss at rubyforge.org > Subject: [Activewarehouse-discuss] Reusing .ctl files > > > I have an ETL process that needs to run in two modes: > > > - Loading files > - Streaming > > > My destination needs to be bulk loaded when reading from files, and > writing directly to the DB when streaming. > > I've managed to subclass the SAX parser to read from STDIN for > streaming. > > I tried splitting the source and destination into different .ctl files > and wrapping them in a batch file, but the pipeline was not propagated > between the control files. > > What's the best way to factor out and reuse portions of the ETL > pipeline? > > Thanks > -Bob > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > From thibaut.barrere at gmail.com Fri May 9 16:13:42 2008 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 9 May 2008 22:13:42 +0200 Subject: [Activewarehouse-discuss] Reusing .ctl files In-Reply-To: References: Message-ID: <4a68b8cf0805091313s67de5f23g18cb94d4a22a75ca@mail.gmail.com> Hi Bob, > What's the best way to factor out and reuse portions of the ETL pipeline? as Darin said, you can conditionally declare sources, transforms and destinations when needed. You can also factor out useful code to a 'common.rb', shared across all your scripts: def some_macro_transform(args...) transform :the_column, :default, :default_value => ''... transform :... transform .. end or again: def parse_french_date(string) Date.strptime(string,'%d/%m/%Y') end to go further, each of my ctl files starts roughly the same way (note the reused logic in bold) table = 'the_table_for_this_ctl' source_file = 'the_data.csv' output_file = *output_file*(source_file) bulk_file = *bulk_file*(source_file) # converted from latin1 to utf-8 preprocessed_file = *preprocessed_file*(source_file) available_fields = *CsvHelper.get_fields_names*(preprocessed_file) The methods or classes you see in bold are reusable, unit-tested assets. To pass conditions to your scripts, one useful trick is that you can pass data to your script using ENV: ENV['datawarehouse_mode'] = 'streaming' # be sure to pick up a non-conflicting name throw "error" unless system("myctl.ctl") (and myctl.ctl would use if ENV['datawarehouse_mode'] == 'streaming' inside). cheers! Thibaut Barr?re -- [freelance developer] http://www.logeek.fr [blog] http://evolvingworker.com - tools for a better day [blog] http://blog.logeek.fr - about writing software -------------- next part -------------- An HTML attachment was scrubbed... URL: From bob.cotton at rallydev.com Mon May 12 22:16:53 2008 From: bob.cotton at rallydev.com (Bob Cotton) Date: Mon, 12 May 2008 20:16:53 -0600 Subject: [Activewarehouse-discuss] Activewarehouse Demo Message-ID: In the process of learning activewarehouse, I got the demo working under Rails 2.0.2/MySql I've published by changes on github, under the 'fixes' branch: http://github.com/bcotton/activewarehouse-demo/commits/fixes Everything works except "example4", with the dynamic queries and drill-downs, still looking into that. It would be great to have Marty integrate the YUI stuff. Still don't grok how to get that installed and working. I can provide patches if needed. -Bob From mghaught at gmail.com Tue May 13 17:09:18 2008 From: mghaught at gmail.com (Marty Haught) Date: Tue, 13 May 2008 15:09:18 -0600 Subject: [Activewarehouse-discuss] Activewarehouse Demo In-Reply-To: References: Message-ID: <57f29e620805131409s6efd6af0u533ae8b5541596d8@mail.gmail.com> Hi Bob, I'll see if I can get a current tutorial for YUI integration posted soon. Thanks for updating the demo. Cheers, Marty On Mon, May 12, 2008 at 8:16 PM, Bob Cotton wrote: > > In the process of learning activewarehouse, I got the demo working under > Rails 2.0.2/MySql > > I've published by changes on github, under the 'fixes' branch: > > http://github.com/bcotton/activewarehouse-demo/commits/fixes > > Everything works except "example4", with the dynamic queries and > drill-downs, still looking into that. > > It would be great to have Marty integrate the YUI stuff. Still don't > grok how to get that installed and working. > > I can provide patches if needed. > > -Bob > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From mghaught at gmail.com Wed May 14 15:26:00 2008 From: mghaught at gmail.com (Marty Haught) Date: Wed, 14 May 2008 13:26:00 -0600 Subject: [Activewarehouse-discuss] AW reports question In-Reply-To: <1ef730060805040940ob6ad150g25f3225f8b5008a0@mail.gmail.com> References: <1ef730060805040940ob6ad150g25f3225f8b5008a0@mail.gmail.com> Message-ID: <57f29e620805141226l623cf8d7n8c086f6418bd6edd@mail.gmail.com> Hi Mike, Sorry for the delay. I was going to see if someone else replied first. On Sun, May 4, 2008 at 10:40 AM, Mike Yard wrote: > Hello, > I have two questions about reports i AW. Hope you can help me out here. > Q1) I'm trying to figure out if it's at all possible to show a > stringvalue where you'd normally aggregate. > For example: > The following report has a row_dimension_name with persons and a > column_dimension_name with information about the person. > > Sex Beard Date of Birth > -------------| ------- |--------- | ----------------- > Person1 | Male | Yes | xxxx-xx-xx > Person2 | Male | No | xxxx-xx-xx > Person3 | Male | No | xxxx-xx-xx > > Is this possible? Well, I've never done it. The issue that comes to mind is that you don't know that you're working with a single row from the fact table. Thus if your current dimension is comprised of 10 or so fact rows how would you know what to return? You might try using a type of max. It wouldn't insure that all rows for that dimension are identical but it may return the value as you want. Try this in your fact: aggregate :gender, :type => 'max' You could make mods to the code to return the first of the list but it would have the same downside as max would. > Q2) Also, is it possible to show just 2 row_dimension names without > displaying a column_dimension_name? > E.g. > > Person 1 | address_of_person_1 > Person 2 | address_of_person_2 We sort of did this. First, the rendering code needs two dimensions. The same is true for the cube engine. I know an empty dimension is on the wish list but I don't see that happening soon. What you could try is fake it out. We made a dimension called EmptyDimension. We created a table just like for any other dimension with an id and name. It had one row, which was the string we wanted displayed on the form. We called our statistics. Then you add this dimension to your fact like any other dimension and give all fact rows the foreign key to the empty dimension table. Not as cool as it could be but it worked for us. Cheers, Marty From mghaught at gmail.com Wed May 14 18:35:19 2008 From: mghaught at gmail.com (Marty Haught) Date: Wed, 14 May 2008 16:35:19 -0600 Subject: [Activewarehouse-discuss] AW reports question In-Reply-To: <57f29e620805141226l623cf8d7n8c086f6418bd6edd@mail.gmail.com> References: <1ef730060805040940ob6ad150g25f3225f8b5008a0@mail.gmail.com> <57f29e620805141226l623cf8d7n8c086f6418bd6edd@mail.gmail.com> Message-ID: <57f29e620805141535q18fd4615pef456a954eef49a9@mail.gmail.com> Hi Mike, There's another option. My last email's suggestions assume you want to use a TableReport with your fact table. You can access a fact just like any other ActiveRecord object directly. You can add traditional belongs_to associations on your fact to the dimensions you want. There's nothing stopping you wrong writing a different kind of report with your fact data. If you want drill down and dimensional slicing/dicing then you'll want to stay with a TableReport otherwise you can go to ActiveRecord directly and display the data like you would on a normal Rails app. Hope that helps. Cheers, Marty From anthonyeden at gmail.com Fri May 16 14:23:04 2008 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 16 May 2008 11:23:04 -0700 Subject: [Activewarehouse-discuss] ActiveWarehouse now on github Message-ID: Hello all, I know it has been a while but I wanted to update you all and let you know that ActiveWarehouse is now in GitHub: http://github.com/aeden/activewarehouse/tree/master I've also separated out Rails SQL Views and Adapter Extensions into their own github repos since those projects are really outside the scope of ActiveWarehouse. Those repos are: http://github.com/aeden/rails_sql_views/tree/master http://github.com/aeden/adapter_extensions/tree/master respectively. If you make some hot changes to any of these projects on a fork and would like it introduced into my repo then please feel free to send a pull request. I still consider myself a newbie on git though so it make take a bit for changes to get pushed into my repo...however given the power of git and github that is mostly irrelevant since you can cherry pick the things you need anyhow. :-) I'm not going to make any promises at this time about resuming my activity on ActiveWarehouse however I know that there are others like Thibaut Barr?re, Marty Haught and Bob Cotton who have been busy using and contributing to ActiveWarehouse, so it is alive and well. I'm pretty confident that this move to git will help make this process of improvement even more open. Thanks to all! V/r Anthony PS Consider the subversion repos at RubyForge EOL at this point...I'll be updating the web pages and such in the near future.