From obrien.andrew at gmail.com Wed May 23 14:45:29 2007 From: obrien.andrew at gmail.com (Andrew O'Brien) Date: Wed, 23 May 2007 14:45:29 -0400 Subject: [Activewarehouse-discuss] Slight problem with date_dimension generator Message-ID: I had a slight problem with the date_dimension generator migration template. The migration creates a column called "sql_date". If you run "rake warehouse:build_date_dimension" it doesn't set any value for that key. However, since sql_date can't be null, the task fails. After looking at date_dimension_builder.rb I changed the column to "sql_date_stamp" and it works. Is this a bug or did I just miss an option somewhere along the way? Thanks, Andrew From anthonyeden at gmail.com Wed May 23 14:53:32 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 23 May 2007 14:53:32 -0400 Subject: [Activewarehouse-discuss] Slight problem with date_dimension generator In-Reply-To: References: Message-ID: It's probably a bug. I'll take a look at it this evening. V/r Anthony On 5/23/07, Andrew O'Brien wrote: > I had a slight problem with the date_dimension generator migration > template. The migration creates a column called "sql_date". If you > run "rake warehouse:build_date_dimension" it doesn't set any value for > that key. However, since sql_date can't be null, the task fails. > > After looking at date_dimension_builder.rb I changed the column to > "sql_date_stamp" and it works. > > Is this a bug or did I just miss an option somewhere along the way? > > Thanks, > Andrew > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Cell: 808 782-5046 Current Location: Melbourne, FL From anthonyeden at gmail.com Wed May 23 16:26:42 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 23 May 2007 16:26:42 -0400 Subject: [Activewarehouse-discuss] Slight problem with date_dimension generator In-Reply-To: References: Message-ID: I've changed the generator to use sql_date_stamp. V/r Anthony On 5/23/07, Andrew O'Brien wrote: > I had a slight problem with the date_dimension generator migration > template. The migration creates a column called "sql_date". If you > run "rake warehouse:build_date_dimension" it doesn't set any value for > that key. However, since sql_date can't be null, the task fails. > > After looking at date_dimension_builder.rb I changed the column to > "sql_date_stamp" and it works. > > Is this a bug or did I just miss an option somewhere along the way? > > Thanks, > Andrew > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Cell: 808 782-5046 Current Location: Melbourne, FL From j56ach at verizon.net Mon May 28 16:15:37 2007 From: j56ach at verizon.net (Gary Fleshman) Date: Mon, 28 May 2007 16:15:37 -0400 Subject: [Activewarehouse-discuss] Date Dimension Builder Issue Message-ID: Using rev 763 of rails_warehouse/trunk, I was able to load all of the dimension tables, but my first attempt to load the file_revision_facts.ctl failed: "Exiting due to exceeding error threshold: 100". Running again with options -l 101 and -s showed that each record failed. This is what came to light: - db/migrate/001_create_date_dimension.rb defines a field ':calendar_week_ending_date' - plugin/ ... /date_dimension_builder.rb:48 comments out the corresponding record My workaround: commenting out the offending field in 001_create_date_dimension.rb. This resulted in a successful load of the file_revision_facts table. Thank you for this fantastic example/tutorial. Gary Fleshman New Forge Technologies Vienna, VA From j56ach at verizon.net Mon May 28 15:49:18 2007 From: j56ach at verizon.net (Gary Fleshman) Date: Mon, 28 May 2007 15:49:18 -0400 Subject: [Activewarehouse-discuss] (no subject) Message-ID: Using rev 763 of rails_warehouse/trunk, I was able to load all of the dimension tables, but my first attempt to load the file_revision_facts.ctl failed: "Exiting due to exceeding error threshold: 100". Running again with options -l 101 and -s showed that each record failed. This is what came to light: - db/migrate/001_create_date_dimension.rb defines a field ':calendar_week_ending_date' - plugin/ ... /date_dimension_builder.rb:48 comments out the corresponding record My workaround: commenting out the offending field in 001_create_date_dimension.rb. This resulted in a successful load of the file_revision_facts table. Thank you for this fantastic example/tutorial. Gary Fleshman New Forge Technologies Vienna, VA From timshadel at gmail.com Tue May 29 16:38:53 2007 From: timshadel at gmail.com (Tim Shadel) Date: Tue, 29 May 2007 13:38:53 -0700 Subject: [Activewarehouse-discuss] (no subject) In-Reply-To: References: Message-ID: <4313B840-0E1D-47F1-9BEF-C0379C64D0C5@gmail.com> I had that happen to me last night as well. I found a different solution. I changed line 16 from if value to this if value || name.to_s !~ /id/ The problem comes in that the migration for date dimension includes a 'calendar_week_ending_date' column, while the date dimension generator for AW doesn't populate that field at all, leaving it null. When the post_process kicks in, it doesn't see a null value for that column and everything after that gets loaded into the wrong columns. You don't discover this until you try to load your Fact table, and the error log shows that it can't find the date for some value like '2004-11-01'. If you look at the SQL_TIME_STAMP column, you'll see it's messed up. Line 16, mentioned above, made sure that there were no null fields in the date dimension. I figured I wanted to get that field worked out soon, though, and opted for this solution. Obviously, your solution works just beautifully (and I should have read your email before diving in last night!). Hope that provides another perspective, Tim On May 28, 2007, at 12:49 PM, Gary Fleshman wrote: > Using rev 763 of rails_warehouse/trunk, I was able to load all of the > dimension tables, but my first attempt to load the > file_revision_facts.ctl failed: "Exiting due to exceeding error > threshold: 100". > > Running again with options -l 101 and -s showed that each record > failed. > > This is what came to light: > - db/migrate/001_create_date_dimension.rb defines a field > ':calendar_week_ending_date' > - plugin/ ... /date_dimension_builder.rb:48 comments out the > corresponding record > > My workaround: commenting out the offending field in > 001_create_date_dimension.rb. This resulted in a successful load of > the file_revision_facts table. > > Thank you for this fantastic example/tutorial. > > Gary Fleshman > New Forge Technologies > Vienna, VA > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss From kbedell at courseadvisor.com Wed May 30 13:01:25 2007 From: kbedell at courseadvisor.com (Kevin Bedell) Date: Wed, 30 May 2007 13:01:25 -0400 Subject: [Activewarehouse-discuss] Introduction Message-ID: Hello all -- I've just joined the list and am looking at using this project on a DW project I have going on. Can anyone point me to some info that can get me started? A co-worker of mine just returned from the railsconf and recommended this as a project I may want to look at. We're already a rails shop. If we use it, we will likely contribute code as well. Thanks! Kevin Bedell Courseadvisor.com From anthonyeden at gmail.com Wed May 30 13:29:10 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 30 May 2007 13:29:10 -0400 Subject: [Activewarehouse-discuss] Introduction In-Reply-To: References: Message-ID: Welcome, Kevin. I recently published fairly comprehensive guide for the ActiveWarehouse ETL library at http://activewarehouse.rubyforge.org/docs/activewarehouse-etl.html. This is the library you'll use to get data into the warehouse. For the ActiveWarehouse plugin you can read through the tutorial at http://anthonyeden.com/2006/12/20/activewarehouse-example-with-rails-svn-logs, however it is out of date so the code samples will probably not work right now. I will be updating that tutorial as soon as I can and will also be publishing a guide for the ActiveWarehouse plugin. Finally, feel free to ask questions here if you get stumped. V/r Anthony Eden On 5/30/07, Kevin Bedell wrote: > Hello all -- I've just joined the list and am looking at using this project > on a DW project I have going on. > > Can anyone point me to some info that can get me started? A co-worker of > mine just returned from the railsconf and recommended this as a project I > may want to look at. We're already a rails shop. If we use it, we will > likely contribute code as well. > > Thanks! > Kevin Bedell > Courseadvisor.com > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Cell: 808 782-5046 Current Location: Melbourne, FL From j56ach at verizon.net Wed May 30 16:18:33 2007 From: j56ach at verizon.net (Gary Fleshman) Date: Wed, 30 May 2007 16:18:33 -0400 Subject: [Activewarehouse-discuss] Date Dimension Builder Issue In-Reply-To: <4313B840-0E1D-47F1-9BEF-C0379C64D0C5@gmail.com> References: <4313B840-0E1D-47F1-9BEF-C0379C64D0C5@gmail.com> Message-ID: I prefer your enhancement to date_dimension.ctl, as it protects ETL, regardless of the values returned by the activewarehouse plugin. Considering that - ETL is a client (user) of the activewarehouse plugin - Definition of the dimension and fact tables occurs within the db/ migrate files (expected behavior) Would additional protection be worthwhile - Prior to *.ctl post processing, validate (via ~ unit test) the migrate definition consistent with the anticipated bulk import Gary Fleshman New Forge Technologies Vienna, VA ......................................... On May 29, 2007, at 4:38 PM, Tim Shadel wrote: > I had that happen to me last night as well. I found a different > solution. > > I changed line 16 from > if value > to this > if value || name.to_s !~ /id/ > > The problem comes in that the migration for date dimension includes > a 'calendar_week_ending_date' column, while the date dimension > generator for AW doesn't populate that field at all, leaving it > null. When the post_process kicks in, it doesn't see a null value > for that column and everything after that gets loaded into the > wrong columns. You don't discover this until you try to load your > Fact table, and the error log shows that it can't find the date for > some value like '2004-11-01'. If you look at the SQL_TIME_STAMP > column, you'll see it's messed up. > > Line 16, mentioned above, made sure that there were no null fields > in the date dimension. I figured I wanted to get that field worked > out soon, though, and opted for this solution. Obviously, your > solution works just beautifully (and I should have read your email > before diving in last night!). > > Hope that provides another perspective, > > Tim > > On May 28, 2007, at 12:49 PM, Gary Fleshman wrote: > >> Using rev 763 of rails_warehouse/trunk, I was able to load all of the >> dimension tables, but my first attempt to load the >> file_revision_facts.ctl failed: "Exiting due to exceeding error >> threshold: 100". >> >> Running again with options -l 101 and -s showed that each record >> failed. >> >> This is what came to light: >> - db/migrate/001_create_date_dimension.rb defines a field >> ':calendar_week_ending_date' >> - plugin/ ... /date_dimension_builder.rb:48 comments out the >> corresponding record >> >> My workaround: commenting out the offending field in >> 001_create_date_dimension.rb. This resulted in a successful load of >> the file_revision_facts table. >> >> Thank you for this fantastic example/tutorial. >> >> Gary Fleshman >> New Forge Technologies >> Vienna, VA >> >> _______________________________________________ >> Activewarehouse-discuss mailing list >> Activewarehouse-discuss at rubyforge.org >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From anthonyeden at gmail.com Wed May 30 16:21:48 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 30 May 2007 16:21:48 -0400 Subject: [Activewarehouse-discuss] Date Dimension Builder Issue In-Reply-To: References: <4313B840-0E1D-47F1-9BEF-C0379C64D0C5@gmail.com> Message-ID: Can someone please provide a patch and I'll apply it. Thanks. V/r Anthony On 5/30/07, Gary Fleshman wrote: > I prefer your enhancement to date_dimension.ctl, as it protects ETL, > regardless of the values returned by the activewarehouse plugin. > > Considering that > - ETL is a client (user) of the activewarehouse plugin > - Definition of the dimension and fact tables occurs within the db/ > migrate files (expected behavior) > > Would additional protection be worthwhile > - Prior to *.ctl post processing, validate (via ~ unit test) the > migrate definition consistent with the anticipated bulk import > > Gary Fleshman > New Forge Technologies > Vienna, VA > ......................................... > > > On May 29, 2007, at 4:38 PM, Tim Shadel wrote: > > > I had that happen to me last night as well. I found a different > > solution. > > > > I changed line 16 from > > if value > > to this > > if value || name.to_s !~ /id/ > > > > The problem comes in that the migration for date dimension includes > > a 'calendar_week_ending_date' column, while the date dimension > > generator for AW doesn't populate that field at all, leaving it > > null. When the post_process kicks in, it doesn't see a null value > > for that column and everything after that gets loaded into the > > wrong columns. You don't discover this until you try to load your > > Fact table, and the error log shows that it can't find the date for > > some value like '2004-11-01'. If you look at the SQL_TIME_STAMP > > column, you'll see it's messed up. > > > > Line 16, mentioned above, made sure that there were no null fields > > in the date dimension. I figured I wanted to get that field worked > > out soon, though, and opted for this solution. Obviously, your > > solution works just beautifully (and I should have read your email > > before diving in last night!). > > > > Hope that provides another perspective, > > > > Tim > > > > On May 28, 2007, at 12:49 PM, Gary Fleshman wrote: > > > >> Using rev 763 of rails_warehouse/trunk, I was able to load all of the > >> dimension tables, but my first attempt to load the > >> file_revision_facts.ctl failed: "Exiting due to exceeding error > >> threshold: 100". > >> > >> Running again with options -l 101 and -s showed that each record > >> failed. > >> > >> This is what came to light: > >> - db/migrate/001_create_date_dimension.rb defines a field > >> ':calendar_week_ending_date' > >> - plugin/ ... /date_dimension_builder.rb:48 comments out the > >> corresponding record > >> > >> My workaround: commenting out the offending field in > >> 001_create_date_dimension.rb. This resulted in a successful load of > >> the file_revision_facts table. > >> > >> Thank you for this fantastic example/tutorial. > >> > >> Gary Fleshman > >> New Forge Technologies > >> Vienna, VA > >> > >> _______________________________________________ > >> Activewarehouse-discuss mailing list > >> Activewarehouse-discuss at rubyforge.org > >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Cell: 808 782-5046 Current Location: Melbourne, FL From anthonyeden at gmail.com Wed May 30 16:57:53 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Wed, 30 May 2007 16:57:53 -0400 Subject: [Activewarehouse-discuss] Date Dimension Builder Issue In-Reply-To: References: <4313B840-0E1D-47F1-9BEF-C0379C64D0C5@gmail.com> Message-ID: Scratch that, I just went and applied the change directly. V/r Anthony On 5/30/07, Anthony Eden wrote: > Can someone please provide a patch and I'll apply it. Thanks. > > V/r > Anthony > > On 5/30/07, Gary Fleshman wrote: > > I prefer your enhancement to date_dimension.ctl, as it protects ETL, > > regardless of the values returned by the activewarehouse plugin. > > > > Considering that > > - ETL is a client (user) of the activewarehouse plugin > > - Definition of the dimension and fact tables occurs within the db/ > > migrate files (expected behavior) > > > > Would additional protection be worthwhile > > - Prior to *.ctl post processing, validate (via ~ unit test) the > > migrate definition consistent with the anticipated bulk import > > > > Gary Fleshman > > New Forge Technologies > > Vienna, VA > > ......................................... > > > > > > On May 29, 2007, at 4:38 PM, Tim Shadel wrote: > > > > > I had that happen to me last night as well. I found a different > > > solution. > > > > > > I changed line 16 from > > > if value > > > to this > > > if value || name.to_s !~ /id/ > > > > > > The problem comes in that the migration for date dimension includes > > > a 'calendar_week_ending_date' column, while the date dimension > > > generator for AW doesn't populate that field at all, leaving it > > > null. When the post_process kicks in, it doesn't see a null value > > > for that column and everything after that gets loaded into the > > > wrong columns. You don't discover this until you try to load your > > > Fact table, and the error log shows that it can't find the date for > > > some value like '2004-11-01'. If you look at the SQL_TIME_STAMP > > > column, you'll see it's messed up. > > > > > > Line 16, mentioned above, made sure that there were no null fields > > > in the date dimension. I figured I wanted to get that field worked > > > out soon, though, and opted for this solution. Obviously, your > > > solution works just beautifully (and I should have read your email > > > before diving in last night!). > > > > > > Hope that provides another perspective, > > > > > > Tim > > > > > > On May 28, 2007, at 12:49 PM, Gary Fleshman wrote: > > > > > >> Using rev 763 of rails_warehouse/trunk, I was able to load all of the > > >> dimension tables, but my first attempt to load the > > >> file_revision_facts.ctl failed: "Exiting due to exceeding error > > >> threshold: 100". > > >> > > >> Running again with options -l 101 and -s showed that each record > > >> failed. > > >> > > >> This is what came to light: > > >> - db/migrate/001_create_date_dimension.rb defines a field > > >> ':calendar_week_ending_date' > > >> - plugin/ ... /date_dimension_builder.rb:48 comments out the > > >> corresponding record > > >> > > >> My workaround: commenting out the offending field in > > >> 001_create_date_dimension.rb. This resulted in a successful load of > > >> the file_revision_facts table. > > >> > > >> Thank you for this fantastic example/tutorial. > > >> > > >> Gary Fleshman > > >> New Forge Technologies > > >> Vienna, VA > > >> > > >> _______________________________________________ > > >> Activewarehouse-discuss mailing list > > >> Activewarehouse-discuss at rubyforge.org > > >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > -- > Cell: 808 782-5046 > Current Location: Melbourne, FL > -- Cell: 808 782-5046 Current Location: Melbourne, FL