From thibaut.barrere at gmail.com  Wed Apr  1 06:15:28 2009
From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=)
Date: Wed, 1 Apr 2009 12:15:28 +0200
Subject: [Activewarehouse-discuss] truncate decimal?
In-Reply-To: <7c581bf40903310809g7d6895c4t9abc6cc2758a5ef5@mail.gmail.com>
References: <7c581bf40903310809g7d6895c4t9abc6cc2758a5ef5@mail.gmail.com>
Message-ID: <4a68b8cf0904010315r6830103uf6809a9198356918@mail.gmail.com>

Hello Giovanni,
it would be useful to know if the decimals are still there in the bulk file
(I suppose so) or if they are still there.

If it can help you, in my cases I use :float in the migration (which is good
enough for me) and things seem to work fine.

cheers,

-- Thibaut

2009/3/31 Giovanni Messina <jgameit at gmail.com>

> Using this post_process I lose the decimal places of which there are
> suggested_retail_price in outfile
>
> ---------------
> in the ctl file:
>
> post_process :bulk_import, {
>    :file => outfile,
>    :truncate => true,
>    :columns => [:id,:description,:suggested_retail_price],
>    :field_separator => "\t",
>    :target => :development,
>    :table => 'product_dimension'
>  }
>
> -----------------
>
> the migration file is
>
> :id => :integer,
> :description => :string,
> :suggested_retail_price => :decimal
>
> -------------------
>
> how do I act?
>
> Giovanni
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20090401/8597ca54/attachment.html>

From jgameit at gmail.com  Wed Apr  1 07:01:21 2009
From: jgameit at gmail.com (Giovanni Messina)
Date: Wed, 1 Apr 2009 13:01:21 +0200
Subject: [Activewarehouse-discuss] truncate decimal?
In-Reply-To: <4a68b8cf0904010315r6830103uf6809a9198356918@mail.gmail.com>
References: <7c581bf40903310809g7d6895c4t9abc6cc2758a5ef5@mail.gmail.com>
	<4a68b8cf0904010315r6830103uf6809a9198356918@mail.gmail.com>
Message-ID: <7c581bf40904010401s778094d3w139c36f9f2b14c67@mail.gmail.com>

Hi, thanks for the suggestion

with :float works even if I am surprised that round the value using :decimal

cheers

Giovanni



2009/4/1 Thibaut Barr?re <thibaut.barrere at gmail.com>

> Hello Giovanni,
> it would be useful to know if the decimals are still there in the bulk file
> (I suppose so) or if they are still there.
>
> If it can help you, in my cases I use :float in the migration (which is
> good enough for me) and things seem to work fine.
>
> cheers,
>
> -- Thibaut
>
> 2009/3/31 Giovanni Messina <jgameit at gmail.com>
>
>> Using this post_process I lose the decimal places of which there are
>> suggested_retail_price in outfile
>>
>> ---------------
>> in the ctl file:
>>
>> post_process :bulk_import, {
>>    :file => outfile,
>>    :truncate => true,
>>    :columns => [:id,:description,:suggested_retail_price],
>>    :field_separator => "\t",
>>    :target => :development,
>>    :table => 'product_dimension'
>>  }
>>
>> -----------------
>>
>> the migration file is
>>
>> :id => :integer,
>> :description => :string,
>> :suggested_retail_price => :decimal
>>
>> -------------------
>>
>> how do I act?
>>
>> Giovanni
>>
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20090401/36cd9a53/attachment.html>

From thibaut.barrere at gmail.com  Wed Apr  1 07:14:38 2009
From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=)
Date: Wed, 1 Apr 2009 13:14:38 +0200
Subject: [Activewarehouse-discuss] truncate decimal?
In-Reply-To: <7c581bf40904010401s778094d3w139c36f9f2b14c67@mail.gmail.com>
References: <7c581bf40903310809g7d6895c4t9abc6cc2758a5ef5@mail.gmail.com> 
	<4a68b8cf0904010315r6830103uf6809a9198356918@mail.gmail.com> 
	<7c581bf40904010401s778094d3w139c36f9f2b14c67@mail.gmail.com>
Message-ID: <4a68b8cf0904010414s41a607aoe5710f1ef91623af@mail.gmail.com>

> with :float works even if I am surprised that round the value using
:decimal

I think you'll need to specify a scale and precision if you want to use
:decimal. Here's an example:
      t.decimal :latitude, :precision => 20, :scale => 15
      t.decimal :longitude, :precision => 20, :scale => 15

That's probably why it get rounded. See
http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/TableDefinition.htmlfor
the ranges of precision and scale depending on your DB.

cheers!

-- Thibaut

On Wed, Apr 1, 2009 at 1:01 PM, Giovanni Messina <jgameit at gmail.com> wrote:

> Hi, thanks for the suggestion
>
> with :float works even if I am surprised that round the value using
> :decimal
>
> cheers
>
> Giovanni
>
>
>
> 2009/4/1 Thibaut Barr?re <thibaut.barrere at gmail.com>
>
> Hello Giovanni,
>> it would be useful to know if the decimals are still there in the bulk
>> file (I suppose so) or if they are still there.
>>
>> If it can help you, in my cases I use :float in the migration (which is
>> good enough for me) and things seem to work fine.
>>
>> cheers,
>>
>> -- Thibaut
>>
>> 2009/3/31 Giovanni Messina <jgameit at gmail.com>
>>
>>> Using this post_process I lose the decimal places of which there are
>>> suggested_retail_price in outfile
>>>
>>> ---------------
>>> in the ctl file:
>>>
>>> post_process :bulk_import, {
>>>    :file => outfile,
>>>    :truncate => true,
>>>    :columns => [:id,:description,:suggested_retail_price],
>>>    :field_separator => "\t",
>>>    :target => :development,
>>>    :table => 'product_dimension'
>>>  }
>>>
>>> -----------------
>>>
>>> the migration file is
>>>
>>> :id => :integer,
>>> :description => :string,
>>> :suggested_retail_price => :decimal
>>>
>>> -------------------
>>>
>>> how do I act?
>>>
>>> Giovanni
>>>
>>> _______________________________________________
>>> Activewarehouse-discuss mailing list
>>> Activewarehouse-discuss at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20090401/228c2c56/attachment-0001.html>

From kookster at gmail.com  Wed Apr  1 11:26:02 2009
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Wed, 1 Apr 2009 11:26:02 -0400
Subject: [Activewarehouse-discuss] polymorphic associations
Message-ID: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>

Hey folks,

After a bit of a hiatus to deal with other things, I am back into my
AW implementation.

For one of the fact tables I want to create, it is based off an AR
model's table that uses a polymorphic relationship, so I am thinking
about how to handle this using AW ETL.

I'm thinking it would be best to denormalize this with a row
processor, and looking for advice on how best to do it.

To use my app as an example, we have an activity feed (think facebook
feed, yes, I know, not original) that has the general form:
":user does :action to :action_object"

The :user part is easy, this will be stored in the fact table as an id
relating to the user dimension.

The :action is a string value, so I'll probably create a dimension for
this with all possible values, so this too will end up a reference to
a dimension.
(probably use the foreign key transform for this, right?)

The :action_object is polymorphic, with 2 columns, :action_object_id,
and :action_object_type.

The row processor would:
First, get all possible values for 'action_object_type', e.g. 'Foo', and 'Bar'.
Map the type value to new columns per value, so 'foo_id' and 'bar_id'
in this case will need to be added to each row.
As each row is processed, if a row has the values
action_object_type='Foo', and action_object_id=1,
then the processor would add foo_id=1 and bar_id=nil.
The assumption here is that there are corresponding dimensions for Foo
and Bar to go with these new columns.

Does this make sense?
Has anyone else solved this a different way?
Anyone interested in the polymorphic processor code once I finish it?
(I am testing it now, but looking for validation before I move ahead
using it)

Cheers,

Andrew Kuklewicz

From kookster at gmail.com  Wed Apr  1 17:43:32 2009
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Wed, 1 Apr 2009 17:43:32 -0400
Subject: [Activewarehouse-discuss] polymorphic associations
In-Reply-To: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>
References: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>
Message-ID: <19ba6d7f0904011443i116e347cq1aa55ffdc2dfa70e@mail.gmail.com>

I got it working - attached is the code for it, as this may make more
sense than just my description.

But let me try to explain again anyway.

You have an ActiveRecord model with a polymorphic relationship, the
question is, how do you represent this in star schema?

The polymorphic relationship is represented as 2 columns - foo_id, foo_type.
For example: assume a row =
    {foo_id => 1, foo_type => 'User'}

Really, what I want to be able to do reporting and other good things
is to have a specific key to the user_dimension table, rather than
having to use both these columns as a key.  That is a fine rick for
ActiceRecord, but it's not exactly standard, and the class name
sitting in foo_type is pretty useless outside AR.

So that is what this row processor solves: it will add new a column
value to a row based on the polymorphic values.  Continuing the
example, after you run the processor on a row, you'll have the
following:
    {foo_id => 1, foo_type => 'User', :foo_user_id=>1}

Here it has added :foo_user_id => 1, which can now be used as a
foreign key to the user_dimension, and easily used with standard tools
and queries.

Is no one else using ActiveRecord models with polymorphic relationships?
Anyone who is - do you have a better solution? Or do you find this
denormalizing unnecessary.

Andrew Kuklewicz


On Wed, Apr 1, 2009 at 11:26 AM, Andrew Kuklewicz <kookster at gmail.com> wrote:
> Hey folks,
>
> After a bit of a hiatus to deal with other things, I am back into my
> AW implementation.
>
> For one of the fact tables I want to create, it is based off an AR
> model's table that uses a polymorphic relationship, so I am thinking
> about how to handle this using AW ETL.
>
> I'm thinking it would be best to denormalize this with a row
> processor, and looking for advice on how best to do it.
>
> To use my app as an example, we have an activity feed (think facebook
> feed, yes, I know, not original) that has the general form:
> ":user does :action to :action_object"
>
> The :user part is easy, this will be stored in the fact table as an id
> relating to the user dimension.
>
> The :action is a string value, so I'll probably create a dimension for
> this with all possible values, so this too will end up a reference to
> a dimension.
> (probably use the foreign key transform for this, right?)
>
> The :action_object is polymorphic, with 2 columns, :action_object_id,
> and :action_object_type.
>
> The row processor would:
> First, get all possible values for 'action_object_type', e.g. 'Foo', and 'Bar'.
> Map the type value to new columns per value, so 'foo_id' and 'bar_id'
> in this case will need to be added to each row.
> As each row is processed, if a row has the values
> action_object_type='Foo', and action_object_id=1,
> then the processor would add foo_id=1 and bar_id=nil.
> The assumption here is that there are corresponding dimensions for Foo
> and Bar to go with these new columns.
>
> Does this make sense?
> Has anyone else solved this a different way?
> Anyone interested in the polymorphic processor code once I finish it?
> (I am testing it now, but looking for validation before I move ahead
> using it)
>
> Cheers,
>
> Andrew Kuklewicz
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: polymorphic_processor.rb
Type: application/octet-stream
Size: 3371 bytes
Desc: not available
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20090401/3b36aea7/attachment.obj>

From kidpollo at gmail.com  Wed Apr  1 19:13:43 2009
From: kidpollo at gmail.com (Paco Viramontes)
Date: Wed, 1 Apr 2009 17:13:43 -0600
Subject: [Activewarehouse-discuss] polymorphic associations
In-Reply-To: <19ba6d7f0904011443i116e347cq1aa55ffdc2dfa70e@mail.gmail.com>
References: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>
	<19ba6d7f0904011443i116e347cq1aa55ffdc2dfa70e@mail.gmail.com>
Message-ID: <EC33C36F-E81A-4269-99E2-36D10E60B5C7@gmail.com>

I have used polymorphic associations in AR but for tables with very  
similar attributes

I get your point, but I cant understand whats the benefit for a AW  
table report. If you are generating some kind of custom report.

Or you are trying to use a fact class that can have only one dimension  
but of a different flavor.

What would then be a resulting report?

thanks for ecplaining

PAco

On Apr 1, 2009, at 3:43 PM, Andrew Kuklewicz wrote:

> add new a column
> value to a row based on the polymorphic values.  Co


From kookster at gmail.com  Wed Apr  1 22:08:03 2009
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Wed, 1 Apr 2009 22:08:03 -0400
Subject: [Activewarehouse-discuss] polymorphic associations
In-Reply-To: <EC33C36F-E81A-4269-99E2-36D10E60B5C7@gmail.com>
References: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>
	<19ba6d7f0904011443i116e347cq1aa55ffdc2dfa70e@mail.gmail.com>
	<EC33C36F-E81A-4269-99E2-36D10E60B5C7@gmail.com>
Message-ID: <19ba6d7f0904011908n2a168fedo2fab1780d6d8d92b@mail.gmail.com>

Hi Paco,

Not sure how else to explain it, if you haven't faced it maybe it
doesn't make sense.
I could leave it as a polymorphic relationship in the fact table I
guess, and adjust all queries to that, but I can't then use the AW
functionality to associate this info to dimensions, and can't then use
anything that depends on AW.

I don't think you will find any reference suggesting how AR does
polymorphic relationships as a good idea for how to model a fact
table's relationship to its dimensions.

As I understand it, the whole point of a star schema is very simple,
easily navigable relationships, as denormalized as you can get.
Ideally, you should simple fk references between a dimension and a
fact, and all other fields are aggregate data.  The polymorphic style
of expressing relationships to dimensions does not do this.

more below....


On Wed, Apr 1, 2009 at 7:13 PM, Paco Viramontes <kidpollo at gmail.com> wrote:
> I have used polymorphic associations in AR but for tables with very similar
> attributes

I am happy to use in AR as well, the issue is how to do the same thing
in a fact table, and if the soltion that works so well in AR is a good
answer in a start schema db.


> I get your point, but I cant understand whats the benefit for a AW table
> report. If you are generating some kind of custom report.

I'm not saying I can't work around it, and write reports if I have to,
more I am looking for a design that consistent with good star schema
and data warehousing design principles, and that plays well with the
existing AW code.  For example, I don't think the dimension method in
a fact class handles polymorphic relationships.  Same with using these
values to define a cube, not sure, but pretty cure it would make it
much more difficult if it could be made to work at all.

It is my uncertainty at how well AW supports polymorphic relationships
that lead me to write in, as I could be wrong, maybe it is well
supported and I missed it in the docs/code.

> Or you are trying to use a fact class that can have only one dimension but
> of a different flavor.

There are about 7 different dimensions that a given fact row could be
related to, all of which is contained in the polymorphic relationship
as a single key and type column pair in AR, so this processor is a way
to have the fact show all those possible relationships in its
dimension columns.

> What would then be a resulting report?

Well, imagine that I wanted a report of every time a user made another
user a contact.
After running the polymorphic processor, all I need to do is aggregate
using the action_object_user_id table for the user being contacted.
With the polymorphic relationship I need to know that the
action_object_type value is 'User', and have a where clause to include
only those rows, so I am always adding these kinds of conditions to my
queries and reports, and expecting people to know to do this.

> thanks for ecplaining

Trying to explain, but I am mostly looking to see if there even is a
problem, or if I am imagining it.

Between this and the ActiveMessaging list, you and I are getting to be
real pen pals. :)

Cheers,
-Andrew

> PAco
>
> On Apr 1, 2009, at 3:43 PM, Andrew Kuklewicz wrote:
>
>> add new a column
>> value to a row based on the polymorphic values. ?Co
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>

From kidpollo at gmail.com  Thu Apr  2 01:44:52 2009
From: kidpollo at gmail.com (Paco Viramontes)
Date: Wed, 1 Apr 2009 23:44:52 -0600
Subject: [Activewarehouse-discuss] polymorphic associations
In-Reply-To: <19ba6d7f0904011908n2a168fedo2fab1780d6d8d92b@mail.gmail.com>
References: <19ba6d7f0904010826h278d2218oc1933378b32663df@mail.gmail.com>
	<19ba6d7f0904011443i116e347cq1aa55ffdc2dfa70e@mail.gmail.com>
	<EC33C36F-E81A-4269-99E2-36D10E60B5C7@gmail.com>
	<19ba6d7f0904011908n2a168fedo2fab1780d6d8d92b@mail.gmail.com>
Message-ID: <B6F8B839-6F2A-46C8-8C27-654530C977B8@gmail.com>

YeaH you are right, its like we run into similar things :D

Any how I cannot answer your question, now I understand you and I dont  
think there is a right answer, it looks fine for me actually now I get  
it. It seems very ruby -like solution according to what I undestand to  
be a star schema, yet I ask so much because I am not a Wareousing  
expert may be some one with more Data Warehousing knowledge will  
eventually read this hahaha and give his insight

Thanks again, I also find myself patching or customizing AW, but I am  
never sure if what I do makes sense to anyone else or how to extend  
the api because again my changes seem very custom, personally I would  
like a more detailed roadmap and ROADMAP RECMENDATIONS from people in  
the list.

Cheers

PAco

On Apr 1, 2009, at 8:08 PM, Andrew Kuklewicz wrote:

> Hi Paco,
>
> Not sure how else to explain it, if you haven't faced it maybe it
> doesn't make sense.
> I could leave it as a polymorphic relationship in the fact table I
> guess, and adjust all queries to that, but I can't then use the AW
> functionality to associate this info to dimensions, and can't then use
> anything that depends on AW.
>
> I don't think you will find any reference suggesting how AR does
> polymorphic relationships as a good idea for how to model a fact
> table's relationship to its dimensions.
>
> As I understand it, the whole point of a star schema is very simple,
> easily navigable relationships, as denormalized as you can get.
> Ideally, you should simple fk references between a dimension and a
> fact, and all other fields are aggregate data.  The polymorphic style
> of expressing relationships to dimensions does not do this.
>
> more below....
>
>
> On Wed, Apr 1, 2009 at 7:13 PM, Paco Viramontes <kidpollo at gmail.com>  
> wrote:
>> I have used polymorphic associations in AR but for tables with very  
>> similar
>> attributes
>
> I am happy to use in AR as well, the issue is how to do the same thing
> in a fact table, and if the soltion that works so well in AR is a good
> answer in a start schema db.
>
>
>> I get your point, but I cant understand whats the benefit for a AW  
>> table
>> report. If you are generating some kind of custom report.
>
> I'm not saying I can't work around it, and write reports if I have to,
> more I am looking for a design that consistent with good star schema
> and data warehousing design principles, and that plays well with the
> existing AW code.  For example, I don't think the dimension method in
> a fact class handles polymorphic relationships.  Same with using these
> values to define a cube, not sure, but pretty cure it would make it
> much more difficult if it could be made to work at all.
>
> It is my uncertainty at how well AW supports polymorphic relationships
> that lead me to write in, as I could be wrong, maybe it is well
> supported and I missed it in the docs/code.
>
>> Or you are trying to use a fact class that can have only one  
>> dimension but
>> of a different flavor.
>
> There are about 7 different dimensions that a given fact row could be
> related to, all of which is contained in the polymorphic relationship
> as a single key and type column pair in AR, so this processor is a way
> to have the fact show all those possible relationships in its
> dimension columns.
>
>> What would then be a resulting report?
>
> Well, imagine that I wanted a report of every time a user made another
> user a contact.
> After running the polymorphic processor, all I need to do is aggregate
> using the action_object_user_id table for the user being contacted.
> With the polymorphic relationship I need to know that the
> action_object_type value is 'User', and have a where clause to include
> only those rows, so I am always adding these kinds of conditions to my
> queries and reports, and expecting people to know to do this.
>
>> thanks for ecplaining
>
> Trying to explain, but I am mostly looking to see if there even is a
> problem, or if I am imagining it.
>
> Between this and the ActiveMessaging list, you and I are getting to be
> real pen pals. :)
>
> Cheers,
> -Andrew
>
>> PAco
>>
>> On Apr 1, 2009, at 3:43 PM, Andrew Kuklewicz wrote:
>>
>>> add new a column
>>> value to a row based on the polymorphic values.  Co
>>
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>


From kidpollo at gmail.com  Fri Apr  3 02:41:45 2009
From: kidpollo at gmail.com (Paco Viramontes)
Date: Fri, 3 Apr 2009 00:41:45 -0600
Subject: [Activewarehouse-discuss] change calculated field
Message-ID: <051F3FEB-3DB4-4E1F-9F87-2D735AD70E66@gmail.com>

Hi!

I am trying to figure out what would be the best way to calculate the  
difference between the result of two different time periods

Suppose you have a table report that shows amount of sales in January  
2008 compared against the number of sales in January 2007

Any ideas to implement this in AW?

Thanks

PAco

From kookster at gmail.com  Tue Apr  7 14:29:51 2009
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Tue, 7 Apr 2009 14:29:51 -0400
Subject: [Activewarehouse-discuss] rolap aggregate and pipelined rollup
	aggregate?
Message-ID: <19ba6d7f0904071129n4685fc45w6c4efbe4f424625c@mail.gmail.com>

Hey folks,

Looking through some old code, I found reference to _rollup tables,
which lead me to find the old rolap aggregate implementations,
rolap_aggregate.rb and pipelined_rolap_aggregate.rb.
I see it was removed from the source tree awhile back (May 2007):
http://www.codenotifier.com/projects/249/commits/758

the pipelined_rolap_aggregate.rb looks like exactly what I was looking
for, so I have to ask why this was removed, and if there is something
wrong with it other than not being well maintained since 2007?

In the meantime, I have working (but much less reusable) code that is
generating rollups for me, but this seems like a better way to go
about it, so I'm wondering if maybe I should try to use what I have
learned creating my own rollups to update this aggregate strategy.

Cheers,
Andrew Kuklewicz

From kookster at gmail.com  Tue Apr 21 13:14:24 2009
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Tue, 21 Apr 2009 13:14:24 -0400
Subject: [Activewarehouse-discuss] Need some help on setting up a
	datawarehouse
In-Reply-To: <8f8ea7d0904202348o7159457bu120d785f92e7e61@mail.gmail.com>
References: <8f8ea7d0904202348o7159457bu120d785f92e7e61@mail.gmail.com>
Message-ID: <19ba6d7f0904211014y3d31c3e5rfc4b99cacffeb25d@mail.gmail.com>

> Now I need to populate the watse_facts table. I am not sure how to go about
> it. How do I add waste_id which is a foreign key from waste_dimesions table.

There are a couple ways to do this I have found.

> Also I wanted to add additional information from wastes table to waste_facts
> table like co2e_factor (which I have not included in waste_dimensions table)

I think I can guess what you are trying to do here.  You have one
table, wastes, which combines some general information about a waste,
such as the name and type, and then more specific facts, such as the
co2e_factor, where you might have many rows with the same waste name
and type_id (and other repeated information), but with a few fields
that vary.  The common fields, you want to group and make a
waste_dimension, and the fields that vary you want to then make into
the fact table's values.

For the waste_dimension, that means you need to be doing a group by,
and probably generating a new key for each waste row, or using type_id
if there is a type per waste dimension row.

After that, when you create the fact table, you need to have some
unique value to lookup in the waste dimension (type_id?), and use a
transform in the fact etl script o get the id.

For example, if you want to lookup the date_id to link to a
date_dimension, you can use a date value in the fact data to lookup
the right date_dimension row by comparing with the date value in the
date_dimension. Here is an example of doing this using a transform in
an etl script:

transform :date_id, :foreign_key_lookup, { :resolver =>
SQLResolver.new('date_dimension', 'sql_date_stamp', 'foo_warehouse') }

Andrew Kuklewicz


On Tue, Apr 21, 2009 at 2:48 AM, Shikha Mohan <shikhamohan09 at gmail.com> wrote:
> Hi,
>
> I am new to this datawarehouse concept. I am trying to create a dw. I have
> one dimension and one fact, namely: waste_dimension.rb, waste_fact.rb
>
> Location of elt is db/elt
>
> My database.yml is something like:
>
> etl_execution:
> ? adapter: mysql
> ? database: rails_warehouse_etl_execution
> ? username: root
> ? password: password
> ? host: localhost
>
> warehouse:
> ? adapter: mysql
> ? database: for_demo
> ? username: root
> ? password: password
> ? host: localhost
>
> ?I am using database for extract and load.
>
> ?I am able to create a simple dimension table using following code:
>
> source :in, {
> ??? :database => "for_demo",
> ??? :target => :warehouse,
> ??? :table => "wastes",
> ??? :select => "wastes.name, wastes.type_id",
> ??? :order => "wastes.name"
> },
> [
> ??? :name,
> ??? :type_id
> ]
>
> transform :name, :default, :default_value => "No Name"
>
>
> destination :out, {
> ??? :database => "rails_warehouse_etl_execution",
> ??? :target => :etl_execution,
> ??? :table => 'waste_dimension'
> },
> {
> ??? :order => [
> ??? ??? :name,
> ??? ??? :type_id
> ??? ]
> }
>
> Now I need to populate the watse_facts table. I am not sure how to go about
> it. How do I add waste_id which is a foreign key from waste_dimesions table.
> Also I wanted to add additional information from wastes table to waste_facts
> table like co2e_factor (which I have not included in waste_dimensions table)
>
> Structure for waste_dimension and waste_facts table is as follows:
>
> class CreateWasteDimension < ActiveRecord::Migration
> ? def self.up
> ??? fields = {
> ????? :name => :string,
> ????? :type_id => :integer
> ??? }
> ??? create_table :waste_dimension do |t|
> ????? fields.each do |name,type|
> ??????? t.column name, type
> ????? end
> ??? end
> ??? fields.each do |name,type|
> ????? add_index :waste_dimension, name unless type == :text
> ??? end
> ? end
>
> ? def self.down
> ??? drop_table :waste_dimension
> ? end
> end
>
>
> class CreateWasteFacts < ActiveRecord::Migration
> ? def self.up
> ??? # you should add indexes for each foreign key, but don't add
> ??? # the foreign key itself unless you really know what you are doing.
> ??? create_table :waste_facts do |t|
> ????? t.column :waste_id, :integer, :null => false
> ????? t.column :co2e_factor, :integer, :default => 1
> ??? end
> ??? add_index :waste_facts, :waste_id
> ? end
>
> ? def self.down
> ??? drop_table :waste_facts
> ? end
> end
>
> As I could not get any information on net hence mailing on this list. Please
> help me resolve this problem. Thanks!
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>

From btelles at gmail.com  Tue Apr 21 18:18:16 2009
From: btelles at gmail.com (Bernardo Telles)
Date: Tue, 21 Apr 2009 18:18:16 -0400
Subject: [Activewarehouse-discuss] Adding and removing fields with
	activewarehouse-etl
Message-ID: <189177b20904211518y4a014f88kda2917c9832c9d52@mail.gmail.com>

Hi there,
I'm pretty new to activewarehouse, and was wondering what techniques are
used to add and remove fields from the destination file. Right now it
appears I have to delete fields by omitting them from the bulk import
section, and add them by either renaming them on the fly or copying them.
(Please let me know if this is not what activewarehouse is for :-).

Here's the goal:
take a list of employees and their phone numbers, and place each phone
number on a different row with a corresponding phone number type like so:

original format and sample row:
=> id, first_name, last_name, phone, fax
=>10,ben,tom,4444444,5555555

final format and TWO rows:
=>id,number,phone_category
=>1,4444444,phone
=>2,5555555,fax

Notice I removed the first and last name columns...here are the beginnings
of a row_processor (clearly written incorrectly or against conventions):
      def process(row)
        bp = row[:business_phone]
        fp = row[:fax_phone]
        rows = []
        biz = {:business_phone => bp, :first_name => 'biz'}
        fax = {:business_phone => fp, :first_name => 'fax'}
        rows =[biz,fax]
      end
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20090421/ddfb6ceb/attachment.html>