From dbitla at gmail.com  Wed Oct  1 16:21:46 2008
From: dbitla at gmail.com (Dash Bitla)
Date: Thu, 2 Oct 2008 01:51:46 +0530
Subject: [Activewarehouse-discuss] Welcome to the
	"Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
References: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
Message-ID: <9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>

Hi All,

I would like to know if this plugin is being used in any of the production
online BI products already - can you please pass on some info.
I just want to know how actively this plugin is being maintained and pain
points, issues etc.
We are planning to use this for one of our product - and want to see how I
can help enhance the product as we use it.
Any responses would be great!

Best Regards
Dash
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20081002/4fbf1bd4/attachment.html>

From mghaught at gmail.com  Wed Oct  1 18:32:53 2008
From: mghaught at gmail.com (Marty Haught)
Date: Wed, 1 Oct 2008 16:32:53 -0600
Subject: [Activewarehouse-discuss] Welcome to the
	"Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
References: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
	<9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
Message-ID: <57f29e620810011532o50c89ddbx98e0e770b02e3b5d@mail.gmail.com>

Hi Dash,

We have been using ActiveWarehouse for nearly a year in production
with no issues.  The plug-in is not yet Rails 2.1 compliant but I
believe a couple of us are looking into it.  Development isn't
terribly active but as it works for most of us, I'm not sure it's an
issue.  Does that give you enough information?

Cheers,
Marty

On Wed, Oct 1, 2008 at 2:21 PM, Dash Bitla <dbitla at gmail.com> wrote:
> Hi All,
> I would like to know if this plugin is being used in any of the production
> online BI products already - can you please pass on some info.
> I just want to know how actively this plugin is being maintained and pain
> points, issues etc.
> We are planning to use this for one of our product - and want to see how I
> can help enhance the product as we use it.
> Any responses would be great!
> Best Regards
> Dash
>
>
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>

From ribeiro.luis at gmail.com  Wed Oct  1 19:26:09 2008
From: ribeiro.luis at gmail.com (Luis Landeiro Ribeiro)
Date: Thu, 2 Oct 2008 00:26:09 +0100
Subject: [Activewarehouse-discuss] Welcome to the
	"Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <57f29e620810011532o50c89ddbx98e0e770b02e3b5d@mail.gmail.com>
References: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
	<9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
	<57f29e620810011532o50c89ddbx98e0e770b02e3b5d@mail.gmail.com>
Message-ID: <70e02b660810011626t67ec7b41oc36ff5365a03e017@mail.gmail.com>

I for one, would like to see a live application, to know how this plug-in
can empower a dw application.

Cheers,
Luis Landeiro

On Wed, Oct 1, 2008 at 11:32 PM, Marty Haught <mghaught at gmail.com> wrote:

> Hi Dash,
>
> We have been using ActiveWarehouse for nearly a year in production
> with no issues.  The plug-in is not yet Rails 2.1 compliant but I
> believe a couple of us are looking into it.  Development isn't
> terribly active but as it works for most of us, I'm not sure it's an
> issue.  Does that give you enough information?
>
> Cheers,
> Marty
>
> On Wed, Oct 1, 2008 at 2:21 PM, Dash Bitla <dbitla at gmail.com> wrote:
> > Hi All,
> > I would like to know if this plugin is being used in any of the
> production
> > online BI products already - can you please pass on some info.
> > I just want to know how actively this plugin is being maintained and pain
> > points, issues etc.
> > We are planning to use this for one of our product - and want to see how
> I
> > can help enhance the product as we use it.
> > Any responses would be great!
> > Best Regards
> > Dash
> >
> >
> >
> > _______________________________________________
> > Activewarehouse-discuss mailing list
> > Activewarehouse-discuss at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
> >
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20081002/85fd616d/attachment.html>

From thibaut.barrere at gmail.com  Thu Oct  2 01:10:29 2008
From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=)
Date: Thu, 2 Oct 2008 07:10:29 +0200
Subject: [Activewarehouse-discuss] Welcome to the
	"Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
References: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
	<9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
Message-ID: <4a68b8cf0810012210n469bb876r9e1b35037ba21d56@mail.gmail.com>

Hi,

Same answer here - I'm using the ETL part on an internal datawarehouse
for one of my customers. It has been in production for a year without
any issue; evolutions have been easy to do thanks to ruby hackability.

hope this helps

Thibaut Barr?re
--
LoGeek
[blog] http://evolvingworker.com - tools for a better day
[blog] http://blog.logeek.fr - about writing software


On Wed, Oct 1, 2008 at 10:21 PM, Dash Bitla <dbitla at gmail.com> wrote:
> Hi All,
> I would like to know if this plugin is being used in any of the production
> online BI products already - can you please pass on some info.
> I just want to know how actively this plugin is being maintained and pain
> points, issues etc.
> We are planning to use this for one of our product - and want to see how I
> can help enhance the product as we use it.
> Any responses would be great!
> Best Regards
> Dash
>
>
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>

From dwlist at stellarmadness.org  Thu Oct  2 02:25:29 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Wed, 01 Oct 2008 23:25:29 -0700
Subject: [Activewarehouse-discuss] Welcome to the
 "Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <4a68b8cf0810012210n469bb876r9e1b35037ba21d56@mail.gmail.com>
Message-ID: <C509B769.16DC%dwlist@stellarmadness.org>

+1

We've also been using the ETL portion for a very intense nightly load. It's
been rock solid, and easy to modify when needed.

Darin


> From: Thibaut Barr?re <thibaut.barrere at gmail.com>
> Date: Thu, 2 Oct 2008 07:10:29 +0200
> To: Dash Bitla <dbitla at gmail.com>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] Welcome to the
> "Activewarehouse-discuss" mailing list (Digest mode)
> 
> Hi,
> 
> Same answer here - I'm using the ETL part on an internal datawarehouse
> for one of my customers. It has been in production for a year without
> any issue; evolutions have been easy to do thanks to ruby hackability.
> 
> hope this helps
> 
> Thibaut Barr?re
> --
> LoGeek
> [blog] http://evolvingworker.com - tools for a better day
> [blog] http://blog.logeek.fr - about writing software
> 
> 
> On Wed, Oct 1, 2008 at 10:21 PM, Dash Bitla <dbitla at gmail.com> wrote:
>> Hi All,
>> I would like to know if this plugin is being used in any of the production
>> online BI products already - can you please pass on some info.
>> I just want to know how actively this plugin is being maintained and pain
>> points, issues etc.
>> We are planning to use this for one of our product - and want to see how I
>> can help enhance the product as we use it.
>> Any responses would be great!
>> Best Regards
>> Dash
>> 
>> 
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 
>> 
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss



From dbitla at gmail.com  Thu Oct  2 11:08:39 2008
From: dbitla at gmail.com (Dash Bitla)
Date: Thu, 2 Oct 2008 20:38:39 +0530
Subject: [Activewarehouse-discuss] Welcome to the
	"Activewarehouse-discuss" mailing list (Digest mode)
In-Reply-To: <57f29e620810011532o50c89ddbx98e0e770b02e3b5d@mail.gmail.com>
References: <mailman.0.1222806351.21541.activewarehouse-discuss@rubyforge.org>
	<9d2029cf0810011321v683fdcc9ne9dccb86d889762d@mail.gmail.com>
	<57f29e620810011532o50c89ddbx98e0e770b02e3b5d@mail.gmail.com>
Message-ID: <9d2029cf0810020808v15373e58hd34d39c8468a0d56@mail.gmail.com>

Thanks for writing on this.What type of database to be used for better
performance with ActiveWarehouse? I mean with MySQL - which type InnoDB,
Infobright, or MyISAM is better?
Any bench marks on how much of data can it handle - I mean can it handle
50GB of data ?
I'll share my experience as I start implementing it for our product.
Any other alternatives to this?

Dash


On Thu, Oct 2, 2008 at 4:02 AM, Marty Haught <mghaught at gmail.com> wrote:

> Hi Dash,
>
> We have been using ActiveWarehouse for nearly a year in production
> with no issues.  The plug-in is not yet Rails 2.1 compliant but I
> believe a couple of us are looking into it.  Development isn't
> terribly active but as it works for most of us, I'm not sure it's an
> issue.  Does that give you enough information?
>
> Cheers,
> Marty
>
> On Wed, Oct 1, 2008 at 2:21 PM, Dash Bitla <dbitla at gmail.com> wrote:
> > Hi All,
> > I would like to know if this plugin is being used in any of the
> production
> > online BI products already - can you please pass on some info.
> > I just want to know how actively this plugin is being maintained and pain
> > points, issues etc.
> > We are planning to use this for one of our product - and want to see how
> I
> > can help enhance the product as we use it.
> > Any responses would be great!
> > Best Regards
> > Dash
> >
> >
> >
> > _______________________________________________
> > Activewarehouse-discuss mailing list
> > Activewarehouse-discuss at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20081002/38fe7a5d/attachment.html>

From dwlist at stellarmadness.org  Tue Oct 14 16:58:47 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Tue, 14 Oct 2008 13:58:47 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand up?
	:)
In-Reply-To: <9d2029cf0810020808v15373e58hd34d39c8468a0d56@mail.gmail.com>
Message-ID: <C51A5617.19FD%dwlist@stellarmadness.org>

Hi all-

I found an old message from Anthony announcing the move to github, and he
offered this url as the new master:

http://github.com/aeden/activewarehouse/tree/master

Is this still the best place to start from if I wanted to do some hacking? I
know that other trees have popped up and I wasn?t sure if it would be better
to start from one of those.

Thanks,
Darin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20081014/46a9bfe9/attachment.html>

From jmaine at blurb.com  Wed Oct 15 13:21:57 2008
From: jmaine at blurb.com (Jacob Maine)
Date: Wed, 15 Oct 2008 10:21:57 -0700 (PDT)
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <31318923.53911224090779919.JavaMail.root@kilgore.blurb.com>
Message-ID: <27155736.54021224091317717.JavaMail.root@kilgore.blurb.com>

> Is this still the best place to start?

I think the github respository is the best place to start.  That's where active development happens, so it's more volatile, but also more up-to-date.  For production systems, you can peg to a certain commit, and for development you can easily get the latest changes.  As you said, there are a few different trees, but I believe github is where they'll eventually come together.

From mghaught at gmail.com  Thu Oct 16 01:43:43 2008
From: mghaught at gmail.com (Marty Haught)
Date: Wed, 15 Oct 2008 23:43:43 -0600
Subject: [Activewarehouse-discuss] would the real codebase please stand
	up? :)
In-Reply-To: <C51A5617.19FD%dwlist@stellarmadness.org>
References: <9d2029cf0810020808v15373e58hd34d39c8468a0d56@mail.gmail.com>
	<C51A5617.19FD%dwlist@stellarmadness.org>
Message-ID: <57f29e620810152243l2d90dd29i92ba01344bac4068@mail.gmail.com>

> I found an old message from Anthony announcing the move to github, and he
> offered this url as the new master:
>
> http://github.com/aeden/activewarehouse/tree/master
>
> Is this still the best place to start from if I wanted to do some hacking? I
> know that other trees have popped up and I wasn't sure if it would be better
> to start from one of those.

I would go off this repo for now.  I don't know if anyone else has a
more active fork than Anthony's.  I will probably have some commits in
the next month or so that I'll give back.  We haven't identified when
we'll upgrade our app to 2.1 but if we do so shortly then I'll have
code to upgrade the plugin.  If anyone else does this before us,
please post to the list.

Cheers,
Marty

From anthonyeden at gmail.com  Thu Oct 16 06:59:41 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Thu, 16 Oct 2008 06:59:41 -0400
Subject: [Activewarehouse-discuss] would the real codebase please stand
	up? :)
In-Reply-To: <57f29e620810152243l2d90dd29i92ba01344bac4068@mail.gmail.com>
References: <9d2029cf0810020808v15373e58hd34d39c8468a0d56@mail.gmail.com>
	<C51A5617.19FD%dwlist@stellarmadness.org>
	<57f29e620810152243l2d90dd29i92ba01344bac4068@mail.gmail.com>
Message-ID: <bd41ed320810160359y3e6a57d8s573b196d864cfe2f@mail.gmail.com>

On Thu, Oct 16, 2008 at 1:43 AM, Marty Haught <mghaught at gmail.com> wrote:
>> I found an old message from Anthony announcing the move to github, and he
>> offered this url as the new master:
>>
>> http://github.com/aeden/activewarehouse/tree/master
>>
>> Is this still the best place to start from if I wanted to do some hacking? I
>> know that other trees have popped up and I wasn't sure if it would be better
>> to start from one of those.
>
> I would go off this repo for now.  I don't know if anyone else has a
> more active fork than Anthony's.  I will probably have some commits in
> the next month or so that I'll give back.  We haven't identified when
> we'll upgrade our app to 2.1 but if we do so shortly then I'll have
> code to upgrade the plugin.  If anyone else does this before us,
> please post to the list.

I've started the upgrade to 2.1. I've already modified ActiveWarehouse
ETL to handle the new migration process and I have it running right
now under 2.1. Once I get past my first ETL load on the project I'm
working on I'll move onto the AW plugin.

-Anthony

From dwlist at stellarmadness.org  Thu Oct 16 12:11:10 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Thu, 16 Oct 2008 09:11:10 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <bd41ed320810160359y3e6a57d8s573b196d864cfe2f@mail.gmail.com>
Message-ID: <C51CB5AE.1A95%dwlist@stellarmadness.org>

Cool - that is what I was going to start working on. :)

Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
try it out with our ETL (and I have a bunch of test cases for it).

Thanks!
Darin

> From: Anthony Eden <anthonyeden at gmail.com>
> Date: Thu, 16 Oct 2008 06:59:41 -0400
> To: Marty Haught <mghaught at gmail.com>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
> ETL to handle the new migration process and I have it running right
> now under 2.1. Once I get past my first ETL load on the project I'm
> working on I'll move onto the AW plugin.
> 
> -Anthony
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss



From anthonyeden at gmail.com  Thu Oct 16 15:23:04 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Thu, 16 Oct 2008 15:23:04 -0400
Subject: [Activewarehouse-discuss] would the real codebase please stand
	up? :)
In-Reply-To: <C51CB5AE.1A95%dwlist@stellarmadness.org>
References: <bd41ed320810160359y3e6a57d8s573b196d864cfe2f@mail.gmail.com>
	<C51CB5AE.1A95%dwlist@stellarmadness.org>
Message-ID: <bd41ed320810161223u71b6494bx5df65059e689a5c9@mail.gmail.com>

I think I have most of the fixes to ETL in for the 2.1 changes. I've
started changing over some of the tests to use shoulda 1.x, so if you
want to run the built in tests then you'll need the shoulda gem. There
are currently two failing tests related to the SCD implementation.
Jacob, was that an area you worked on? If so could you take a look at
the failing tests? Also, I currently have tested with Postgres locally
but not MySQL, so if someone wants to fight through the test suite on
MySQL, be my guest. :-)

Thanks.

-Anthony

On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
<dwlist at stellarmadness.org> wrote:
> Cool - that is what I was going to start working on. :)
>
> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
> try it out with our ETL (and I have a bunch of test cases for it).
>
> Thanks!
> Darin
>
>> From: Anthony Eden <anthonyeden at gmail.com>
>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>> To: Marty Haught <mghaught at gmail.com>
>> Cc: <activewarehouse-discuss at rubyforge.org>
>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>> up? :)
>>
>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>> ETL to handle the new migration process and I have it running right
>> now under 2.1. Once I get past my first ETL load on the project I'm
>> working on I'll move onto the AW plugin.
>>
>> -Anthony
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>

From dwlist at stellarmadness.org  Thu Oct 16 22:14:21 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Thu, 16 Oct 2008 19:14:21 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <bd41ed320810161223u71b6494bx5df65059e689a5c9@mail.gmail.com>
Message-ID: <C51D430D.1AB3%dwlist@stellarmadness.org>

I just pulled the latest from your github repo and ran the ETL test suite
locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
those were the same ones you mentioned in your last message(?).

There was one other error:

  1) Failure:
test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
<ActiveRecord::StatementInvalid> exception expected but none was thrown.

It turns out that MySQL is much less strict about what goes into the bulk
loader - if you try to load a row with an empty value into a column that
disallows null, MySQL just skips it without an error. Nice! :)

I'm not sure how you'd like to handle that. I tried poking around a bit to
see if there's a way to force MySQL into being more strict, but I couldn't
get anything to work reliably. The simplest fix would be to do something
like this in the test:

    unless ETL::Engine.connection(:data_warehouse).class ==
ActiveRecord::ConnectionAdapters::MysqlAdapter
      assert_raise(ActiveRecord::StatementInvalid) {
do_bulk_import('bulk_import_with_empties.txt') }
    end

which sort of invalidates the test, but it doesn't seem like there's a way
to make this test actually work with MySQL. Your call. ;)

Darin


> From: Anthony Eden <anthonyeden at gmail.com>
> Date: Thu, 16 Oct 2008 15:23:04 -0400
> To: Darin Wilson <dwlist at stellarmadness.org>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> I think I have most of the fixes to ETL in for the 2.1 changes. I've
> started changing over some of the tests to use shoulda 1.x, so if you
> want to run the built in tests then you'll need the shoulda gem. There
> are currently two failing tests related to the SCD implementation.
> Jacob, was that an area you worked on? If so could you take a look at
> the failing tests? Also, I currently have tested with Postgres locally
> but not MySQL, so if someone wants to fight through the test suite on
> MySQL, be my guest. :-)
> 
> Thanks.
> 
> -Anthony
> 
> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
> <dwlist at stellarmadness.org> wrote:
>> Cool - that is what I was going to start working on. :)
>> 
>> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>> try it out with our ETL (and I have a bunch of test cases for it).
>> 
>> Thanks!
>> Darin
>> 
>>> From: Anthony Eden <anthonyeden at gmail.com>
>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>> To: Marty Haught <mghaught at gmail.com>
>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>>> up? :)
>>> 
>>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>>> ETL to handle the new migration process and I have it running right
>>> now under 2.1. Once I get past my first ETL load on the project I'm
>>> working on I'll move onto the AW plugin.
>>> 
>>> -Anthony
>>> _______________________________________________
>>> Activewarehouse-discuss mailing list
>>> Activewarehouse-discuss at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 



From anthonyeden at gmail.com  Fri Oct 17 07:42:55 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Fri, 17 Oct 2008 07:42:55 -0400
Subject: [Activewarehouse-discuss] would the real codebase please stand
	up? :)
In-Reply-To: <C51D430D.1AB3%dwlist@stellarmadness.org>
References: <bd41ed320810161223u71b6494bx5df65059e689a5c9@mail.gmail.com>
	<C51D430D.1AB3%dwlist@stellarmadness.org>
Message-ID: <bd41ed320810170442y71b71435w1b35e630da880eee@mail.gmail.com>

I'm seeing even more errors with MySQL. A lot of them look like this:

Mysql::Error: #42000The used command is not allowed with this MySQL
version: LOAD DATA LOCAL INFILE './test/output/scd_test_type_2.txt'
INTO TABLE person_dimension FIELDS TERMINATED BY ','

What are the errors you are seeing when you run against MySQL?

-A

On Thu, Oct 16, 2008 at 10:14 PM, Darin Wilson
<dwlist at stellarmadness.org> wrote:
> I just pulled the latest from your github repo and ran the ETL test suite
> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
> those were the same ones you mentioned in your last message(?).
>
> There was one other error:
>
>  1) Failure:
> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
>
> It turns out that MySQL is much less strict about what goes into the bulk
> loader - if you try to load a row with an empty value into a column that
> disallows null, MySQL just skips it without an error. Nice! :)
>
> I'm not sure how you'd like to handle that. I tried poking around a bit to
> see if there's a way to force MySQL into being more strict, but I couldn't
> get anything to work reliably. The simplest fix would be to do something
> like this in the test:
>
>    unless ETL::Engine.connection(:data_warehouse).class ==
> ActiveRecord::ConnectionAdapters::MysqlAdapter
>      assert_raise(ActiveRecord::StatementInvalid) {
> do_bulk_import('bulk_import_with_empties.txt') }
>    end
>
> which sort of invalidates the test, but it doesn't seem like there's a way
> to make this test actually work with MySQL. Your call. ;)
>
> Darin
>
>
>> From: Anthony Eden <anthonyeden at gmail.com>
>> Date: Thu, 16 Oct 2008 15:23:04 -0400
>> To: Darin Wilson <dwlist at stellarmadness.org>
>> Cc: <activewarehouse-discuss at rubyforge.org>
>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>> up? :)
>>
>> I think I have most of the fixes to ETL in for the 2.1 changes. I've
>> started changing over some of the tests to use shoulda 1.x, so if you
>> want to run the built in tests then you'll need the shoulda gem. There
>> are currently two failing tests related to the SCD implementation.
>> Jacob, was that an area you worked on? If so could you take a look at
>> the failing tests? Also, I currently have tested with Postgres locally
>> but not MySQL, so if someone wants to fight through the test suite on
>> MySQL, be my guest. :-)
>>
>> Thanks.
>>
>> -Anthony
>>
>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>> <dwlist at stellarmadness.org> wrote:
>>> Cool - that is what I was going to start working on. :)
>>>
>>> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>>> try it out with our ETL (and I have a bunch of test cases for it).
>>>
>>> Thanks!
>>> Darin
>>>
>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>>> To: Marty Haught <mghaught at gmail.com>
>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>>>> up? :)
>>>>
>>>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>>>> ETL to handle the new migration process and I have it running right
>>>> now under 2.1. Once I get past my first ETL load on the project I'm
>>>> working on I'll move onto the AW plugin.
>>>>
>>>> -Anthony
>>>> _______________________________________________
>>>> Activewarehouse-discuss mailing list
>>>> Activewarehouse-discuss at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>
>>>
>>> _______________________________________________
>>> Activewarehouse-discuss mailing list
>>> Activewarehouse-discuss at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>
>
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>

From dwlist at stellarmadness.org  Fri Oct 17 10:08:42 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Fri, 17 Oct 2008 07:08:42 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <bd41ed320810170442y71b71435w1b35e630da880eee@mail.gmail.com>
Message-ID: <C51DEA7A.1AE3%dwlist@stellarmadness.org>

The error you're seeing means you need to apply the
mysql_adapter_opt_local_infile.patch in AdapterExtensions. If you've already
done that, then it's possible that the patch needs to be tweaked slightly
for Rails 2.1 (but I don't think this is the case). In any event, the
connect method in MysqlAdapter needs to have this line:

@connection.options(Mysql::OPT_LOCAL_INFILE, true)

(BTW, I don't quite get why the patch is necessary. It seems like this could
be handled entirely within AdapterExtensions itself. I'll look into this.)

The rest of the errors I'm seeing look roughly like this one:

  2) Failure:
test: when working with a slowly changing dimension of type 1 on run 2
should skip the load if there is no change. (ScdTest)
    [./test/scd_test.rb:47:in `__bind_1224209299_596896'
     /opt/local/lib/ruby/gems/1.8/gems/Shoulda-1.2.0/lib/shoulda.rb:226:in
`call'
     /opt/local/lib/ruby/gems/1.8/gems/Shoulda-1.2.0/lib/shoulda.rb:226:in
`test: when working with a slowly changing dimension of type 1 on run 2
should skip the load if there is no change. ']:
scheduled load expected to be empty, was 1 records.
<false> is not true.

D

> From: Anthony Eden <anthonyeden at gmail.com>
> Date: Fri, 17 Oct 2008 07:42:55 -0400
> To: Darin Wilson <dwlist at stellarmadness.org>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> I'm seeing even more errors with MySQL. A lot of them look like this:
> 
> Mysql::Error: #42000The used command is not allowed with this MySQL
> version: LOAD DATA LOCAL INFILE './test/output/scd_test_type_2.txt'
> INTO TABLE person_dimension FIELDS TERMINATED BY ','
> 
> What are the errors you are seeing when you run against MySQL?
> 
> -A
> 
> On Thu, Oct 16, 2008 at 10:14 PM, Darin Wilson
> <dwlist at stellarmadness.org> wrote:
>> I just pulled the latest from your github repo and ran the ETL test suite
>> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
>> those were the same ones you mentioned in your last message(?).
>> 
>> There was one other error:
>> 
>>  1) Failure:
>> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
>> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
>> 
>> It turns out that MySQL is much less strict about what goes into the bulk
>> loader - if you try to load a row with an empty value into a column that
>> disallows null, MySQL just skips it without an error. Nice! :)
>> 
>> I'm not sure how you'd like to handle that. I tried poking around a bit to
>> see if there's a way to force MySQL into being more strict, but I couldn't
>> get anything to work reliably. The simplest fix would be to do something
>> like this in the test:
>> 
>>    unless ETL::Engine.connection(:data_warehouse).class ==
>> ActiveRecord::ConnectionAdapters::MysqlAdapter
>>      assert_raise(ActiveRecord::StatementInvalid) {
>> do_bulk_import('bulk_import_with_empties.txt') }
>>    end
>> 
>> which sort of invalidates the test, but it doesn't seem like there's a way
>> to make this test actually work with MySQL. Your call. ;)
>> 
>> Darin
>> 
>> 
>>> From: Anthony Eden <anthonyeden at gmail.com>
>>> Date: Thu, 16 Oct 2008 15:23:04 -0400
>>> To: Darin Wilson <dwlist at stellarmadness.org>
>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>>> up? :)
>>> 
>>> I think I have most of the fixes to ETL in for the 2.1 changes. I've
>>> started changing over some of the tests to use shoulda 1.x, so if you
>>> want to run the built in tests then you'll need the shoulda gem. There
>>> are currently two failing tests related to the SCD implementation.
>>> Jacob, was that an area you worked on? If so could you take a look at
>>> the failing tests? Also, I currently have tested with Postgres locally
>>> but not MySQL, so if someone wants to fight through the test suite on
>>> MySQL, be my guest. :-)
>>> 
>>> Thanks.
>>> 
>>> -Anthony
>>> 
>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>>> <dwlist at stellarmadness.org> wrote:
>>>> Cool - that is what I was going to start working on. :)
>>>> 
>>>> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>>>> try it out with our ETL (and I have a bunch of test cases for it).
>>>> 
>>>> Thanks!
>>>> Darin
>>>> 
>>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>>>> To: Marty Haught <mghaught at gmail.com>
>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please
>>>>> stand
>>>>> up? :)
>>>>> 
>>>>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>>>>> ETL to handle the new migration process and I have it running right
>>>>> now under 2.1. Once I get past my first ETL load on the project I'm
>>>>> working on I'll move onto the AW plugin.
>>>>> 
>>>>> -Anthony
>>>>> _______________________________________________
>>>>> Activewarehouse-discuss mailing list
>>>>> Activewarehouse-discuss at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Activewarehouse-discuss mailing list
>>>> Activewarehouse-discuss at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>> 
>> 
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 



From jmaine at blurb.com  Fri Oct 17 13:56:13 2008
From: jmaine at blurb.com (Jacob Maine)
Date: Fri, 17 Oct 2008 10:56:13 -0700 (PDT)
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <bd41ed320810161223u71b6494bx5df65059e689a5c9@mail.gmail.com>
Message-ID: <3517087.84211224266173229.JavaMail.root@kilgore.blurb.com>

Anthony - tests are passing for me with Postgres.  I think the issue is that I made some changes to adapter_extensions.  I just sent you a pull request for adapter_extensions so let me know if that fixes tests.  I also sent a pull request for the activewarehouse code we've been using in production.  It only contains a few commits from the point that you last pulled from my branch, but it would be best to be on the same page.  I still have your pull request outstanding - I hope to get to that this weekend.  If that exposes test failures, I'll let you know.

Best,
Jacob

----- Anthony Eden <anthonyeden at gmail.com> wrote:
> I think I have most of the fixes to ETL in for the 2.1 changes. I've
> started changing over some of the tests to use shoulda 1.x, so if you
> want to run the built in tests then you'll need the shoulda gem. There
> are currently two failing tests related to the SCD implementation.
> Jacob, was that an area you worked on? If so could you take a look at
> the failing tests? Also, I currently have tested with Postgres locally
> but not MySQL, so if someone wants to fight through the test suite on
> MySQL, be my guest. :-)
> 
> Thanks.
> 
> -Anthony
> 
> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
> <dwlist at stellarmadness.org> wrote:
> > Cool - that is what I was going to start working on. :)
> >
> > Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
> > try it out with our ETL (and I have a bunch of test cases for it).
> >
> > Thanks!
> > Darin
> >
> >> From: Anthony Eden <anthonyeden at gmail.com>
> >> Date: Thu, 16 Oct 2008 06:59:41 -0400
> >> To: Marty Haught <mghaught at gmail.com>
> >> Cc: <activewarehouse-discuss at rubyforge.org>
> >> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> >> up? :)
> >>
> >> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
> >> ETL to handle the new migration process and I have it running right
> >> now under 2.1. Once I get past my first ETL load on the project I'm
> >> working on I'll move onto the AW plugin.
> >>
> >> -Anthony
> >> _______________________________________________
> >> Activewarehouse-discuss mailing list
> >> Activewarehouse-discuss at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
> >
> > _______________________________________________
> > Activewarehouse-discuss mailing list
> > Activewarehouse-discuss at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss


From jmaine at blurb.com  Fri Oct 17 14:07:53 2008
From: jmaine at blurb.com (Jacob Maine)
Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT)
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <C51D430D.1AB3%dwlist@stellarmadness.org>
Message-ID: <26015763.84641224266873854.JavaMail.root@kilgore.blurb.com>

Hi Darin - 

Please pull from my adapter_extensions branch (http://github.com/mainej/adapter_extensions/tree) or wait until Anthony pulls my changes.  Unfortunately, I'm not sure that will fix test_bulk_import_with_empties.  I can't be sure because I don't have MySQL installed, and don't have time to do that right now.  

The change I made to adapter_extensions allows Postgres to import empty strings as nulls.  That's what the test is about.  I put the test in for both adapters, hoping that MySQL would do the right thing out of the box.  If it's not working, I suggest either making your proposed change, or looking at adapter_extensions to see if you can get MySQL to handle empty strings in a more useful way.  I'll look at this stuff more this weekend to make sure my system is up-to-date and tests are passing for Postgres.

Hopefully with two groups developing for Postgres and MySQL, we'll get AW & AW ETL back into a stable state.

Best, 
Jacob


----- Darin Wilson <dwlist at stellarmadness.org> wrote:
> I just pulled the latest from your github repo and ran the ETL test suite
> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
> those were the same ones you mentioned in your last message(?).
> 
> There was one other error:
> 
>   1) Failure:
> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
> 
> It turns out that MySQL is much less strict about what goes into the bulk
> loader - if you try to load a row with an empty value into a column that
> disallows null, MySQL just skips it without an error. Nice! :)
> 
> I'm not sure how you'd like to handle that. I tried poking around a bit to
> see if there's a way to force MySQL into being more strict, but I couldn't
> get anything to work reliably. The simplest fix would be to do something
> like this in the test:
> 
>     unless ETL::Engine.connection(:data_warehouse).class ==
> ActiveRecord::ConnectionAdapters::MysqlAdapter
>       assert_raise(ActiveRecord::StatementInvalid) {
> do_bulk_import('bulk_import_with_empties.txt') }
>     end
> 
> which sort of invalidates the test, but it doesn't seem like there's a way
> to make this test actually work with MySQL. Your call. ;)
> 
> Darin
> 
> 
> > From: Anthony Eden <anthonyeden at gmail.com>
> > Date: Thu, 16 Oct 2008 15:23:04 -0400
> > To: Darin Wilson <dwlist at stellarmadness.org>
> > Cc: <activewarehouse-discuss at rubyforge.org>
> > Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> > up? :)
> > 
> > I think I have most of the fixes to ETL in for the 2.1 changes. I've
> > started changing over some of the tests to use shoulda 1.x, so if you
> > want to run the built in tests then you'll need the shoulda gem. There
> > are currently two failing tests related to the SCD implementation.
> > Jacob, was that an area you worked on? If so could you take a look at
> > the failing tests? Also, I currently have tested with Postgres locally
> > but not MySQL, so if someone wants to fight through the test suite on
> > MySQL, be my guest. :-)
> > 
> > Thanks.
> > 
> > -Anthony
> > 
> > On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
> > <dwlist at stellarmadness.org> wrote:
> >> Cool - that is what I was going to start working on. :)
> >> 
> >> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
> >> try it out with our ETL (and I have a bunch of test cases for it).
> >> 
> >> Thanks!
> >> Darin
> >> 
> >>> From: Anthony Eden <anthonyeden at gmail.com>
> >>> Date: Thu, 16 Oct 2008 06:59:41 -0400
> >>> To: Marty Haught <mghaught at gmail.com>
> >>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> >>> up? :)
> >>> 
> >>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
> >>> ETL to handle the new migration process and I have it running right
> >>> now under 2.1. Once I get past my first ETL load on the project I'm
> >>> working on I'll move onto the AW plugin.
> >>> 
> >>> -Anthony
> >>> _______________________________________________
> >>> Activewarehouse-discuss mailing list
> >>> Activewarehouse-discuss at rubyforge.org
> >>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >> 
> >> 
> >> _______________________________________________
> >> Activewarehouse-discuss mailing list
> >> Activewarehouse-discuss at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >> 
> 
> 
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss


From anthonyeden at gmail.com  Sat Oct 18 08:44:46 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Sat, 18 Oct 2008 08:44:46 -0400
Subject: [Activewarehouse-discuss] would the real codebase please stand
	up? :)
In-Reply-To: <26015763.84641224266873854.JavaMail.root@kilgore.blurb.com>
References: <C51D430D.1AB3%dwlist@stellarmadness.org>
	<26015763.84641224266873854.JavaMail.root@kilgore.blurb.com>
Message-ID: <bd41ed320810180544t47ac8002tb3b1cfd28da497f2@mail.gmail.com>

First the good news: I have AW ETL and AW plugin running under Rails
2.1. I've only been able to use AW plugin as a gem plugin at the
moment, and I did that by running rake reinstall in the
activewarehouse/activewarehouse directory (after fixing a few things).
There are warnings coming up about use of deprecated items, and there
is something funky with the way the generators are working, but it is
working.

Now the bad news: I gave up on MySQL for the moment. I use Postgres
and while I have MySQL on my OS X laptop, installing the MySQL ruby
driver is turning out to be an adventure.

Anyhow, I've committed some changes that let me run everything under
Rails 2.1, but this is definitely a case of use at your own risk.

-A

On Fri, Oct 17, 2008 at 2:07 PM, Jacob Maine <jmaine at blurb.com> wrote:
> Hi Darin -
>
> Please pull from my adapter_extensions branch (http://github.com/mainej/adapter_extensions/tree) or wait until Anthony pulls my changes.  Unfortunately, I'm not sure that will fix test_bulk_import_with_empties.  I can't be sure because I don't have MySQL installed, and don't have time to do that right now.
>
> The change I made to adapter_extensions allows Postgres to import empty strings as nulls.  That's what the test is about.  I put the test in for both adapters, hoping that MySQL would do the right thing out of the box.  If it's not working, I suggest either making your proposed change, or looking at adapter_extensions to see if you can get MySQL to handle empty strings in a more useful way.  I'll look at this stuff more this weekend to make sure my system is up-to-date and tests are passing for Postgres.
>
> Hopefully with two groups developing for Postgres and MySQL, we'll get AW & AW ETL back into a stable state.
>
> Best,
> Jacob
>
>
> ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
>> I just pulled the latest from your github repo and ran the ETL test suite
>> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
>> those were the same ones you mentioned in your last message(?).
>>
>> There was one other error:
>>
>>   1) Failure:
>> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
>> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
>>
>> It turns out that MySQL is much less strict about what goes into the bulk
>> loader - if you try to load a row with an empty value into a column that
>> disallows null, MySQL just skips it without an error. Nice! :)
>>
>> I'm not sure how you'd like to handle that. I tried poking around a bit to
>> see if there's a way to force MySQL into being more strict, but I couldn't
>> get anything to work reliably. The simplest fix would be to do something
>> like this in the test:
>>
>>     unless ETL::Engine.connection(:data_warehouse).class ==
>> ActiveRecord::ConnectionAdapters::MysqlAdapter
>>       assert_raise(ActiveRecord::StatementInvalid) {
>> do_bulk_import('bulk_import_with_empties.txt') }
>>     end
>>
>> which sort of invalidates the test, but it doesn't seem like there's a way
>> to make this test actually work with MySQL. Your call. ;)
>>
>> Darin
>>
>>
>> > From: Anthony Eden <anthonyeden at gmail.com>
>> > Date: Thu, 16 Oct 2008 15:23:04 -0400
>> > To: Darin Wilson <dwlist at stellarmadness.org>
>> > Cc: <activewarehouse-discuss at rubyforge.org>
>> > Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>> > up? :)
>> >
>> > I think I have most of the fixes to ETL in for the 2.1 changes. I've
>> > started changing over some of the tests to use shoulda 1.x, so if you
>> > want to run the built in tests then you'll need the shoulda gem. There
>> > are currently two failing tests related to the SCD implementation.
>> > Jacob, was that an area you worked on? If so could you take a look at
>> > the failing tests? Also, I currently have tested with Postgres locally
>> > but not MySQL, so if someone wants to fight through the test suite on
>> > MySQL, be my guest. :-)
>> >
>> > Thanks.
>> >
>> > -Anthony
>> >
>> > On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>> > <dwlist at stellarmadness.org> wrote:
>> >> Cool - that is what I was going to start working on. :)
>> >>
>> >> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>> >> try it out with our ETL (and I have a bunch of test cases for it).
>> >>
>> >> Thanks!
>> >> Darin
>> >>
>> >>> From: Anthony Eden <anthonyeden at gmail.com>
>> >>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>> >>> To: Marty Haught <mghaught at gmail.com>
>> >>> Cc: <activewarehouse-discuss at rubyforge.org>
>> >>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>> >>> up? :)
>> >>>
>> >>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>> >>> ETL to handle the new migration process and I have it running right
>> >>> now under 2.1. Once I get past my first ETL load on the project I'm
>> >>> working on I'll move onto the AW plugin.
>> >>>
>> >>> -Anthony
>> >>> _______________________________________________
>> >>> Activewarehouse-discuss mailing list
>> >>> Activewarehouse-discuss at rubyforge.org
>> >>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> >>
>> >>
>> >> _______________________________________________
>> >> Activewarehouse-discuss mailing list
>> >> Activewarehouse-discuss at rubyforge.org
>> >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> >>
>>
>>
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>

From dwlist at stellarmadness.org  Sat Oct 18 11:03:13 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Sat, 18 Oct 2008 08:03:13 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <bd41ed320810180544t47ac8002tb3b1cfd28da497f2@mail.gmail.com>
Message-ID: <C51F48C1.1B52%dwlist@stellarmadness.org>

That's great news! I'll give the ETL plugin a try with mysql, but I don't
have anything that's running the rails plugin yet - maybe someone else can
give that a try?

Jacob, I saw your message about your fixes to adapter-extensions, I just
haven't had a chance to try them out yet. I hope to do that sometime over
the weekend.

Thanks all,
D


> From: Anthony Eden <anthonyeden at gmail.com>
> Date: Sat, 18 Oct 2008 08:44:46 -0400
> To: Jacob Maine <jmaine at blurb.com>
> Cc: Darin Wilson <dwlist at stellarmadness.org>,
> <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> First the good news: I have AW ETL and AW plugin running under Rails
> 2.1. I've only been able to use AW plugin as a gem plugin at the
> moment, and I did that by running rake reinstall in the
> activewarehouse/activewarehouse directory (after fixing a few things).
> There are warnings coming up about use of deprecated items, and there
> is something funky with the way the generators are working, but it is
> working.
> 
> Now the bad news: I gave up on MySQL for the moment. I use Postgres
> and while I have MySQL on my OS X laptop, installing the MySQL ruby
> driver is turning out to be an adventure.
> 
> Anyhow, I've committed some changes that let me run everything under
> Rails 2.1, but this is definitely a case of use at your own risk.
> 
> -A
> 
> On Fri, Oct 17, 2008 at 2:07 PM, Jacob Maine <jmaine at blurb.com> wrote:
>> Hi Darin -
>> 
>> Please pull from my adapter_extensions branch
>> (http://github.com/mainej/adapter_extensions/tree) or wait until Anthony
>> pulls my changes.  Unfortunately, I'm not sure that will fix
>> test_bulk_import_with_empties.  I can't be sure because I don't have MySQL
>> installed, and don't have time to do that right now.
>> 
>> The change I made to adapter_extensions allows Postgres to import empty
>> strings as nulls.  That's what the test is about.  I put the test in for both
>> adapters, hoping that MySQL would do the right thing out of the box.  If it's
>> not working, I suggest either making your proposed change, or looking at
>> adapter_extensions to see if you can get MySQL to handle empty strings in a
>> more useful way.  I'll look at this stuff more this weekend to make sure my
>> system is up-to-date and tests are passing for Postgres.
>> 
>> Hopefully with two groups developing for Postgres and MySQL, we'll get AW &
>> AW ETL back into a stable state.
>> 
>> Best,
>> Jacob
>> 
>> 
>> ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
>>> I just pulled the latest from your github repo and ran the ETL test suite
>>> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
>>> those were the same ones you mentioned in your last message(?).
>>> 
>>> There was one other error:
>>> 
>>>   1) Failure:
>>> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
>>> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
>>> 
>>> It turns out that MySQL is much less strict about what goes into the bulk
>>> loader - if you try to load a row with an empty value into a column that
>>> disallows null, MySQL just skips it without an error. Nice! :)
>>> 
>>> I'm not sure how you'd like to handle that. I tried poking around a bit to
>>> see if there's a way to force MySQL into being more strict, but I couldn't
>>> get anything to work reliably. The simplest fix would be to do something
>>> like this in the test:
>>> 
>>>     unless ETL::Engine.connection(:data_warehouse).class ==
>>> ActiveRecord::ConnectionAdapters::MysqlAdapter
>>>       assert_raise(ActiveRecord::StatementInvalid) {
>>> do_bulk_import('bulk_import_with_empties.txt') }
>>>     end
>>> 
>>> which sort of invalidates the test, but it doesn't seem like there's a way
>>> to make this test actually work with MySQL. Your call. ;)
>>> 
>>> Darin
>>> 
>>> 
>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>> Date: Thu, 16 Oct 2008 15:23:04 -0400
>>>> To: Darin Wilson <dwlist at stellarmadness.org>
>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>>>> up? :)
>>>> 
>>>> I think I have most of the fixes to ETL in for the 2.1 changes. I've
>>>> started changing over some of the tests to use shoulda 1.x, so if you
>>>> want to run the built in tests then you'll need the shoulda gem. There
>>>> are currently two failing tests related to the SCD implementation.
>>>> Jacob, was that an area you worked on? If so could you take a look at
>>>> the failing tests? Also, I currently have tested with Postgres locally
>>>> but not MySQL, so if someone wants to fight through the test suite on
>>>> MySQL, be my guest. :-)
>>>> 
>>>> Thanks.
>>>> 
>>>> -Anthony
>>>> 
>>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>>>> <dwlist at stellarmadness.org> wrote:
>>>>> Cool - that is what I was going to start working on. :)
>>>>> 
>>>>> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>>>>> try it out with our ETL (and I have a bunch of test cases for it).
>>>>> 
>>>>> Thanks!
>>>>> Darin
>>>>> 
>>>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>>>>> To: Marty Haught <mghaught at gmail.com>
>>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please
>>>>>> stand
>>>>>> up? :)
>>>>>> 
>>>>>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>>>>>> ETL to handle the new migration process and I have it running right
>>>>>> now under 2.1. Once I get past my first ETL load on the project I'm
>>>>>> working on I'll move onto the AW plugin.
>>>>>> 
>>>>>> -Anthony
>>>>>> _______________________________________________
>>>>>> Activewarehouse-discuss mailing list
>>>>>> Activewarehouse-discuss at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Activewarehouse-discuss mailing list
>>>>> Activewarehouse-discuss at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Activewarehouse-discuss mailing list
>>> Activewarehouse-discuss at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>> 



From dwlist at stellarmadness.org  Wed Oct 22 16:34:34 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Wed, 22 Oct 2008 13:34:34 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <26015763.84641224266873854.JavaMail.root@kilgore.blurb.com>
Message-ID: <C524DC6A.1C0E%dwlist@stellarmadness.org>

Hi Jacob-

I finally got around to testing your fixes. Unfortunately, MySQL does not
convert empty strings to nulls out of the box, nor could I find a way to
make it support this behavior. (This was after a small amount of googling
around; there may be an answer somewhere that I just couldn't find - maybe
someone here knows?).

I'll keep poking around a little bit and see if I can see a way around this
- it would be a bummer if the ETL framework exhibited radically different
behavior depending on which adapter it was using.

Darin
  


> From: Jacob Maine <jmaine at blurb.com>
> Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT)
> To: Darin Wilson <dwlist at stellarmadness.org>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> Hi Darin - 
> 
> Please pull from my adapter_extensions branch
> (http://github.com/mainej/adapter_extensions/tree) or wait until Anthony pulls
> my changes.  Unfortunately, I'm not sure that will fix
> test_bulk_import_with_empties.  I can't be sure because I don't have MySQL
> installed, and don't have time to do that right now.
> 
> The change I made to adapter_extensions allows Postgres to import empty
> strings as nulls.  That's what the test is about.  I put the test in for both
> adapters, hoping that MySQL would do the right thing out of the box.  If it's
> not working, I suggest either making your proposed change, or looking at
> adapter_extensions to see if you can get MySQL to handle empty strings in a
> more useful way.  I'll look at this stuff more this weekend to make sure my
> system is up-to-date and tests are passing for Postgres.
> 
> Hopefully with two groups developing for Postgres and MySQL, we'll get AW & AW
> ETL back into a stable state.
> 
> Best, 
> Jacob
> 
> 
> ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
>> I just pulled the latest from your github repo and ran the ETL test suite
>> locally against MySQL. I got 10 errors that were about SCDs, so I'm assuming
>> those were the same ones you mentioned in your last message(?).
>> 
>> There was one other error:
>> 
>>   1) Failure:
>> test_bulk_import_with_empties(ProcessorTest) [./test/processor_test.rb:19]:
>> <ActiveRecord::StatementInvalid> exception expected but none was thrown.
>> 
>> It turns out that MySQL is much less strict about what goes into the bulk
>> loader - if you try to load a row with an empty value into a column that
>> disallows null, MySQL just skips it without an error. Nice! :)
>> 
>> I'm not sure how you'd like to handle that. I tried poking around a bit to
>> see if there's a way to force MySQL into being more strict, but I couldn't
>> get anything to work reliably. The simplest fix would be to do something
>> like this in the test:
>> 
>>     unless ETL::Engine.connection(:data_warehouse).class ==
>> ActiveRecord::ConnectionAdapters::MysqlAdapter
>>       assert_raise(ActiveRecord::StatementInvalid) {
>> do_bulk_import('bulk_import_with_empties.txt') }
>>     end
>> 
>> which sort of invalidates the test, but it doesn't seem like there's a way
>> to make this test actually work with MySQL. Your call. ;)
>> 
>> Darin
>> 
>> 
>>> From: Anthony Eden <anthonyeden at gmail.com>
>>> Date: Thu, 16 Oct 2008 15:23:04 -0400
>>> To: Darin Wilson <dwlist at stellarmadness.org>
>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
>>> up? :)
>>> 
>>> I think I have most of the fixes to ETL in for the 2.1 changes. I've
>>> started changing over some of the tests to use shoulda 1.x, so if you
>>> want to run the built in tests then you'll need the shoulda gem. There
>>> are currently two failing tests related to the SCD implementation.
>>> Jacob, was that an area you worked on? If so could you take a look at
>>> the failing tests? Also, I currently have tested with Postgres locally
>>> but not MySQL, so if someone wants to fight through the test suite on
>>> MySQL, be my guest. :-)
>>> 
>>> Thanks.
>>> 
>>> -Anthony
>>> 
>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>>> <dwlist at stellarmadness.org> wrote:
>>>> Cool - that is what I was going to start working on. :)
>>>> 
>>>> Please let us know when your 2.1 fixes to ETL are checked in - I'd like to
>>>> try it out with our ETL (and I have a bunch of test cases for it).
>>>> 
>>>> Thanks!
>>>> Darin
>>>> 
>>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>>>> To: Marty Haught <mghaught at gmail.com>
>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase please
>>>>> stand
>>>>> up? :)
>>>>> 
>>>>> I've started the upgrade to 2.1. I've already modified ActiveWarehouse
>>>>> ETL to handle the new migration process and I have it running right
>>>>> now under 2.1. Once I get past my first ETL load on the project I'm
>>>>> working on I'll move onto the AW plugin.
>>>>> 
>>>>> -Anthony
>>>>> _______________________________________________
>>>>> Activewarehouse-discuss mailing list
>>>>> Activewarehouse-discuss at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Activewarehouse-discuss mailing list
>>>> Activewarehouse-discuss at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>> 
>> 
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> 



From jmaine at blurb.com  Wed Oct 22 22:05:35 2008
From: jmaine at blurb.com (Jacob Maine)
Date: Wed, 22 Oct 2008 19:05:35 -0700 (PDT)
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <30374220.141431224727488786.JavaMail.root@kilgore.blurb.com>
Message-ID: <26121095.141451224727535693.JavaMail.root@kilgore.blurb.com>

Hi Darin -

It looks like MySQL supports a few different ways of converting text into nulls.  As you pointed out, I can't find documentation about turning empty strings into nulls.  However, it looks like you can convert \N (or NULL, depending on some other settings) into a null.  See http://dev.mysql.com/doc/refman/6.0/en/load-data.html starting with "Handling of NULL values varies ...".

Similarly, by default Postgres treats \N as a null, although it can be configured to act differently.  See http://www.postgresql.org/docs/8.3/interactive/sql-copy.html starting with "null string".

Maybe the best way to handle this would be to make AW output nulls as \N instead of empty strings.  Would you be willing to look into that?

I agree it's best to have AW-ETL work the same for both adapters.

Best,
Jacob


----- "Darin Wilson" <dwlist at stellarmadness.org> wrote:

> Hi Jacob-
> 
> I finally got around to testing your fixes. Unfortunately, MySQL does
> not
> convert empty strings to nulls out of the box, nor could I find a way
> to
> make it support this behavior. (This was after a small amount of
> googling
> around; there may be an answer somewhere that I just couldn't find -
> maybe
> someone here knows?).
> 
> I'll keep poking around a little bit and see if I can see a way around
> this
> - it would be a bummer if the ETL framework exhibited radically
> different
> behavior depending on which adapter it was using.
> 
> Darin
>   
> 
> 
> > From: Jacob Maine <jmaine at blurb.com>
> > Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT)
> > To: Darin Wilson <dwlist at stellarmadness.org>
> > Cc: <activewarehouse-discuss at rubyforge.org>
> > Subject: Re: [Activewarehouse-discuss] would the real codebase
> please stand
> > up? :)
> > 
> > Hi Darin - 
> > 
> > Please pull from my adapter_extensions branch
> > (http://github.com/mainej/adapter_extensions/tree) or wait until
> Anthony pulls
> > my changes.  Unfortunately, I'm not sure that will fix
> > test_bulk_import_with_empties.  I can't be sure because I don't have
> MySQL
> > installed, and don't have time to do that right now.
> > 
> > The change I made to adapter_extensions allows Postgres to import
> empty
> > strings as nulls.  That's what the test is about.  I put the test in
> for both
> > adapters, hoping that MySQL would do the right thing out of the box.
>  If it's
> > not working, I suggest either making your proposed change, or
> looking at
> > adapter_extensions to see if you can get MySQL to handle empty
> strings in a
> > more useful way.  I'll look at this stuff more this weekend to make
> sure my
> > system is up-to-date and tests are passing for Postgres.
> > 
> > Hopefully with two groups developing for Postgres and MySQL, we'll
> get AW & AW
> > ETL back into a stable state.
> > 
> > Best, 
> > Jacob
> > 
> > 
> > ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
> >> I just pulled the latest from your github repo and ran the ETL test
> suite
> >> locally against MySQL. I got 10 errors that were about SCDs, so I'm
> assuming
> >> those were the same ones you mentioned in your last message(?).
> >> 
> >> There was one other error:
> >> 
> >>   1) Failure:
> >> test_bulk_import_with_empties(ProcessorTest)
> [./test/processor_test.rb:19]:
> >> <ActiveRecord::StatementInvalid> exception expected but none was
> thrown.
> >> 
> >> It turns out that MySQL is much less strict about what goes into
> the bulk
> >> loader - if you try to load a row with an empty value into a column
> that
> >> disallows null, MySQL just skips it without an error. Nice! :)
> >> 
> >> I'm not sure how you'd like to handle that. I tried poking around a
> bit to
> >> see if there's a way to force MySQL into being more strict, but I
> couldn't
> >> get anything to work reliably. The simplest fix would be to do
> something
> >> like this in the test:
> >> 
> >>     unless ETL::Engine.connection(:data_warehouse).class ==
> >> ActiveRecord::ConnectionAdapters::MysqlAdapter
> >>       assert_raise(ActiveRecord::StatementInvalid) {
> >> do_bulk_import('bulk_import_with_empties.txt') }
> >>     end
> >> 
> >> which sort of invalidates the test, but it doesn't seem like
> there's a way
> >> to make this test actually work with MySQL. Your call. ;)
> >> 
> >> Darin
> >> 
> >> 
> >>> From: Anthony Eden <anthonyeden at gmail.com>
> >>> Date: Thu, 16 Oct 2008 15:23:04 -0400
> >>> To: Darin Wilson <dwlist at stellarmadness.org>
> >>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>> Subject: Re: [Activewarehouse-discuss] would the real codebase
> please stand
> >>> up? :)
> >>> 
> >>> I think I have most of the fixes to ETL in for the 2.1 changes.
> I've
> >>> started changing over some of the tests to use shoulda 1.x, so if
> you
> >>> want to run the built in tests then you'll need the shoulda gem.
> There
> >>> are currently two failing tests related to the SCD
> implementation.
> >>> Jacob, was that an area you worked on? If so could you take a look
> at
> >>> the failing tests? Also, I currently have tested with Postgres
> locally
> >>> but not MySQL, so if someone wants to fight through the test suite
> on
> >>> MySQL, be my guest. :-)
> >>> 
> >>> Thanks.
> >>> 
> >>> -Anthony
> >>> 
> >>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
> >>> <dwlist at stellarmadness.org> wrote:
> >>>> Cool - that is what I was going to start working on. :)
> >>>> 
> >>>> Please let us know when your 2.1 fixes to ETL are checked in -
> I'd like to
> >>>> try it out with our ETL (and I have a bunch of test cases for
> it).
> >>>> 
> >>>> Thanks!
> >>>> Darin
> >>>> 
> >>>>> From: Anthony Eden <anthonyeden at gmail.com>
> >>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
> >>>>> To: Marty Haught <mghaught at gmail.com>
> >>>>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
> please
> >>>>> stand
> >>>>> up? :)
> >>>>> 
> >>>>> I've started the upgrade to 2.1. I've already modified
> ActiveWarehouse
> >>>>> ETL to handle the new migration process and I have it running
> right
> >>>>> now under 2.1. Once I get past my first ETL load on the project
> I'm
> >>>>> working on I'll move onto the AW plugin.
> >>>>> 
> >>>>> -Anthony
> >>>>> _______________________________________________
> >>>>> Activewarehouse-discuss mailing list
> >>>>> Activewarehouse-discuss at rubyforge.org
> >>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >>>> 
> >>>> 
> >>>> _______________________________________________
> >>>> Activewarehouse-discuss mailing list
> >>>> Activewarehouse-discuss at rubyforge.org
> >>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >>>> 
> >> 
> >> 
> >> _______________________________________________
> >> Activewarehouse-discuss mailing list
> >> Activewarehouse-discuss at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> > 
> 
> 
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss

From dwlist at stellarmadness.org  Thu Oct 23 18:05:16 2008
From: dwlist at stellarmadness.org (Darin Wilson)
Date: Thu, 23 Oct 2008 15:05:16 -0700
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <26121095.141451224727535693.JavaMail.root@kilgore.blurb.com>
Message-ID: <C526432C.1C44%dwlist@stellarmadness.org>

Sure, I can take a look. I'm assuming that this would have to be handled at
the destination layer?

D


> From: Jacob Maine <jmaine at blurb.com>
> Date: Wed, 22 Oct 2008 19:05:35 -0700 (PDT)
> To: Darin Wilson <dwlist at stellarmadness.org>
> Cc: <activewarehouse-discuss at rubyforge.org>
> Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> up? :)
> 
> Hi Darin -
> 
> It looks like MySQL supports a few different ways of converting text into
> nulls.  As you pointed out, I can't find documentation about turning empty
> strings into nulls.  However, it looks like you can convert \N (or NULL,
> depending on some other settings) into a null.  See
> http://dev.mysql.com/doc/refman/6.0/en/load-data.html starting with "Handling
> of NULL values varies ...".
> 
> Similarly, by default Postgres treats \N as a null, although it can be
> configured to act differently.  See
> http://www.postgresql.org/docs/8.3/interactive/sql-copy.html starting with
> "null string".
> 
> Maybe the best way to handle this would be to make AW output nulls as \N
> instead of empty strings.  Would you be willing to look into that?
> 
> I agree it's best to have AW-ETL work the same for both adapters.
> 
> Best,
> Jacob
> 
> 
> ----- "Darin Wilson" <dwlist at stellarmadness.org> wrote:
> 
>> Hi Jacob-
>> 
>> I finally got around to testing your fixes. Unfortunately, MySQL does
>> not
>> convert empty strings to nulls out of the box, nor could I find a way
>> to
>> make it support this behavior. (This was after a small amount of
>> googling
>> around; there may be an answer somewhere that I just couldn't find -
>> maybe
>> someone here knows?).
>> 
>> I'll keep poking around a little bit and see if I can see a way around
>> this
>> - it would be a bummer if the ETL framework exhibited radically
>> different
>> behavior depending on which adapter it was using.
>> 
>> Darin
>>   
>> 
>> 
>>> From: Jacob Maine <jmaine at blurb.com>
>>> Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT)
>>> To: Darin Wilson <dwlist at stellarmadness.org>
>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
>> please stand
>>> up? :)
>>> 
>>> Hi Darin - 
>>> 
>>> Please pull from my adapter_extensions branch
>>> (http://github.com/mainej/adapter_extensions/tree) or wait until
>> Anthony pulls
>>> my changes.  Unfortunately, I'm not sure that will fix
>>> test_bulk_import_with_empties.  I can't be sure because I don't have
>> MySQL
>>> installed, and don't have time to do that right now.
>>> 
>>> The change I made to adapter_extensions allows Postgres to import
>> empty
>>> strings as nulls.  That's what the test is about.  I put the test in
>> for both
>>> adapters, hoping that MySQL would do the right thing out of the box.
>>  If it's
>>> not working, I suggest either making your proposed change, or
>> looking at
>>> adapter_extensions to see if you can get MySQL to handle empty
>> strings in a
>>> more useful way.  I'll look at this stuff more this weekend to make
>> sure my
>>> system is up-to-date and tests are passing for Postgres.
>>> 
>>> Hopefully with two groups developing for Postgres and MySQL, we'll
>> get AW & AW
>>> ETL back into a stable state.
>>> 
>>> Best, 
>>> Jacob
>>> 
>>> 
>>> ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
>>>> I just pulled the latest from your github repo and ran the ETL test
>> suite
>>>> locally against MySQL. I got 10 errors that were about SCDs, so I'm
>> assuming
>>>> those were the same ones you mentioned in your last message(?).
>>>> 
>>>> There was one other error:
>>>> 
>>>>   1) Failure:
>>>> test_bulk_import_with_empties(ProcessorTest)
>> [./test/processor_test.rb:19]:
>>>> <ActiveRecord::StatementInvalid> exception expected but none was
>> thrown.
>>>> 
>>>> It turns out that MySQL is much less strict about what goes into
>> the bulk
>>>> loader - if you try to load a row with an empty value into a column
>> that
>>>> disallows null, MySQL just skips it without an error. Nice! :)
>>>> 
>>>> I'm not sure how you'd like to handle that. I tried poking around a
>> bit to
>>>> see if there's a way to force MySQL into being more strict, but I
>> couldn't
>>>> get anything to work reliably. The simplest fix would be to do
>> something
>>>> like this in the test:
>>>> 
>>>>     unless ETL::Engine.connection(:data_warehouse).class ==
>>>> ActiveRecord::ConnectionAdapters::MysqlAdapter
>>>>       assert_raise(ActiveRecord::StatementInvalid) {
>>>> do_bulk_import('bulk_import_with_empties.txt') }
>>>>     end
>>>> 
>>>> which sort of invalidates the test, but it doesn't seem like
>> there's a way
>>>> to make this test actually work with MySQL. Your call. ;)
>>>> 
>>>> Darin
>>>> 
>>>> 
>>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>>> Date: Thu, 16 Oct 2008 15:23:04 -0400
>>>>> To: Darin Wilson <dwlist at stellarmadness.org>
>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
>> please stand
>>>>> up? :)
>>>>> 
>>>>> I think I have most of the fixes to ETL in for the 2.1 changes.
>> I've
>>>>> started changing over some of the tests to use shoulda 1.x, so if
>> you
>>>>> want to run the built in tests then you'll need the shoulda gem.
>> There
>>>>> are currently two failing tests related to the SCD
>> implementation.
>>>>> Jacob, was that an area you worked on? If so could you take a look
>> at
>>>>> the failing tests? Also, I currently have tested with Postgres
>> locally
>>>>> but not MySQL, so if someone wants to fight through the test suite
>> on
>>>>> MySQL, be my guest. :-)
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> -Anthony
>>>>> 
>>>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
>>>>> <dwlist at stellarmadness.org> wrote:
>>>>>> Cool - that is what I was going to start working on. :)
>>>>>> 
>>>>>> Please let us know when your 2.1 fixes to ETL are checked in -
>> I'd like to
>>>>>> try it out with our ETL (and I have a bunch of test cases for
>> it).
>>>>>> 
>>>>>> Thanks!
>>>>>> Darin
>>>>>> 
>>>>>>> From: Anthony Eden <anthonyeden at gmail.com>
>>>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
>>>>>>> To: Marty Haught <mghaught at gmail.com>
>>>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
>>>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
>> please
>>>>>>> stand
>>>>>>> up? :)
>>>>>>> 
>>>>>>> I've started the upgrade to 2.1. I've already modified
>> ActiveWarehouse
>>>>>>> ETL to handle the new migration process and I have it running
>> right
>>>>>>> now under 2.1. Once I get past my first ETL load on the project
>> I'm
>>>>>>> working on I'll move onto the AW plugin.
>>>>>>> 
>>>>>>> -Anthony
>>>>>>> _______________________________________________
>>>>>>> Activewarehouse-discuss mailing list
>>>>>>> Activewarehouse-discuss at rubyforge.org
>>>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Activewarehouse-discuss mailing list
>>>>>> Activewarehouse-discuss at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Activewarehouse-discuss mailing list
>>>> Activewarehouse-discuss at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>>> 
>> 
>> 
>> _______________________________________________
>> Activewarehouse-discuss mailing list
>> Activewarehouse-discuss at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss



From jmaine at blurb.com  Thu Oct 23 19:08:38 2008
From: jmaine at blurb.com (Jacob Maine)
Date: Thu, 23 Oct 2008 16:08:38 -0700 (PDT)
Subject: [Activewarehouse-discuss] would the real codebase please stand
 up? :)
In-Reply-To: <C526432C.1C44%dwlist@stellarmadness.org>
Message-ID: <26716934.150691224803318104.JavaMail.root@kilgore.blurb.com>

Yes, I believe it would involve modifying file_destination.rb.

----- Darin Wilson <dwlist at stellarmadness.org> wrote:
> Sure, I can take a look. I'm assuming that this would have to be handled at
> the destination layer?
> 
> D
> 
> 
> > From: Jacob Maine <jmaine at blurb.com>
> > Date: Wed, 22 Oct 2008 19:05:35 -0700 (PDT)
> > To: Darin Wilson <dwlist at stellarmadness.org>
> > Cc: <activewarehouse-discuss at rubyforge.org>
> > Subject: Re: [Activewarehouse-discuss] would the real codebase please stand
> > up? :)
> > 
> > Hi Darin -
> > 
> > It looks like MySQL supports a few different ways of converting text into
> > nulls.  As you pointed out, I can't find documentation about turning empty
> > strings into nulls.  However, it looks like you can convert \N (or NULL,
> > depending on some other settings) into a null.  See
> > http://dev.mysql.com/doc/refman/6.0/en/load-data.html starting with "Handling
> > of NULL values varies ...".
> > 
> > Similarly, by default Postgres treats \N as a null, although it can be
> > configured to act differently.  See
> > http://www.postgresql.org/docs/8.3/interactive/sql-copy.html starting with
> > "null string".
> > 
> > Maybe the best way to handle this would be to make AW output nulls as \N
> > instead of empty strings.  Would you be willing to look into that?
> > 
> > I agree it's best to have AW-ETL work the same for both adapters.
> > 
> > Best,
> > Jacob
> > 
> > 
> > ----- "Darin Wilson" <dwlist at stellarmadness.org> wrote:
> > 
> >> Hi Jacob-
> >> 
> >> I finally got around to testing your fixes. Unfortunately, MySQL does
> >> not
> >> convert empty strings to nulls out of the box, nor could I find a way
> >> to
> >> make it support this behavior. (This was after a small amount of
> >> googling
> >> around; there may be an answer somewhere that I just couldn't find -
> >> maybe
> >> someone here knows?).
> >> 
> >> I'll keep poking around a little bit and see if I can see a way around
> >> this
> >> - it would be a bummer if the ETL framework exhibited radically
> >> different
> >> behavior depending on which adapter it was using.
> >> 
> >> Darin
> >>   
> >> 
> >> 
> >>> From: Jacob Maine <jmaine at blurb.com>
> >>> Date: Fri, 17 Oct 2008 11:07:53 -0700 (PDT)
> >>> To: Darin Wilson <dwlist at stellarmadness.org>
> >>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>> Subject: Re: [Activewarehouse-discuss] would the real codebase
> >> please stand
> >>> up? :)
> >>> 
> >>> Hi Darin - 
> >>> 
> >>> Please pull from my adapter_extensions branch
> >>> (http://github.com/mainej/adapter_extensions/tree) or wait until
> >> Anthony pulls
> >>> my changes.  Unfortunately, I'm not sure that will fix
> >>> test_bulk_import_with_empties.  I can't be sure because I don't have
> >> MySQL
> >>> installed, and don't have time to do that right now.
> >>> 
> >>> The change I made to adapter_extensions allows Postgres to import
> >> empty
> >>> strings as nulls.  That's what the test is about.  I put the test in
> >> for both
> >>> adapters, hoping that MySQL would do the right thing out of the box.
> >>  If it's
> >>> not working, I suggest either making your proposed change, or
> >> looking at
> >>> adapter_extensions to see if you can get MySQL to handle empty
> >> strings in a
> >>> more useful way.  I'll look at this stuff more this weekend to make
> >> sure my
> >>> system is up-to-date and tests are passing for Postgres.
> >>> 
> >>> Hopefully with two groups developing for Postgres and MySQL, we'll
> >> get AW & AW
> >>> ETL back into a stable state.
> >>> 
> >>> Best, 
> >>> Jacob
> >>> 
> >>> 
> >>> ----- Darin Wilson <dwlist at stellarmadness.org> wrote:
> >>>> I just pulled the latest from your github repo and ran the ETL test
> >> suite
> >>>> locally against MySQL. I got 10 errors that were about SCDs, so I'm
> >> assuming
> >>>> those were the same ones you mentioned in your last message(?).
> >>>> 
> >>>> There was one other error:
> >>>> 
> >>>>   1) Failure:
> >>>> test_bulk_import_with_empties(ProcessorTest)
> >> [./test/processor_test.rb:19]:
> >>>> <ActiveRecord::StatementInvalid> exception expected but none was
> >> thrown.
> >>>> 
> >>>> It turns out that MySQL is much less strict about what goes into
> >> the bulk
> >>>> loader - if you try to load a row with an empty value into a column
> >> that
> >>>> disallows null, MySQL just skips it without an error. Nice! :)
> >>>> 
> >>>> I'm not sure how you'd like to handle that. I tried poking around a
> >> bit to
> >>>> see if there's a way to force MySQL into being more strict, but I
> >> couldn't
> >>>> get anything to work reliably. The simplest fix would be to do
> >> something
> >>>> like this in the test:
> >>>> 
> >>>>     unless ETL::Engine.connection(:data_warehouse).class ==
> >>>> ActiveRecord::ConnectionAdapters::MysqlAdapter
> >>>>       assert_raise(ActiveRecord::StatementInvalid) {
> >>>> do_bulk_import('bulk_import_with_empties.txt') }
> >>>>     end
> >>>> 
> >>>> which sort of invalidates the test, but it doesn't seem like
> >> there's a way
> >>>> to make this test actually work with MySQL. Your call. ;)
> >>>> 
> >>>> Darin
> >>>> 
> >>>> 
> >>>>> From: Anthony Eden <anthonyeden at gmail.com>
> >>>>> Date: Thu, 16 Oct 2008 15:23:04 -0400
> >>>>> To: Darin Wilson <dwlist at stellarmadness.org>
> >>>>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
> >> please stand
> >>>>> up? :)
> >>>>> 
> >>>>> I think I have most of the fixes to ETL in for the 2.1 changes.
> >> I've
> >>>>> started changing over some of the tests to use shoulda 1.x, so if
> >> you
> >>>>> want to run the built in tests then you'll need the shoulda gem.
> >> There
> >>>>> are currently two failing tests related to the SCD
> >> implementation.
> >>>>> Jacob, was that an area you worked on? If so could you take a look
> >> at
> >>>>> the failing tests? Also, I currently have tested with Postgres
> >> locally
> >>>>> but not MySQL, so if someone wants to fight through the test suite
> >> on
> >>>>> MySQL, be my guest. :-)
> >>>>> 
> >>>>> Thanks.
> >>>>> 
> >>>>> -Anthony
> >>>>> 
> >>>>> On Thu, Oct 16, 2008 at 12:11 PM, Darin Wilson
> >>>>> <dwlist at stellarmadness.org> wrote:
> >>>>>> Cool - that is what I was going to start working on. :)
> >>>>>> 
> >>>>>> Please let us know when your 2.1 fixes to ETL are checked in -
> >> I'd like to
> >>>>>> try it out with our ETL (and I have a bunch of test cases for
> >> it).
> >>>>>> 
> >>>>>> Thanks!
> >>>>>> Darin
> >>>>>> 
> >>>>>>> From: Anthony Eden <anthonyeden at gmail.com>
> >>>>>>> Date: Thu, 16 Oct 2008 06:59:41 -0400
> >>>>>>> To: Marty Haught <mghaught at gmail.com>
> >>>>>>> Cc: <activewarehouse-discuss at rubyforge.org>
> >>>>>>> Subject: Re: [Activewarehouse-discuss] would the real codebase
> >> please
> >>>>>>> stand
> >>>>>>> up? :)
> >>>>>>> 
> >>>>>>> I've started the upgrade to 2.1. I've already modified
> >> ActiveWarehouse
> >>>>>>> ETL to handle the new migration process and I have it running
> >> right
> >>>>>>> now under 2.1. Once I get past my first ETL load on the project
> >> I'm
> >>>>>>> working on I'll move onto the AW plugin.
> >>>>>>> 
> >>>>>>> -Anthony
> >>>>>>> _______________________________________________
> >>>>>>> Activewarehouse-discuss mailing list
> >>>>>>> Activewarehouse-discuss at rubyforge.org
> >>>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >>>>>> 
> >>>>>> 
> >>>>>> _______________________________________________
> >>>>>> Activewarehouse-discuss mailing list
> >>>>>> Activewarehouse-discuss at rubyforge.org
> >>>>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >>>>>> 
> >>>> 
> >>>> 
> >>>> _______________________________________________
> >>>> Activewarehouse-discuss mailing list
> >>>> Activewarehouse-discuss at rubyforge.org
> >>>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >>> 
> >> 
> >> 
> >> _______________________________________________
> >> Activewarehouse-discuss mailing list
> >> Activewarehouse-discuss at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> 
> 
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss


From anthonyeden at gmail.com  Fri Oct 24 11:23:40 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Fri, 24 Oct 2008 11:23:40 -0400
Subject: [Activewarehouse-discuss] Moving ActiveWarehouse subprojects to
	their own repos
Message-ID: <bd41ed320810240823u14713469mc3bb4a932aa962d2@mail.gmail.com>

Morning all,

I'd like to move the subprojects currently living in the
activewarehouse repo into their own github repos. This will especially
useful for the AW plugin since it can then be installed using
script/plugin.

Any objections?

-Anthony

From thibaut.barrere at gmail.com  Fri Oct 24 11:36:09 2008
From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=)
Date: Fri, 24 Oct 2008 17:36:09 +0200
Subject: [Activewarehouse-discuss] Moving ActiveWarehouse subprojects to
	their own repos
In-Reply-To: <bd41ed320810240823u14713469mc3bb4a932aa962d2@mail.gmail.com>
References: <bd41ed320810240823u14713469mc3bb4a932aa962d2@mail.gmail.com>
Message-ID: <4a68b8cf0810240836m49995893y310d7a1613dd6a11@mail.gmail.com>

>
> Any objections?
>

Fine for me.

-- Thibaut
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20081024/4d4f3605/attachment.html>