From thibaut.barrere at gmail.com Fri Dec 7 10:46:55 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 7 Dec 2007 16:46:55 +0100 Subject: [Activewarehouse-discuss] Modelisation question - handling unknown, unreliable values in integer columns Message-ID: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> Hi! I have a "duration" attribute in one of my facts tables. Until now this attribute was stored as a string (could be "5","100", but also "Unreliable","Unknown"). Recently the need to compute average values on this attribute arised, sliced by dimension. Does anyone know the typical workaround for such a situation ? I've got one idea: replacing this column by two; one would keep the quality of the data, "Reliable","Unreliable","Unknown" to be able to filter relevant data, the other one would keep an integer value, and some default value when nothing reliable or known is available. What do you think ? PS: if some of you are coming to http://www.metrorubyonrails.com/ (Paris), I'll give an AW-ETL project feedback presentation there. cheers Thibaut Barr?re -- http://www.logeek.fr -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071207/1f60c850/attachment.html From anthonyeden at gmail.com Fri Dec 7 10:53:32 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 7 Dec 2007 10:53:32 -0500 Subject: [Activewarehouse-discuss] Modelisation question - handling unknown, unreliable values in integer columns In-Reply-To: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> References: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> Message-ID: Shouldn't duration be a dimension? If the value is being stored in the fact table as a measurement then it should be an additive or semi-additive value, of which string is neither. Would it be possible for you to store it in a dimension? V/r Anthony On Dec 7, 2007 10:46 AM, Thibaut Barr?re wrote: > Hi! > > I have a "duration" attribute in one of my facts tables. Until now this > attribute was stored as a string (could be "5","100", but also > "Unreliable","Unknown"). > Recently the need to compute average values on this attribute arised, > sliced by dimension. > > Does anyone know the typical workaround for such a situation ? > > I've got one idea: replacing this column by two; one would keep the quality > of the data, "Reliable","Unreliable","Unknown" to be able to filter relevant > data, the other one would keep an integer value, and some default value when > nothing reliable or known is available. > > What do you think ? > > PS: if some of you are coming to http://www.metrorubyonrails.com/ (Paris), > I'll give an AW-ETL project feedback presentation there. > > cheers > > Thibaut Barr?re > -- > http://www.logeek.fr > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > From thibaut.barrere at gmail.com Fri Dec 7 11:00:32 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 7 Dec 2007 17:00:32 +0100 Subject: [Activewarehouse-discuss] Modelisation question - handling unknown, unreliable values in integer columns In-Reply-To: References: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> Message-ID: <4a68b8cf0712070800u1650ebf7ta3c284d471b6ca1d@mail.gmail.com> > Shouldn't duration be a dimension? If the value is being stored in the > fact table as a measurement then it should be an additive or > semi-additive value, of which string is neither. Would it be possible > for you to store it in a dimension? Yes it's possible, although less comfortable ! I've kept some dimension-like fields as attribute of the fact for simplicity so far. >> I've got one idea: replacing this column by two; one would keep the quality >> of the data, "Reliable","Unreliable", "Unknown" to be able to filter relevant >> data, the other one would keep an integer value, and some default value when >> nothing reliable or known is available. If I transform it into a dimension, should I split the value into two like I suggested, to allow filtering ? Or is there another practice ? -- Thibaut -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071207/68e0a890/attachment.html From anthonyeden at gmail.com Fri Dec 7 11:07:01 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Fri, 7 Dec 2007 11:07:01 -0500 Subject: [Activewarehouse-discuss] Modelisation question - handling unknown, unreliable values in integer columns In-Reply-To: <4a68b8cf0712070800u1650ebf7ta3c284d471b6ca1d@mail.gmail.com> References: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> <4a68b8cf0712070800u1650ebf7ta3c284d471b6ca1d@mail.gmail.com> Message-ID: (Sorry, resending so the list gets the response as well.) On Dec 7, 2007 11:00 AM, Thibaut Barr?re wrote: > > > > Shouldn't duration be a dimension? If the value is being stored in the > > fact table as a measurement then it should be an additive or > > semi-additive value, of which string is neither. Would it be possible > > for you to store it in a dimension? > > Yes it's possible, although less comfortable ! I've kept some dimension-like > fields as attribute of the fact for simplicity so far. I would suggest that unless it is a FK to a dimension, a degenerate dimension (which in this case may have been the case for the duration) or a measurement (which also may be the case for the duration) then it should not be in the fact table. I think the question of duration is a bit sticky because on the one hand it is a measurement, but on the other it can have values that are non-numeric if you allow it to do so (which you do). > > >> I've got one idea: replacing this column by two; one would keep the > quality > >> of the data, "Reliable","Unreliable", "Unknown" to be able to filter > relevant > >> data, the other one would keep an integer value, and some default value > when > >> nothing reliable or known is available. > > If I transform it into a dimension, should I split the value into two like I > suggested, to allow filtering ? Or is there another practice ? Yes, I think you should separate it into multiple values. V/r Anthony From thibaut.barrere at gmail.com Fri Dec 7 11:14:03 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Fri, 7 Dec 2007 17:14:03 +0100 Subject: [Activewarehouse-discuss] Modelisation question - handling unknown, unreliable values in integer columns In-Reply-To: References: <4a68b8cf0712070746h640b58a4yf05c52315fdf1c@mail.gmail.com> <4a68b8cf0712070800u1650ebf7ta3c284d471b6ca1d@mail.gmail.com> Message-ID: <4a68b8cf0712070814o27b3c89fx2a20867dba194120@mail.gmail.com> That's a bit of edge case yes.. Thanks for your answer, appreciated! -- Thibaut I would suggest that unless it is a FK to a dimension, a degenerate > dimension (which in this case may have been the case for the duration) > or a measurement (which also may be the case for the duration) then it > should not be in the fact table. I think the question of duration is a > bit sticky because on the one hand it is a measurement, but on the > other it can have values that are non-numeric if you allow it to do so > (which you do). > > > If I transform it into a dimension, should I split the value into two > like I > > suggested, to allow filtering ? Or is there another practice ? > > Yes, I think you should separate it into multiple values. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071207/f6b3b771/attachment.html From phylae at gmail.com Thu Dec 13 18:29:55 2007 From: phylae at gmail.com (Paul Cortens) Date: Thu, 13 Dec 2007 15:29:55 -0800 Subject: [Activewarehouse-discuss] Newbie needs help! Message-ID: <001601c83de0$122df6f0$3689e4d0$@com> Hi, I have been trying to work through the tutorial posted here, and I haven't been having much luck. When I try to run author_dimension.ctl I get the following error. ****************** F:\ruby_testing\DW\rails_warehouse\db\etl>etl author_dimension.ctl Using AdapterExtensions Starting ETL process initializing ETL engine Processing author_dimension.ctl Executing screens Screens passed Executing post processes c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record/connecti on_a dapters/abstract_adapter.rb:150:in `log': Mysql::Error: #42000The used command i s not allowed with this MySQL version: LOAD DATA LOCAL INFILE './author_dimensio n.txt' INTO TABLE author_dimension FIELDS TERMINATED BY ',' (name) (ActiveRecord ::StatementInvalid) from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record /connection_adapters/mysql_adapter.rb:281:in `execute' from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter _extensions/connection_adapters/mysql_adapter.rb:46:in `do_bulk_load' from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter _extensions/connection_adapters/abstract_adapter.rb:18:in `bulk_load' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib /etl/processor/bulk_import_processor.rb:68:in `process' from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record /connection_adapters/abstract/database_statements.rb:66:in `transaction' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib /etl/processor/bulk_import_processor.rb:58:in `process' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib /etl/engine.rb:485:in `post_process' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib /etl/engine.rb:484:in `each' ... 13 levels... from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.1/lib/active_suppo rt/dependencies.rb:496:in `require' from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 from c:/ruby/bin/etl:16:in `load' from c:/ruby/bin/etl:16 ****************** This is my environment: Windows XP Pro SP2. rails -v => 2.0.1 ruby -v => ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32] gem -v => 0.9.4 MySQL 'select @@version;' => 5.0.27-community-nt If I run the command "LOAD DATA LOCAL INFILE './author_dimension.txt' INTO TABLE author_dimension FIELDS TERMINATED BY ',' (name);" directly in MySQL, then the data imports just fine. (Actually it is not completely fine, it I get line breaks at the end of some of the strings.) Any ideas? Thanks, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071213/3cd5abff/attachment-0001.html From phylae at gmail.com Thu Dec 13 21:18:48 2007 From: phylae at gmail.com (Paul Cortens) Date: Thu, 13 Dec 2007 18:18:48 -0800 Subject: [Activewarehouse-discuss] Newbie needs help! In-Reply-To: References: <001601c83de0$122df6f0$3689e4d0$@com> <001c01c83ded$5a701ee0$0f505ca0$@com> Message-ID: <002001c83df7$a986cf60$fc946e20$@com> That was it. I was using the ruby version. Once I installed the win32 version it works. THANK YOU!!!! Paul -----Original Message----- From: Chris Williams [mailto:chris.d.williams at gmail.com] Sent: Thursday, December 13, 2007 6:01 PM To: Paul Cortens Subject: Re: [Activewarehouse-discuss] Newbie needs help! Which MySQL gem are you using? I think you want to make sure you are using the win32 version, not the ruby version. On Dec 13, 2007 8:05 PM, Paul Cortens wrote: > Thanks. If I add "LINES TERMINATED BY '\r\n'" to the MySQL query, then I > don't have the line break problem. But I am still stuck as to why > ActiveWarehouse chokes, but I can manually run the exact same SQL code > through MySQL manually. > > Cheers, > Paul > > > -----Original Message----- > From: Chris Williams [mailto:chris.d.williams at gmail.com] > Sent: Thursday, December 13, 2007 3:46 PM > To: Paul Cortens > Subject: Re: [Activewarehouse-discuss] Newbie needs help! > > Hi Paul. I ran in to some of the same problems when running on > Windows. I went through some of my old posts to the list but can't > find exactly what I did. I know I made some code changes to AW in > order to get rid of the \n in the last field in the CSV file. I will > keep searching and see if I can help out at all. > > Chris > > On Dec 13, 2007 6:29 PM, Paul Cortens wrote: > > > > > > > > > > Hi, > > > > I have been trying to work through the tutorial posted > here, > > and I haven't been having much luck. > > > > > > > > > > > > When I try to run author_dimension.ctl I get the following error. > > > > > > > > ****************** > > > > > > > > F:\ruby_testing\DW\rails_warehouse\db\etl>etl author_dimension.ctl > > > > Using AdapterExtensions > > > > Starting ETL process > > > > initializing ETL engine > > > > Processing author_dimension.ctl > > > > > > > > Executing screens > > > > Screens passed > > > > > > > > Executing post processes > > > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record/connecti > on_a > > > > dapters/abstract_adapter.rb:150:in `log': Mysql::Error: #42000The used > > command i > > > > s not allowed with this MySQL version: LOAD DATA LOCAL INFILE > > './author_dimensio > > > > n.txt' INTO TABLE author_dimension FIELDS TERMINATED BY ',' (name) > > (ActiveRecord > > > > ::StatementInvalid) > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > /connection_adapters/mysql_adapter.rb:281:in `execute' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > _extensions/connection_adapters/mysql_adapter.rb:46:in `do_bulk_load' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > _extensions/connection_adapters/abstract_adapter.rb:18:in `bulk_load' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > /etl/processor/bulk_import_processor.rb:68:in `process' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > /connection_adapters/abstract/database_statements.rb:66:in `transaction' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > /etl/processor/bulk_import_processor.rb:58:in `process' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > /etl/engine.rb:485:in `post_process' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > /etl/engine.rb:484:in `each' > > > > ... 13 levels... > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.1/lib/active_suppo > > > > rt/dependencies.rb:496:in `require' > > > > from > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 > > > > > > > > from c:/ruby/bin/etl:16:in `load' > > > > from c:/ruby/bin/etl:16 > > > > > > > > > > > > ****************** > > > > > > > > > > > > This is my environment: > > > > Windows XP Pro SP2. > > > > rails -v => 2.0.1 > > > > ruby -v => ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32] > > > > gem -v => 0.9.4 > > > > MySQL 'select @@version;' => 5.0.27-community-nt > > > > > > > > > > > > If I run the command "LOAD DATA LOCAL INFILE './author_dimension.txt' INTO > > TABLE author_dimension FIELDS TERMINATED BY ',' (name);" directly in > MySQL, > > then the data imports just fine. (Actually it is not completely fine, it I > > get line breaks at the end of some of the strings.) > > > > > > > > Any ideas? > > > > > > > > Thanks, > > > > Paul > > > > > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > From lists at kikobu.com Mon Dec 17 10:46:52 2007 From: lists at kikobu.com (Morten) Date: Mon, 17 Dec 2007 16:46:52 +0100 Subject: [Activewarehouse-discuss] Rails 2.0 Message-ID: <2BC112D2-5517-4025-9508-90CF4D8F0D62@kikobu.com> Hi. Any chance we'll see an update for rails_sql_views that takes the Rails 2.0 adapter refactoring into consideration? Adding "if defined?(...)" to rails_sql_views.rb would do I suppose: require 'rails_sql_views/connection_adapters/postgresql_adapter' if defined?(ActiveRecord::ConnectionAdapters::PostgreSQLAdapter) require 'rails_sql_views/connection_adapters/sqlserver_adapter' if defined?(ActiveRecord::ConnectionAdapters::SqlServerAdapter) Br, Morten From endersonmaia at gmail.com Mon Dec 17 12:50:57 2007 From: endersonmaia at gmail.com (Enderson Maia) Date: Mon, 17 Dec 2007 14:50:57 -0300 Subject: [Activewarehouse-discuss] Newbie needs help! In-Reply-To: <002001c83df7$a986cf60$fc946e20$@com> References: <001601c83de0$122df6f0$3689e4d0$@com> <001c01c83ded$5a701ee0$0f505ca0$@com> <002001c83df7$a986cf60$fc946e20$@com> Message-ID: <228840e70712170950ie9f9874ybaa505d14e88c64c@mail.gmail.com> I have the same problem with MacOS, but I think it's 'cause teh aw-etl is using ar-2.0.1, but it worked with 1.15.6 How can I force it to use ar-1.15.6 ? On Dec 13, 2007 11:18 PM, Paul Cortens wrote: > That was it. I was using the ruby version. Once I installed the win32 > version it works. > > THANK YOU!!!! > > Paul > > -----Original Message----- > From: Chris Williams [mailto:chris.d.williams at gmail.com] > Sent: Thursday, December 13, 2007 6:01 PM > To: Paul Cortens > Subject: Re: [Activewarehouse-discuss] Newbie needs help! > > Which MySQL gem are you using? I think you want to make sure you are > using the win32 version, not the ruby version. > > On Dec 13, 2007 8:05 PM, Paul Cortens wrote: > > Thanks. If I add "LINES TERMINATED BY '\r\n'" to the MySQL query, then > I > > don't have the line break problem. But I am still stuck as to why > > ActiveWarehouse chokes, but I can manually run the exact same SQL code > > through MySQL manually. > > > > Cheers, > > Paul > > > > > > -----Original Message----- > > From: Chris Williams [mailto:chris.d.williams at gmail.com] > > Sent: Thursday, December 13, 2007 3:46 PM > > To: Paul Cortens > > Subject: Re: [Activewarehouse-discuss] Newbie needs help! > > > > Hi Paul. I ran in to some of the same problems when running on > > Windows. I went through some of my old posts to the list but can't > > find exactly what I did. I know I made some code changes to AW in > > order to get rid of the \n in the last field in the CSV file. I will > > keep searching and see if I can help out at all. > > > > Chris > > > > On Dec 13, 2007 6:29 PM, Paul Cortens wrote: > > > > > > > > > > > > > > > Hi, > > > > > > I have been trying to work through the tutorial posted > > here, > > > and I haven't been having much luck. > > > > > > > > > > > > > > > > > > When I try to run author_dimension.ctl I get the following error. > > > > > > > > > > > > ****************** > > > > > > > > > > > > F:\ruby_testing\DW\rails_warehouse\db\etl>etl author_dimension.ctl > > > > > > Using AdapterExtensions > > > > > > Starting ETL process > > > > > > initializing ETL engine > > > > > > Processing author_dimension.ctl > > > > > > > > > > > > Executing screens > > > > > > Screens passed > > > > > > > > > > > > Executing post processes > > > > > > > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1 > /lib/active_record/connecti > > on_a > > > > > > dapters/abstract_adapter.rb:150:in `log': Mysql::Error: #42000The used > > > command i > > > > > > s not allowed with this MySQL version: LOAD DATA LOCAL INFILE > > > './author_dimensio > > > > > > n.txt' INTO TABLE author_dimension FIELDS TERMINATED BY ',' (name) > > > (ActiveRecord > > > > > > ::StatementInvalid) > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > > > /connection_adapters/mysql_adapter.rb:281:in `execute' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > > > _extensions/connection_adapters/mysql_adapter.rb:46:in `do_bulk_load' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > > > _extensions/connection_adapters/abstract_adapter.rb:18:in `bulk_load' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > /etl/processor/bulk_import_processor.rb:68:in `process' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > > > /connection_adapters/abstract/database_statements.rb:66:in > `transaction' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > /etl/processor/bulk_import_processor.rb:58:in `process' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > /etl/engine.rb:485:in `post_process' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > /etl/engine.rb:484:in `each' > > > > > > ... 13 levels... > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.1/lib/active_suppo > > > > > > rt/dependencies.rb:496:in `require' > > > > > > from > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 > > > > > > > > > > > > from c:/ruby/bin/etl:16:in `load' > > > > > > from c:/ruby/bin/etl:16 > > > > > > > > > > > > > > > > > > ****************** > > > > > > > > > > > > > > > > > > This is my environment: > > > > > > Windows XP Pro SP2. > > > > > > rails -v => 2.0.1 > > > > > > ruby -v => ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32] > > > > > > gem -v => 0.9.4 > > > > > > MySQL 'select @@version;' => 5.0.27-community-nt > > > > > > > > > > > > > > > > > > If I run the command "LOAD DATA LOCAL INFILE './author_dimension.txt' > INTO > > > TABLE author_dimension FIELDS TERMINATED BY ',' (name);" directly in > > MySQL, > > > then the data imports just fine. (Actually it is not completely fine, > it > I > > > get line breaks at the end of some of the strings.) > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > Thanks, > > > > > > Paul > > > > > > > > > _______________________________________________ > > > Activewarehouse-discuss mailing list > > > Activewarehouse-discuss at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > > > > > > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > -- Enderson Maia -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071217/10826275/attachment.html From anthonyeden at gmail.com Mon Dec 17 13:00:34 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Mon, 17 Dec 2007 13:00:34 -0500 Subject: [Activewarehouse-discuss] Rails 2.0 In-Reply-To: <2BC112D2-5517-4025-9508-90CF4D8F0D62@kikobu.com> References: <2BC112D2-5517-4025-9508-90CF4D8F0D62@kikobu.com> Message-ID: I'll see if I can spend some time fixing things up and testing it with 2.x. V/r Anthony On 12/17/07, Morten wrote: > > Hi. Any chance we'll see an update for rails_sql_views that takes the > Rails 2.0 adapter refactoring into consideration? > > Adding "if defined?(...)" to rails_sql_views.rb would do I suppose: > > require 'rails_sql_views/connection_adapters/postgresql_adapter' if > defined?(ActiveRecord::ConnectionAdapters::PostgreSQLAdapter) > require 'rails_sql_views/connection_adapters/sqlserver_adapter' if > defined?(ActiveRecord::ConnectionAdapters::SqlServerAdapter) > > Br, > > Morten > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From public at misuse.org Wed Dec 19 22:56:22 2007 From: public at misuse.org (Steve Midgley) Date: Wed, 19 Dec 2007 19:56:22 -0800 Subject: [Activewarehouse-discuss] adapter_extensions Message-ID: <20071220040308.D854618585CE@rubyforge.org> Hi warehouse friends, I took a look at the adapter_extensions code for Postgres today - it's a great idea. I think the code you've got only works for loading files onto a file system accessible by the Postgres server? In a shared hosting environment this may surprisingly limited. I think most people would expect to be able to load files from anywhere visible to the client? I have some Postgres sql code that permits bulk loading data via COPY, but using STDIN as the source, so you can feed it via a file on the client side, rather than from a server-accessible file. Of course it's not as fast as reading a file from a local SCSI array, but sending data to copy COPY over a network wire is way faster than any other remote technique I've found. I'm happy to post this code to the list or to the code maintainer if it's of interest. Just let me know! Sincerely, Steve From bdimchef at wieldim.com Thu Dec 20 10:30:34 2007 From: bdimchef at wieldim.com (Brandon Dimcheff) Date: Thu, 20 Dec 2007 10:30:34 -0500 Subject: [Activewarehouse-discuss] Adapter Extensions Tests Failing Message-ID: I'm trying to add a couple features to the the adapter extensions, but I can't seem to get a few of the tests to run. For instance, test_bulk_load throws: Mysql::Error: The used command is not allowed with this MySQL version: LOAD DATA LOCAL INFILE './test/integration/people.txt' INTO TABLE people According to pretty much whatever I've read, this is caused by local- infile being disabled on either the client or server side. I can run the query in the mysql command line client, so it must be something fishy with ActiveRecord. The strange thing is, I can run that same query in script/console in my activewarehouse-based project and it executes just fine. I just can't seem to figure out what's different about the way ActiveRecord is being used in these tests vs. in a rails app... Any ideas? Thanks, Brandon From anthonyeden at gmail.com Thu Dec 20 13:23:35 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 20 Dec 2007 13:23:35 -0500 Subject: [Activewarehouse-discuss] adapter_extensions In-Reply-To: <20071220040308.D854618585CE@rubyforge.org> References: <20071220040308.D854618585CE@rubyforge.org> Message-ID: Sure, go ahead and post it and I'll see if I can get it in there. Perhaps it can be used as a failover if the local file system is inaccessible. V/r Anthony Eden On 12/19/07, Steve Midgley wrote: > Hi warehouse friends, > > I took a look at the adapter_extensions code for Postgres today - it's > a great idea. I think the code you've got only works for loading files > onto a file system accessible by the Postgres server? In a shared > hosting environment this may surprisingly limited. I think most people > would expect to be able to load files from anywhere visible to the > client? > > I have some Postgres sql code that permits bulk loading data via COPY, > but using STDIN as the source, so you can feed it via a file on the > client side, rather than from a server-accessible file. Of course it's > not as fast as reading a file from a local SCSI array, but sending data > to copy COPY over a network wire is way faster than any other remote > technique I've found. > > I'm happy to post this code to the list or to the code maintainer if > it's of interest. Just let me know! > > Sincerely, > > Steve > > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From anthonyeden at gmail.com Thu Dec 20 13:39:26 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Thu, 20 Dec 2007 13:39:26 -0500 Subject: [Activewarehouse-discuss] Adapter Extensions Tests Failing In-Reply-To: References: Message-ID: It might be that the Rails version is using the mysql adapter that comes with Rails, which is a pure Ruby implementation that has a maximum limit to the number of bytes it can ship over the wire. This problem has come up a couple of times, so I finally pulled the content from emails I had sent to individuals and posted in on my blog: http://anthonyeden.com/2007/12/20/mysql-bulk-load-with-pure-mysql-driver-in-rails V/r Anthony Eden On 12/20/07, Brandon Dimcheff wrote: > I'm trying to add a couple features to the the adapter extensions, but > I can't seem to get a few of the tests to run. For instance, > test_bulk_load throws: > > Mysql::Error: The used command is not allowed with this MySQL version: > LOAD DATA LOCAL INFILE './test/integration/people.txt' INTO TABLE people > > According to pretty much whatever I've read, this is caused by local- > infile being disabled on either the client or server side. I can run > the query in the mysql command line client, so it must be something > fishy with ActiveRecord. The strange thing is, I can run that same > query in script/console in my activewarehouse-based project and it > executes just fine. I just can't seem to figure out what's different > about the way ActiveRecord is being used in these tests vs. in a rails > app... > > Any ideas? > > Thanks, > Brandon > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > From public at misuse.org Fri Dec 21 13:14:24 2007 From: public at misuse.org (Steve Midgley) Date: Fri, 21 Dec 2007 10:14:24 -0800 Subject: [Activewarehouse-discuss] adapter_extensions In-Reply-To: References: <20071220040308.D854618585CE@rubyforge.org> Message-ID: <20071221181433.72BB818585C9@rubyforge.org> Hi, Here's a little writeup I did awhile ago on this: http://www.misuse.org/science/2007/02/16/importing-to-postgresql-from-activerecord-amp-rails/ Let me know if you have trouble getting it going. This method of import is at least 1000 times faster than using standard activerecord inserts.. Steve At 10:23 AM 12/20/2007, Anthony Eden wrote: >Sure, go ahead and post it and I'll see if I can get it in there. >Perhaps it can be used as a failover if the local file system is >inaccessible. > >V/r >Anthony Eden > >On 12/19/07, Steve Midgley wrote: > > Hi warehouse friends, > > > > I took a look at the adapter_extensions code for Postgres today - > it's > > a great idea. I think the code you've got only works for loading > files > > onto a file system accessible by the Postgres server? In a shared > > hosting environment this may surprisingly limited. I think most > people > > would expect to be able to load files from anywhere visible to the > > client? > > > > I have some Postgres sql code that permits bulk loading data via > COPY, > > but using STDIN as the source, so you can feed it via a file on the > > client side, rather than from a server-accessible file. Of course > it's > > not as fast as reading a file from a local SCSI array, but sending > data > > to copy COPY over a network wire is way faster than any other > remote > > technique I've found. > > > > I'm happy to post this code to the list or to the code maintainer > if > > it's of interest. Just let me know! > > > > Sincerely, > > > > Steve > > > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > From anthonyeden at gmail.com Sat Dec 22 20:45:21 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Sat, 22 Dec 2007 20:45:21 -0500 Subject: [Activewarehouse-discuss] Adapter Extensions Tests Failing In-Reply-To: References: Message-ID: Unfortunately it looks like there was some change in 2.0 (I haven't tracked it down yet as to when the change occurred) however the private connect method in AdapterExtensions mysql_adapter.rb is no longer being called. If I patch Rails with the attached patch then I can execute the tests without any problems. If anyone has any good ideas on how to properly override the connect method of mysql_adapter.rb then please feel free to speak up. V/r Anthony Eden On Dec 20, 2007 1:39 PM, Anthony Eden wrote: > It might be that the Rails version is using the mysql adapter that > comes with Rails, which is a pure Ruby implementation that has a > maximum limit to the number of bytes it can ship over the wire. > > This problem has come up a couple of times, so I finally pulled the > content from emails I had sent to individuals and posted in on my > blog: > > http://anthonyeden.com/2007/12/20/mysql-bulk-load-with-pure-mysql-driver-in-rails > > V/r > Anthony Eden > > > On 12/20/07, Brandon Dimcheff wrote: > > I'm trying to add a couple features to the the adapter extensions, but > > I can't seem to get a few of the tests to run. For instance, > > test_bulk_load throws: > > > > Mysql::Error: The used command is not allowed with this MySQL version: > > LOAD DATA LOCAL INFILE './test/integration/people.txt' INTO TABLE people > > > > According to pretty much whatever I've read, this is caused by local- > > infile being disabled on either the client or server side. I can run > > the query in the mysql command line client, so it must be something > > fishy with ActiveRecord. The strange thing is, I can run that same > > query in script/console in my activewarehouse-based project and it > > executes just fine. I just can't seem to figure out what's different > > about the way ActiveRecord is being used in these tests vs. in a rails > > app... > > > > Any ideas? > > > > Thanks, > > Brandon > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: mysql_adapter_opt_local_infile.diff Type: application/octet-stream Size: 779 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071222/f3de4995/attachment.obj From anthonyeden at gmail.com Sat Dec 22 21:00:07 2007 From: anthonyeden at gmail.com (Anthony Eden) Date: Sat, 22 Dec 2007 21:00:07 -0500 Subject: [Activewarehouse-discuss] Newbie needs help! In-Reply-To: <228840e70712170950ie9f9874ybaa505d14e88c64c@mail.gmail.com> References: <001601c83de0$122df6f0$3689e4d0$@com> <001c01c83ded$5a701ee0$0f505ca0$@com> <002001c83df7$a986cf60$fc946e20$@com> <228840e70712170950ie9f9874ybaa505d14e88c64c@mail.gmail.com> Message-ID: I'm not sure if there's a way to force it to use the old AR version without actually patching it. V/r Anthony On Dec 17, 2007 12:50 PM, Enderson Maia wrote: > I have the same problem with MacOS, but I think it's 'cause teh aw-etl is > using ar-2.0.1, but it worked with 1.15.6 > > How can I force it to use ar-1.15.6 ? > > > > On Dec 13, 2007 11:18 PM, Paul Cortens < phylae at gmail.com> wrote: > > That was it. I was using the ruby version. Once I installed the win32 > > version it works. > > > > THANK YOU!!!! > > > > Paul > > > > -----Original Message----- > > From: Chris Williams [mailto:chris.d.williams at gmail.com] > > Sent: Thursday, December 13, 2007 6:01 PM > > To: Paul Cortens > > Subject: Re: [Activewarehouse-discuss] Newbie needs help! > > > > Which MySQL gem are you using? I think you want to make sure you are > > using the win32 version, not the ruby version. > > > > On Dec 13, 2007 8:05 PM, Paul Cortens < phylae at gmail.com> wrote: > > > Thanks. If I add "LINES TERMINATED BY '\r\n'" to the MySQL query, then > I > > > don't have the line break problem. But I am still stuck as to why > > > ActiveWarehouse chokes, but I can manually run the exact same SQL code > > > through MySQL manually. > > > > > > Cheers, > > > Paul > > > > > > > > > -----Original Message----- > > > From: Chris Williams [mailto: chris.d.williams at gmail.com] > > > Sent: Thursday, December 13, 2007 3:46 PM > > > To: Paul Cortens > > > Subject: Re: [Activewarehouse-discuss] Newbie needs help! > > > > > > Hi Paul. I ran in to some of the same problems when running on > > > Windows. I went through some of my old posts to the list but can't > > > find exactly what I did. I know I made some code changes to AW in > > > order to get rid of the \n in the last field in the CSV file. I will > > > keep searching and see if I can help out at all. > > > > > > Chris > > > > > > > > > > > > On Dec 13, 2007 6:29 PM, Paul Cortens < phylae at gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > I have been trying to work through the tutorial posted > > > here, > > > > and I haven't been having much luck. > > > > > > > > > > > > > > > > > > > > > > > > When I try to run author_dimension.ctl I get the following error. > > > > > > > > > > > > > > > > ****************** > > > > > > > > > > > > > > > > F:\ruby_testing\DW\rails_warehouse\db\etl>etl author_dimension.ctl > > > > > > > > Using AdapterExtensions > > > > > > > > Starting ETL process > > > > > > > > initializing ETL engine > > > > > > > > Processing author_dimension.ctl > > > > > > > > > > > > > > > > Executing screens > > > > > > > > Screens passed > > > > > > > > > > > > > > > > Executing post processes > > > > > > > > > > > > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record/connecti > > > on_a > > > > > > > > dapters/abstract_adapter.rb:150:in `log': Mysql::Error: #42000The used > > > > command i > > > > > > > > s not allowed with this MySQL version: LOAD DATA LOCAL INFILE > > > > './author_dimensio > > > > > > > > n.txt' INTO TABLE author_dimension FIELDS TERMINATED BY ',' (name) > > > > (ActiveRecord > > > > > > > > ::StatementInvalid) > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > > > > > /connection_adapters/mysql_adapter.rb:281:in `execute' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > > > > > _extensions/connection_adapters/mysql_adapter.rb:46:in `do_bulk_load' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter > > > > > > > > _extensions/connection_adapters/abstract_adapter.rb:18:in `bulk_load' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse- etl-0.9.0/bin/../lib > > > > > > > > /etl/processor/bulk_import_processor.rb:68:in `process' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.1/lib/active_record > > > > > > > > /connection_adapters/abstract/database_statements.rb:66:in > `transaction' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > > > /etl/processor/bulk_import_processor.rb:58:in `process' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > > > /etl/engine.rb:485:in `post_process' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib > > > > > > > > /etl/engine.rb:484:in `each' > > > > > > > > ... 13 levels... > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.1/lib/active_suppo > > > > > > > > rt/dependencies.rb:496:in `require' > > > > > > > > from > > > > c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28 > > > > > > > > > > > > > > > > from c:/ruby/bin/etl:16:in `load' > > > > > > > > from c:/ruby/bin/etl:16 > > > > > > > > > > > > > > > > > > > > > > > > ****************** > > > > > > > > > > > > > > > > > > > > > > > > This is my environment: > > > > > > > > Windows XP Pro SP2. > > > > > > > > rails -v => 2.0.1 > > > > > > > > ruby -v => ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32] > > > > > > > > gem -v => 0.9.4 > > > > > > > > MySQL 'select @@version;' => 5.0.27-community-nt > > > > > > > > > > > > > > > > > > > > > > > > If I run the command "LOAD DATA LOCAL INFILE './author_dimension.txt' > > INTO > > > > TABLE author_dimension FIELDS TERMINATED BY ',' (name);" directly in > > > MySQL, > > > > then the data imports just fine. (Actually it is not completely fine, > it > > I > > > > get line breaks at the end of some of the strings.) > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Paul > > > > > > > > > > > > _______________________________________________ > > > > Activewarehouse-discuss mailing list > > > > Activewarehouse-discuss at rubyforge.org > > > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > > > > > > > > > > > > > _______________________________________________ > > Activewarehouse-discuss mailing list > > Activewarehouse-discuss at rubyforge.org > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > > > > > -- > Enderson Maia > _______________________________________________ > Activewarehouse-discuss mailing list > Activewarehouse-discuss at rubyforge.org > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss > > From bdimchef at wieldim.com Sun Dec 23 09:55:19 2007 From: bdimchef at wieldim.com (Brandon Dimcheff) Date: Sun, 23 Dec 2007 09:55:19 -0500 Subject: [Activewarehouse-discuss] Adapter Extensions Tests Failing In-Reply-To: References: Message-ID: Hm, yeah. That's pretty much exactly what I ended up doing. I actually ended up patching ActiveRecord to allow me to specify a my.cnf file in my database.yml so I could set *any* MySQL option and not just OPT_LOCAL_INFILE. In my my.cnf I have local-infile under [client]. Everything seems to work fine now. It seems pretty silly that ActiveRecord doesn't allow you to set the low-level options on the MySQL adapter... See attached patch. About getting your patch of connect to be executed... At first glance, it looks like it should work just fine. The other methods are being defined properly and all... Maybe it's loading *before* the actual MysqlAdapter so connect gets overridden by the one from ActiveRecord? - Brandon On Dec 22, 2007, at 20:45, Anthony Eden wrote: > Unfortunately it looks like there was some change in 2.0 (I haven't > tracked it down yet as to when the change occurred) however the > private connect method in AdapterExtensions mysql_adapter.rb is no > longer being called. If I patch Rails with the attached patch then I > can execute the tests without any problems. If anyone has any good > ideas on how to properly override the connect method of > mysql_adapter.rb then please feel free to speak up. > > V/r > Anthony Eden > > On Dec 20, 2007 1:39 PM, Anthony Eden wrote: >> It might be that the Rails version is using the mysql adapter that >> comes with Rails, which is a pure Ruby implementation that has a >> maximum limit to the number of bytes it can ship over the wire. >> >> This problem has come up a couple of times, so I finally pulled the >> content from emails I had sent to individuals and posted in on my >> blog: >> >> http://anthonyeden.com/2007/12/20/mysql-bulk-load-with-pure-mysql-driver-in-rails >> >> V/r >> Anthony Eden >> >> >> On 12/20/07, Brandon Dimcheff wrote: >>> I'm trying to add a couple features to the the adapter extensions, >>> but >>> I can't seem to get a few of the tests to run. For instance, >>> test_bulk_load throws: >>> >>> Mysql::Error: The used command is not allowed with this MySQL >>> version: >>> LOAD DATA LOCAL INFILE './test/integration/people.txt' INTO TABLE >>> people >>> >>> According to pretty much whatever I've read, this is caused by >>> local- >>> infile being disabled on either the client or server side. I can >>> run >>> the query in the mysql command line client, so it must be something >>> fishy with ActiveRecord. The strange thing is, I can run that same >>> query in script/console in my activewarehouse-based project and it >>> executes just fine. I just can't seem to figure out what's >>> different >>> about the way ActiveRecord is being used in these tests vs. in a >>> rails >>> app... >>> >>> Any ideas? >>> >>> Thanks, >>> Brandon >>> _______________________________________________ >>> Activewarehouse-discuss mailing list >>> Activewarehouse-discuss at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss >>> >> > -------------- next part -------------- A non-text attachment was scrubbed... Name: config_file_support.patch Type: application/octet-stream Size: 1402 bytes Desc: not available Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071223/04c0adf5/attachment-0001.obj From phylae at gmail.com Sat Dec 29 07:20:53 2007 From: phylae at gmail.com (Paul Cortens) Date: Sat, 29 Dec 2007 04:20:53 -0800 Subject: [Activewarehouse-discuss] Current User in ctl file? Message-ID: Hi, I have been trying to use RESTful Authentication with ActiveWarehouse ETL. To put it simply, I want to access the id of the currently logged in user during the ETL. I would like it to populate a user_id column. I have a controller which runs bb = IO.popen("cd db/etl \n etl trans.ctl") b = bb.readlines puts b.join This works well enough (for now). The user can click a button and trans.ctlwill be executed. With RESTful Authentication, I can use the current_user method in my views to access the current user. I would like something similar for the control files. I am thinking that passing an argument/parameter would be the best way to do this. Any suggestions? Paul PS. I have also tried using the following in my controller: ETL::Engine.realtime_activity = true ETL::Engine.process('db/etl/trans.ctl') This sort of works. The ctl is executed, but it can't find database.ymlbecause the current directory is not db/etl. Even with this method, I cannot access current_user or session[:user]. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071229/de4239d4/attachment.html From thibaut.barrere at gmail.com Sat Dec 29 08:05:54 2007 From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=) Date: Sat, 29 Dec 2007 14:05:54 +0100 Subject: [Activewarehouse-discuss] Current User in ctl file? In-Reply-To: References: Message-ID: <4a68b8cf0712290505g28b125e7yb48ff687ed1914cc@mail.gmail.com> Hi Paul, > I am thinking that passing an argument/parameter would be the best way to do > this. the easiest way to pass variables to a ctl file is through ENV['myvar'] ; see http://rubyforge.org/pipermail/activewarehouse-discuss/2007-October/000184.html for a previous discussion on this topic. > Any suggestions? Yep - take care of concurrency issues if multiple users achieve a request at the same time, or things won't scale properly. One solution could be to: - do the processing for all users as part of a nightly process - and just achieve a request to the database, filtering by user_id Thibaut -- http://evolvingworker : tools for a better day From phylae at gmail.com Mon Dec 31 05:45:41 2007 From: phylae at gmail.com (Paul Cortens) Date: Mon, 31 Dec 2007 02:45:41 -0800 Subject: [Activewarehouse-discuss] Current User in ctl file? In-Reply-To: <4a68b8cf0712290505g28b125e7yb48ff687ed1914cc@mail.gmail.com> References: <4a68b8cf0712290505g28b125e7yb48ff687ed1914cc@mail.gmail.com> Message-ID: Thanks for the reply! I don't think ENV['my_var'] will work in this case. I definitely need to allow different users to submit requests at the same time. Here is what I am doing now: controller: ctl_file = 'db/etl/trans.ctl' usr_ctl_file = "db/etl/trans[#{current_user.id.to_s}].ctl" File.copy(ctl_file, usr_ctl_file) puts 'Current user is ' + current_user.id.to_s + ' (controller)' ETL::Engine.realtime_activity = true ETL::Engine.process(usr_ctl_file) ctl: #parse the ctl file name and get the user id u = __FILE__.scan(/\[[0-9]+\]/)[0].gsub(/[\[\]]/,"").to_i puts 'Current user is: ' + u.to_s + ' (CTL file)' It "works", but it isn't complete. I still need to set up the input and temporary files to include the user id too. I also need to clean up all these extra files. This approach seems very crude. I was hoping that I could start the etl by just calling a method such as MyCtl.start(user_id). (I don't mean this as a criticism. I am far too new to ActiveWarehouse to criticize any design decisions.) Thanks for the help! Paul On Dec 29, 2007 5:05 AM, Thibaut Barr?re wrote: > Hi Paul, > > > I am thinking that passing an argument/parameter would be the best way > to do > > this. > > the easiest way to pass variables to a ctl file is through > ENV['myvar'] ; see > > http://rubyforge.org/pipermail/activewarehouse-discuss/2007-October/000184.html > for a previous discussion on this topic. > > > Any suggestions? > > Yep - take care of concurrency issues if multiple users achieve a > request at the same time, or things won't scale properly. One solution > could be to: > - do the processing for all users as part of a nightly process > - and just achieve a request to the database, filtering by user_id > > Thibaut > -- > http://evolvingworker : tools for a better day > -- Ezk. 36 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20071231/7c4fb743/attachment.html