, i made this
script (posted here for future reference)

$ cd activewarehouse/etl/trunk
$ find ./lib | egrep ".rb$"| xargs  sed -r -e  "s/<(\/?)div/<$1tt/g" -i

/Jesper

PS i hope you find the rdoc additions helpful and i also hope that you
will review it to make sure it is described the way the code works :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: documentation_patch_03.patch
Type: application/octet-stream
Size: 28928 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080214/71f5f3d0/attachment-0001.obj 

From jesperrr at gmail.com  Thu Feb 14 09:45:02 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Thu, 14 Feb 2008 15:45:02 +0100
Subject: [Activewarehouse-discuss] activewarehouse-etl how to increase log
	level
Message-ID: 

How do i set log level in a .ctl file? what are my options for setting it there?

I looked briefly at ETL::Engine and i think it's hardcoded here? But I
may be wrong?

/Jesper

-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
___________________________________________________________

From jesperrr at gmail.com  Fri Feb 15 18:25:22 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Sat, 16 Feb 2008 00:25:22 +0100
Subject: [Activewarehouse-discuss] Activewarehouse-ETL userguide update
Message-ID: 

Tiny update for the Activewarehouse-ETL userguide.

-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
jesperrr at gmail.com (Private e-mail and Google Talk IM)
___________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/e9816b75/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: userguide-etl-patch.patch
Type: application/octet-stream
Size: 2478 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/e9816b75/attachment.obj 

From jesperrr at gmail.com  Fri Feb 15 18:33:41 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Sat, 16 Feb 2008 00:33:41 +0100
Subject: [Activewarehouse-discuss] rails_warehouse demo project and Rails
	2.0 errors
Message-ID: 

On my Windows XP machine, the rails_warehouse demo project would not start
up. In order to update scripts, I used these steps:

1) added Rails edge to /vendor/rails (rake rails:freeze:edge)

2) ran rake rails:scripts:update

This updated /config/boot.rb to the following (see below the signature). I
also added
/config/initializers/inflections.rb
/config/initializers/mime_types.rb
/script/performance/request


PS. I'm attaching as a patch to this mail
-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
jesperrr at gmail.com (Private e-mail and Google Talk IM)
___________________________________________________________

# Don't change this file!
# Configure your app in config/environment.rb and config/environments/*.rb

RAILS_ROOT = "#{File.dirname(__FILE__)}/.." unless defined?(RAILS_ROOT)

module Rails
  class << self
    def boot!
      unless booted?
        preinitialize
        pick_boot.run
      end
    end

    def booted?
      defined? Rails::Initializer
    end

    def pick_boot
      (vendor_rails? ? VendorBoot : GemBoot).new
    end

    def vendor_rails?
      File.exist?("#{RAILS_ROOT}/vendor/rails")
    end

    # FIXME : Ruby 1.9
    def preinitialize
      load(preinitializer_path) if File.exists?(preinitializer_path)
    end

    def preinitializer_path
      "#{RAILS_ROOT}/config/preinitializer.rb"
    end
  end

  class Boot
    def run
      load_initializer
      Rails::Initializer.run(:set_load_path)
    end
  end

  class VendorBoot < Boot
    def load_initializer
      require "#{RAILS_ROOT}/vendor/rails/railties/lib/initializer"
    end
  end

  class GemBoot < Boot
    def load_initializer
      self.class.load_rubygems
      load_rails_gem
      require 'initializer'
    end

    def load_rails_gem
      if version = self.class.gem_version
        gem 'rails', version
      else
        gem 'rails'
      end
    rescue Gem::LoadError => load_error
      $stderr.puts %(Missing the Rails #{version} gem. Please `gem install
-v=#{version} rails`, update your RAILS_GEM_VERSION setting in
config/environment.rb for the Rails version you do have installed, or
comment out RAILS_GEM_VERSION to use the latest version installed.)
      exit 1
    end

    class << self
      def rubygems_version
        Gem::RubyGemsVersion if defined? Gem::RubyGemsVersion
      end

      def gem_version
        if defined? RAILS_GEM_VERSION
          RAILS_GEM_VERSION
        elsif ENV.include?('RAILS_GEM_VERSION')
          ENV['RAILS_GEM_VERSION']
        else
          parse_gem_version(read_environment_rb)
        end
      end

      def load_rubygems
        require 'rubygems'

        unless rubygems_version >= '0.9.4'
          $stderr.puts %(Rails requires RubyGems >= 0.9.4 (you have
#{rubygems_version}). Please `gem update --system` and try again.)
          exit 1
        end

      rescue LoadError
        $stderr.puts %(Rails requires RubyGems >= 0.9.4. Please install
RubyGems and try again: http://rubygems.rubyforge.org)
        exit 1
      end

      def parse_gem_version(text)
        $1 if text =~
/^[^#]*RAILS_GEM_VERSION\s*=\s*["']([!~<>=]*\s*[\d.]+)["']/
      end

      private
        def read_environment_rb
          File.read("#{RAILS_ROOT}/config/environment.rb")
        end
    end
  end
end

# All that for this:
Rails.boot!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/7e31ea4d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rails_warehouse_rails2.0.patch
Type: application/octet-stream
Size: 7078 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/7e31ea4d/attachment-0001.obj 

From jesperrr at gmail.com  Fri Feb 15 18:52:22 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Sat, 16 Feb 2008 00:52:22 +0100
Subject: [Activewarehouse-discuss] problems using bulk_import (to mysql on
	win xp)
Message-ID: 

Hi there
I have been trying for days to get bulk import working with
Activewarehouse-ETL
Here is my rule in employees.ctl:

post_process :bulk_import, {
  :file     =>  'output/employees.out.txt',
  :columns  =>  EMPLOYEE_COLS,
  :truncate =>  true,
  :field_separator =>   ',',
  :field_enclosure =>  '"',
  #:line_separator  =>  '\n',
  :ignore   =>  1,
  :target   =>  :development,
  :table    =>  'employees'
}

====This is the error I get when running:====

> C:\Documents and Settings\Administrator\My
> Documents\rails\ourpeople\import>etl -c ../config/database.yml --limit 200
> employees.ctl
> Using AdapterExtensions
> Starting ETL process
> initializing ETL engine
> Processing employees.ctl
> Source: employees.csv
> Limiting enabled: 200
> Reached limit of 200
>
> Executing screens
> Screens passed
>
> Executing post processes
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
> `log': Mysql::Error: #42000The used command is not allowed with this MySQL
> version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My
> Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE
> employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES
> (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id)
> (ActiveRecord::StatementInvalid)
>     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in
> `execute'
>     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:46:in
> `do_bulk_load'
>     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in
> `bulk_load'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in
> `process'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in
> `transaction'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in
> `process'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in
> `post_process'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in
> `each'
>      ... 13 levels...
>     from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> `require'
>     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0
> /bin/etl:28
>     from c:/ruby/bin/etl:19:in `load'
>     from c:/ruby/bin/etl:19
>
====end of error====

I have tried a lot of different combinations of flags in that bulk_import
directive. Am i getting it right? Could you please show me some working
examples.

Also, for adapter_extensions there seems to be a patch adding this line to
the mysql adapter

> --- activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> (revision 8484)
> +++ activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> (working copy)
> @@ -467,6 +467,7 @@
>            if encoding
>              @connection.options(Mysql::SET_CHARSET_NAME, encoding) rescue
> nil
>            end
> +          @connection.options(Mysql::OPT_LOCAL_INFILE, true)
>            @connection.ssl_set(@config[:sslkey], @config[:sslcert],
> @config[:sslca], @config[:sslcapath], @config[:sslcipher]) if
> @config[:sslkey]
>            @connection.real_connect(*@connection_options)
>            execute("SET NAMES '#{encoding}'") if encoding
>


Well, i did not apply the patch to version 0.4.0 because it seemed like it
was already in the code (however, not "true" but "1" at the
end of the line)

The most odd thing is that I can easily run a bulk import via the local sql
command:

> /ourpeople/import/>
> $ mysql ourpeople_development < import.sql --user=root -p
>

Where the import.sql contains

> LOAD DATA LOCAL INFILE './output/employees.out.txt'     INTO TABLE
> employees     CHARACTER SET latin1 FIELDS TERMINATED BY ','  IGNORE 1 LINES
> ;


Any suggestions/ideas to what i might be doing wrong? Any examples of
working code you can share?


-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
jesperrr at gmail.com (Private e-mail and Google Talk IM)
___________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/c9dcc378/attachment.html 

From mghaught at gmail.com  Fri Feb 15 18:58:58 2008
From: mghaught at gmail.com (Marty Haught)
Date: Fri, 15 Feb 2008 16:58:58 -0700
Subject: [Activewarehouse-discuss] problems using bulk_import (to mysql
	on win xp)
In-Reply-To: 
References: 
Message-ID: <57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>

Hi Jesper,

My understanding is that this is an issue with how ActiveRecord works
with MySQL.  I believe there was a change in how MySQL did bulk update
and that ActiveRecord's way of calling this no longer works.  Sorry, I
don't have more technical information on the issue.  Our work around
has been to turn off bulk uploads.  Maybe someone else who has more
detailed information on this can speak up.

Cheers,
Marty

On Fri, Feb 15, 2008 at 4:52 PM, Jesper R?nn-Jensen  wrote:
> Hi there
> I have been trying for days to get bulk import working with
> Activewarehouse-ETL
> Here is my rule in employees.ctl:
>
> post_process :bulk_import, {
>   :file     =>  'output/employees.out.txt',
>    :columns  =>  EMPLOYEE_COLS,
>   :truncate =>  true,
>   :field_separator =>   ',',
>   :field_enclosure =>  '"',
>   #:line_separator  =>  '\n',
>   :ignore   =>  1,
>    :target   =>  :development,
>   :table    =>  'employees'
> }
>
> ====This is the error I get when running:====
> > C:\Documents and Settings\Administrator\My
> Documents\rails\ourpeople\import>etl -c ../config/database.yml --limit 200
> employees.ctl
> > Using AdapterExtensions
> > Starting ETL process
> > initializing ETL engine
> > Processing employees.ctl
> > Source: employees.csv
> > Limiting enabled: 200
> > Reached limit of 200
> >
> > Executing screens
> > Screens passed
> >
> > Executing post processes
> >
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
> `log': Mysql::Error: #42000The used command is not allowed with this MySQL
> version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My
> Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE
> employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES
> (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id)
> (ActiveRecord::StatementInvalid)
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in
> `execute'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:46:in
> `do_bulk_load'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in
> `bulk_load'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in
> `process'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in
> `transaction'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in
> `process'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in
> `post_process'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in
> `each'
> >      ... 13 levels...
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> `require'
> >     from
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28
> >     from c:/ruby/bin/etl:19:in `load'
> >     from c:/ruby/bin/etl:19
> >
> ====end of error====
>
> I have tried a lot of different combinations of flags in that bulk_import
> directive. Am i getting it right? Could you please show me some working
> examples.
>
> Also, for adapter_extensions there seems to be a patch adding this line to
> the mysql adapter
> >
> > --- activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> (revision 8484)
> > +++ activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> (working copy)
> > @@ -467,6 +467,7 @@
> >            if encoding
> >              @connection.options(Mysql::SET_CHARSET_NAME, encoding) rescue
> nil
> >            end
> > +          @connection.options(Mysql::OPT_LOCAL_INFILE, true)
> >            @connection.ssl_set(@config[:sslkey], @config[:sslcert],
> @config[:sslca], @config[:sslcapath], @config[:sslcipher]) if
> @config[:sslkey]
> >            @connection.real_connect(*@connection_options)
> >            execute("SET NAMES '#{encoding}'") if encoding
> >
>
>
> Well, i did not apply the patch to version 0.4.0 because it seemed like it
> was already in the code (however, not "true" but "1" at the
>  end of the line)
>
> The most odd thing is that I can easily run a bulk import via the local sql
> command:
> > /ourpeople/import/>
> > $ mysql ourpeople_development < import.sql --user=root -p
> >
>
> Where the import.sql contains
> > LOAD DATA LOCAL INFILE './output/employees.out.txt'     INTO TABLE
> employees     CHARACTER SET latin1 FIELDS TERMINATED BY ','  IGNORE 1 LINES
> ;
>
> Any suggestions/ideas to what i might be doing wrong? Any examples of
> working code you can share?
>
>
> --
> ___________________________________________________________
>
> Jesper R?nn-Jensen / Capgemini Denmark
> Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
> www.dk.capgemini.com / Blog http://justaddwater.dk/
>  jesperrr at gmail.com (Private e-mail and Google Talk IM)
> ___________________________________________________________
> _______________________________________________
>  Activewarehouse-discuss mailing list
>  Activewarehouse-discuss at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>

From phylae at gmail.com  Fri Feb 15 19:06:19 2008
From: phylae at gmail.com (Paul Cortens)
Date: Fri, 15 Feb 2008 16:06:19 -0800
Subject: [Activewarehouse-discuss] problems using bulk_import (to mysql
	on win xp)
In-Reply-To: 
References: 
Message-ID: 

are you using the Ruby MySQL gem or the native gem?

I had the exact same error when using the Ruby gem.

Paul

On Feb 15, 2008 3:52 PM, Jesper R?nn-Jensen  wrote:

> Hi there
> I have been trying for days to get bulk import working with
> Activewarehouse-ETL
> Here is my rule in employees.ctl:
>
> post_process :bulk_import, {
>   :file     =>  'output/employees.out.txt',
>   :columns  =>  EMPLOYEE_COLS,
>   :truncate =>  true,
>   :field_separator =>   ',',
>   :field_enclosure =>  '"',
>   #:line_separator  =>  '\n',
>   :ignore   =>  1,
>   :target   =>  :development,
>   :table    =>  'employees'
> }
>
> ====This is the error I get when running:====
>
> > C:\Documents and Settings\Administrator\My
> > Documents\rails\ourpeople\import>etl -c ../config/database.yml --limit 200
> > employees.ctl
> > Using AdapterExtensions
> > Starting ETL process
> > initializing ETL engine
> > Processing employees.ctl
> > Source: employees.csv
> > Limiting enabled: 200
> > Reached limit of 200
> >
> > Executing screens
> > Screens passed
> >
> > Executing post processes
> > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
> > `log': Mysql::Error: #42000The used command is not allowed with this MySQL
> > version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My
> > Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE
> > employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES
> > (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id)
> > (ActiveRecord::StatementInvalid)
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in
> > `execute'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:46:in
> > `do_bulk_load'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in
> > `bulk_load'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in
> > `process'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in
> > `transaction'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in
> > `process'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in
> > `post_process'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in
> > `each'
> >      ... 13 levels...
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> > `require'
> >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0
> > /bin/etl:28
> >     from c:/ruby/bin/etl:19:in `load'
> >     from c:/ruby/bin/etl:19
> >
> ====end of error====
>
> I have tried a lot of different combinations of flags in that bulk_import
> directive. Am i getting it right? Could you please show me some working
> examples.
>
> Also, for adapter_extensions there seems to be a patch adding this line to
> the mysql adapter
>
> > ---
> > activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> > (revision 8484)
> > +++
> > activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> > (working copy)
> > @@ -467,6 +467,7 @@
> >            if encoding
> >              @connection.options(Mysql::SET_CHARSET_NAME, encoding)
> > rescue nil
> >            end
> > +          @connection.options(Mysql::OPT_LOCAL_INFILE, true)
> >            @connection.ssl_set(@config[:sslkey], @config[:sslcert],
> > @config[:sslca], @config[:sslcapath], @config[:sslcipher]) if
> > @config[:sslkey]
> >            @connection.real_connect(*@connection_options)
> >            execute("SET NAMES '#{encoding}'") if encoding
> >
>
>
> Well, i did not apply the patch to version 0.4.0 because it seemed like it
> was already in the code (however, not "true" but "1" at the
> end of the line)
>
> The most odd thing is that I can easily run a bulk import via the local
> sql command:
>
> > /ourpeople/import/>
> > $ mysql ourpeople_development < import.sql --user=root -p
> >
>
> Where the import.sql contains
>
> > LOAD DATA LOCAL INFILE './output/employees.out.txt'     INTO TABLE
> > employees     CHARACTER SET latin1 FIELDS TERMINATED BY ','  IGNORE 1 LINES
> > ;
>
>
> Any suggestions/ideas to what i might be doing wrong? Any examples of
> working code you can share?
>
>
> --
> ___________________________________________________________
>
> Jesper R?nn-Jensen / Capgemini Denmark
> Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
> www.dk.capgemini.com / Blog http://justaddwater.dk/
> jesperrr at gmail.com (Private e-mail and Google Talk IM)
> ___________________________________________________________
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>


-- 


Ezk. 36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080215/a2385470/attachment-0001.html 

From phylae at gmail.com  Fri Feb 15 19:09:00 2008
From: phylae at gmail.com (Paul Cortens)
Date: Fri, 15 Feb 2008 16:09:00 -0800
Subject: [Activewarehouse-discuss] problems using bulk_import (to mysql
	on win xp)
In-Reply-To: 
References: 
	
Message-ID: 

Check out this:
http://rubyforge.org/pipermail/activewarehouse-discuss/2007-December/000224.html


On Feb 15, 2008 4:06 PM, Paul Cortens  wrote:

> are you using the Ruby MySQL gem or the native gem?
>
> I had the exact same error when using the Ruby gem.
>
> Paul
>
> On Feb 15, 2008 3:52 PM, Jesper R?nn-Jensen  wrote:
>
> > Hi there
> > I have been trying for days to get bulk import working with
> > Activewarehouse-ETL
> > Here is my rule in employees.ctl:
> >
> > post_process :bulk_import, {
> >   :file     =>  'output/employees.out.txt',
> >   :columns  =>  EMPLOYEE_COLS,
> >   :truncate =>  true,
> >   :field_separator =>   ',',
> >   :field_enclosure =>  '"',
> >   #:line_separator  =>  '\n',
> >   :ignore   =>  1,
> >   :target   =>  :development,
> >   :table    =>  'employees'
> > }
> >
> > ====This is the error I get when running:====
> >
> > > C:\Documents and Settings\Administrator\My
> > > Documents\rails\ourpeople\import>etl -c ../config/database.yml --limit 200
> > > employees.ctl
> > > Using AdapterExtensions
> > > Starting ETL process
> > > initializing ETL engine
> > > Processing employees.ctl
> > > Source: employees.csv
> > > Limiting enabled: 200
> > > Reached limit of 200
> > >
> > > Executing screens
> > > Screens passed
> > >
> > > Executing post processes
> > > c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
> > > `log': Mysql::Error: #42000The used command is not allowed with this MySQL
> > > version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My
> > > Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE
> > > employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES
> > > (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id)
> > > (ActiveRecord::StatementInvalid)
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in
> > > `execute'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:46:in
> > > `do_bulk_load'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in
> > > `bulk_load'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in
> > > `process'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in
> > > `transaction'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in
> > > `process'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in
> > > `post_process'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in
> > > `each'
> > >      ... 13 levels...
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
> > > `require'
> > >     from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0
> > > /bin/etl:28
> > >     from c:/ruby/bin/etl:19:in `load'
> > >     from c:/ruby/bin/etl:19
> > >
> > ====end of error====
> >
> > I have tried a lot of different combinations of flags in that
> > bulk_import directive. Am i getting it right? Could you please show me some
> > working examples.
> >
> > Also, for adapter_extensions there seems to be a patch adding this line
> > to the mysql adapter
> >
> > > ---
> > > activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> > > (revision 8484)
> > > +++
> > > activerecord/lib/active_record/connection_adapters/mysql_adapter.rb
> > > (working copy)
> > > @@ -467,6 +467,7 @@
> > >            if encoding
> > >              @connection.options(Mysql::SET_CHARSET_NAME, encoding)
> > > rescue nil
> > >            end
> > > +          @connection.options(Mysql::OPT_LOCAL_INFILE, true)
> > >            @connection.ssl_set(@config[:sslkey], @config[:sslcert],
> > > @config[:sslca], @config[:sslcapath], @config[:sslcipher]) if
> > > @config[:sslkey]
> > >            @connection.real_connect(*@connection_options)
> > >            execute("SET NAMES '#{encoding}'") if encoding
> > >
> >
> >
> > Well, i did not apply the patch to version 0.4.0 because it seemed like
> > it was already in the code (however, not "true" but "1" at the
> > end of the line)
> >
> > The most odd thing is that I can easily run a bulk import via the local
> > sql command:
> >
> > > /ourpeople/import/>
> > > $ mysql ourpeople_development < import.sql --user=root -p
> > >
> >
> > Where the import.sql contains
> >
> > > LOAD DATA LOCAL INFILE './output/employees.out.txt'     INTO TABLE
> > > employees     CHARACTER SET latin1 FIELDS TERMINATED BY ','  IGNORE 1 LINES
> > > ;
> >
> >
> > Any suggestions/ideas to what i might be doing wrong? Any examples of
> > working code you can share?
> >
> >
> > --
> > ___________________________________________________________
> >
> > Jesper R?nn-Jensen / Capgemini Denmark
> > Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
> > www.dk.capgemini.com / Blog http://justaddwater.dk/
> > jesperrr at gmail.com (Private e-mail and Google Talk IM)
> > ___________________________________________________________
> > _______________________________________________
> > Activewarehouse-discuss mailing list
> > Activewarehouse-discuss at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
> >
>
>
> --
>
>
> Ezk. 36




-- 


Ezk. 36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080215/f486ec81/attachment.html 

From mghaught at gmail.com  Fri Feb 15 19:11:20 2008
From: mghaught at gmail.com (Marty Haught)
Date: Fri, 15 Feb 2008 17:11:20 -0700
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import (to mysql
	on win xp)
In-Reply-To: 
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
Message-ID: <57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>

---------- Forwarded message ----------
From: Chris Williams 
Date: Fri, Feb 15, 2008 at 5:09 PM
Subject: Re: [Activewarehouse-discuss] problems using bulk_import (to
mysql on win xp)
To: Marty Haught 


I just did a gem list on my windows box and this is what it showed for mysql

 mysql (2.7.1)
    A win32-native build of the MySQL API module for Ruby.

 I would uninstall the mysql gem and reinstall to make sure you are
 using the win32 version.

 Chris

From chris.d.williams at gmail.com  Fri Feb 15 19:52:51 2008
From: chris.d.williams at gmail.com (Chris Williams)
Date: Fri, 15 Feb 2008 19:52:51 -0500
Subject: [Activewarehouse-discuss] Anyone using AW-ETL with JRuby
Message-ID: 

I am trying to get JRuby to work with some simple etl scripts and I am
having issues.  A few questions...

* I am using the trunk of AW-ETL
* What version of active record should I be using?  Right now I am at 2.0.2.
* I am trying JRuby 1.1RC1
* I updated the etl script to add the following so it would find my jdbc drivers

if RUBY_PLATFORM =~ /java/
   require 'rubygems'
   gem 'activerecord-jdbc-adapter'
   require 'jdbc_adapter'
end

Below is the error I am seeing
jruby -S etl obs_dimension.ctl
Using AdapterExtensions
C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:263:in
`load_m
issing_constant': uninitialized constant ActiveRecord::Base (NameError)
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:2
53:in `const_missing'
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.1/bin/../lib/etl/engine.rb:3
        from C:/projects/jruby-1.1RC1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27
        from C:/projects/jruby-1.1RC1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from C:/projects/jruby-1.1RC1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:4
96:in `new_constants_in'
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:3
18:in `require'
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:4
96
        from C:/projects/jruby-1.1RC1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from C:/projects/jruby-1.1RC1/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in
`require'
        from C:/projects/jruby-1.1RC1/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.1/bin/etl:34
        from C:/projects/jruby-1.1RC1/bin/etl:19:in `load'
        from C:/projects/jruby-1.1RC1/bin/etl:19
        from file:/C:/projects/jruby-1.1RC1/lib/jruby.jar!/jruby/commands.rb:14:in
`load'
        from file:/C:/projects/jruby-1.1RC1/lib/jruby.jar!/jruby/commands.rb:14:in
`etl'
        from file:/C:/projects/jruby-1.1RC1/lib/jruby.jar!/jruby/commands.rb:11

Thanks
CW

From phylae at gmail.com  Tue Feb 19 22:07:41 2008
From: phylae at gmail.com (Paul Cortens)
Date: Tue, 19 Feb 2008 19:07:41 -0800
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import (to
	mysql on win xp)
In-Reply-To: <57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
	<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
Message-ID: 

Hi,

I did some hacking around with the tests for the mysql-2.7 gem, and I can
get it to do a bulk import without an error. I didn't go through all of the
options (like delimited by, etc).

However, when I tried running the adapter_extensions tests, I got errors. I
even tried overriding the execute method to remove logging, but that made no
difference.

I am running out of ideas. Maybe active record uses some weird configuration
with MySQL???

Paul

On Feb 15, 2008 4:11 PM, Marty Haught  wrote:

> ---------- Forwarded message ----------
> From: Chris Williams 
> Date: Fri, Feb 15, 2008 at 5:09 PM
> Subject: Re: [Activewarehouse-discuss] problems using bulk_import (to
> mysql on win xp)
> To: Marty Haught 
>
>
> I just did a gem list on my windows box and this is what it showed for
> mysql
>
>  mysql (2.7.1)
>    A win32-native build of the MySQL API module for Ruby.
>
>  I would uninstall the mysql gem and reinstall to make sure you are
>  using the win32 version.
>
>  Chris
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>



-- 


Ezk. 36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080219/934b4794/attachment.html 

From phylae at gmail.com  Wed Feb 20 02:51:47 2008
From: phylae at gmail.com (Paul Cortens)
Date: Tue, 19 Feb 2008 23:51:47 -0800
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import (to
	mysql on win xp)
In-Reply-To: 
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
	<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
	
Message-ID: 

I think I have made some progress. I added one line to mysql_adapter.rb in
the adapter extensions plugin:


Index: ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
===================================================================

--- ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
(revision 906)
+++ ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
(working copy)
@@ -43,6 +43,9 @@
         end
         q << " IGNORE #{options[:ignore]} LINES" if options[:ignore]
         q << " (#{options[:columns].join(',')})" if options[:columns]
+
+        @connection = Mysql.new(*@connection_options)
+
         execute(q)
       end


Things seem to be working. It appears that if close() gets run on a Mysql
object, you can no longer use LOAD DATA LOCAL INFILE, even if you call
real_reconnect(). You have to actually create a new object. However,
according to here: http://dev.rubyonrails.org/ticket/9688 the behavior is
different (possibly the opposite) when using the pure-Ruby Mysql gem.

Does anyone know what kinds of problems I could run into by adding the above
line? Will I be messing up all my other active record objects?

Cheers,
Paul




On Feb 19, 2008 7:07 PM, Paul Cortens  wrote:

> Hi,
>
> I did some hacking around with the tests for the mysql-2.7 gem, and I can
> get it to do a bulk import without an error. I didn't go through all of the
> options (like delimited by, etc).
>
> However, when I tried running the adapter_extensions tests, I got errors.
> I even tried overriding the execute method to remove logging, but that made
> no difference.
>
> I am running out of ideas. Maybe active record uses some weird
> configuration with MySQL???
>
> Paul
>
>
> On Feb 15, 2008 4:11 PM, Marty Haught  wrote:
>
> > ---------- Forwarded message ----------
> > From: Chris Williams 
> > Date: Fri, Feb 15, 2008 at 5:09 PM
> > Subject: Re: [Activewarehouse-discuss] problems using bulk_import (to
> > mysql on win xp)
> > To: Marty Haught 
> >
> >
> > I just did a gem list on my windows box and this is what it showed for
> > mysql
> >
> >  mysql (2.7.1)
> >    A win32-native build of the MySQL API module for Ruby.
> >
> >  I would uninstall the mysql gem and reinstall to make sure you are
> >  using the win32 version.
> >
> >  Chris
> > _______________________________________________
> > Activewarehouse-discuss mailing list
> > Activewarehouse-discuss at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> >
>
>
>
> --
>
>
> Ezk. 36




-- 


Ezk. 36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080219/7b3108d1/attachment-0001.html 

From phylae at gmail.com  Wed Feb 20 02:55:52 2008
From: phylae at gmail.com (Paul Cortens)
Date: Tue, 19 Feb 2008 23:55:52 -0800
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import (to
	mysql on win xp)
In-Reply-To: 
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
	<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
	
	
Message-ID: 

That should be real_connect() not real_reconnect().

If anyone is wondering where these method names are coming from, take a look
here:
http://tmtm.org/en/mysql/ruby/

Paul


On Feb 19, 2008 11:51 PM, Paul Cortens  wrote:

> I think I have made some progress. I added one line to mysql_adapter.rb in
> the adapter extensions plugin:
>
>
> Index: ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
> ===================================================================
> --- ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
> (revision 906)
> +++ ../../lib/adapter_extensions/connection_adapters/mysql_adapter.rb
> (working copy)
> @@ -43,6 +43,9 @@
>          end
>          q << " IGNORE #{options[:ignore]} LINES" if options[:ignore]
>          q << " (#{options[:columns].join(',')})" if options[:columns]
> +
> +        @connection = Mysql.new(*@connection_options)
> +
>          execute(q)
>        end
>
>
> Things seem to be working. It appears that if close() gets run on a Mysql
> object, you can no longer use LOAD DATA LOCAL INFILE, even if you call
> real_reconnect(). You have to actually create a new object. However,
> according to here: http://dev.rubyonrails.org/ticket/9688 the behavior is
> different (possibly the opposite) when using the pure-Ruby Mysql gem.
>
> Does anyone know what kinds of problems I could run into by adding the
> above line? Will I be messing up all my other active record objects?
>
> Cheers,
> Paul
>
>
>
>
>
> On Feb 19, 2008 7:07 PM, Paul Cortens  wrote:
>
> > Hi,
> >
> > I did some hacking around with the tests for the mysql-2.7 gem, and I
> > can get it to do a bulk import without an error. I didn't go through all of
> > the options (like delimited by, etc).
> >
> > However, when I tried running the adapter_extensions tests, I got
> > errors. I even tried overriding the execute method to remove logging, but
> > that made no difference.
> >
> > I am running out of ideas. Maybe active record uses some weird
> > configuration with MySQL???
> >
> > Paul
> >
> >
> > On Feb 15, 2008 4:11 PM, Marty Haught  wrote:
> >
> > > ---------- Forwarded message ----------
> > > From: Chris Williams 
> > > Date: Fri, Feb 15, 2008 at 5:09 PM
> > > Subject: Re: [Activewarehouse-discuss] problems using bulk_import (to
> > > mysql on win xp)
> > > To: Marty Haught 
> > >
> > >
> > > I just did a gem list on my windows box and this is what it showed for
> > > mysql
> > >
> > >  mysql (2.7.1)
> > >    A win32-native build of the MySQL API module for Ruby.
> > >
> > >  I would uninstall the mysql gem and reinstall to make sure you are
> > >  using the win32 version.
> > >
> > >  Chris
> > > _______________________________________________
> > > Activewarehouse-discuss mailing list
> > > Activewarehouse-discuss at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
> > >
> >
> >
> >
> > --
> >
> >
> > Ezk. 36
>
>
>
>
> --
>
>
> Ezk. 36




-- 


Ezk. 36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080219/955892a9/attachment.html 

From jesperrr at gmail.com  Wed Feb 20 06:27:18 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Wed, 20 Feb 2008 12:27:18 +0100
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import (to
	mysql on win xp)
In-Reply-To: 
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
	<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
	
	
Message-ID: 

Hi Paul
Thanks for putting effort into this strange weird behaviour.

The change you suggest, however, makes no difference for me. I still get the
same error. I'm including stack trace below signature.

/Jesper


On Feb 20, 2008 8:51 AM, Paul Cortens wrote:

> I think I have made some progress. I added one line to mysql_adapter.rb in
> the adapter extensions plugin:
> +        @connection = Mysql.new(*@connection_options)
>



-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
___________________________________________________________

Executing screens
Screens passed

Executing post processes
c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in
`log': Mysql::Error: #42000The used command is not allowed with this MySQL
version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My
Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE
employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES
(id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id)
(ActiveRecord::StatementInvalid)
    from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in
`execute'
    from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:49:in
`do_bulk_load'
    from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in
`bulk_load'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in
`process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in
`transaction'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in
`process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in
`post_process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in
`each'
     ... 13 levels...
    from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in
`require'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28
    from c:/ruby/bin/etl:19:in `load'
    from c:/ruby/bin/etl:19
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080220/d18a54d7/attachment.html 

From phylae at gmail.com  Wed Feb 20 13:46:19 2008
From: phylae at gmail.com (Paul Cortens)
Date: Wed, 20 Feb 2008 10:46:19 -0800
Subject: [Activewarehouse-discuss] Fwd: problems using bulk_import
	(to	mysql on win xp)
In-Reply-To: 
References: 	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>		<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>		
	
Message-ID: <004701c873f0$e1f6eae0$a5e4c0a0$@com>

Jesper,

 

Have you tried running the tests that come with the adapter_extensions plugin? Those were the same tests that I was running, so maybe that might help us identify the difference that is causing the problem.

 

Let me know if you need help running the tests.

 

Paul

 

From: activewarehouse-discuss-bounces at rubyforge.org [mailto:activewarehouse-discuss-bounces at rubyforge.org] On Behalf Of Jesper R?nn-Jensen
Sent: Wednesday, February 20, 2008 3:27 AM
To: activewarehouse-discuss at rubyforge.org
Subject: Re: [Activewarehouse-discuss] Fwd: problems using bulk_import (to mysql on win xp)

 

Hi Paul
Thanks for putting effort into this strange weird behaviour.

The change you suggest, however, makes no difference for me. I still get the same error. I'm including stack trace below signature.

/Jesper



On Feb 20, 2008 8:51 AM, Paul Cortens wrote:

I think I have made some progress. I added one line to mysql_adapter.rb in the adapter extensions plugin:
+        @connection = Mysql.new(*@connection_options)




-- 
___________________________________________________________ 

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/ 
___________________________________________________________

Executing screens
Screens passed

Executing post processes
c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in `log': Mysql::Error: #42000The used command is not allowed with this MySQL version: LOAD DATA LOCAL INFILE 'C:/Documents and Settings/Administrator/My Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id) (ActiveRecord::StatementInvalid)
    from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in `execute'
    from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:49:in `do_bulk_load'
    from c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in `bulk_load'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in `process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in `process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in `post_process'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in `each'
     ... 13 levels...
    from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in `require'
    from c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28
    from c:/ruby/bin/etl:19:in `load'
    from c:/ruby/bin/etl:19

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080220/bc9014be/attachment-0001.html 

From mwlang at mdlogix.com  Thu Feb 21 12:37:11 2008
From: mwlang at mdlogix.com (Michael Lang)
Date: Thu, 21 Feb 2008 12:37:11 -0500
Subject: [Activewarehouse-discuss] Helper Tables Support?
In-Reply-To: <004701c873f0$e1f6eae0$a5e4c0a0$@com>
References: 	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>		<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>			
	<004701c873f0$e1f6eae0$a5e4c0a0$@com>
Message-ID: <47BDB6C7.6030404@mdlogix.com>

Does the Active Warehouse project support the notion of "Helper 
Tables"?  In other words, many-to-many relationships between important 
dimensions.

http://www.dbmsmag.com/9808d05.html
http://www.intelligententerprise.com/010810/412warehouse1_1.jhtml

For my particular example. 
 - I have a table of patients. 
 - I have a table of visits for those patients, each visit having a 
primary diagnosis. 
 - I have another table of patient diagnoses keyed off the visits table 
in which those patients have between 1 (the primary diagnosis in the 
table of visit is duplicated here) to as many as 30 diagnoses ascribed 
to them.  Each diagnosis is an additional row in this table.

So, that is:  Patients >-- Visits >-- Diagnoses

I want to ask the question: "how many patients came in for a visit who 
were diagnosed with A as primary symptom and also had B diagnosed at the 
time of visitation?" 

Helper tables seems to be the way to go, but I'm not sure how to do the 
weighted_value column nor how to leverage the Active Warehouse 
components for this.

Any suggestions on how to proceed to model such that I'm not counting 
the same patient twice regardless of number of visits or number of 
diagnoses recorded for a visit? 

Regards,

Michael


From mwlang at mdlogix.com  Tue Feb 26 09:54:10 2008
From: mwlang at mdlogix.com (Michael Lang)
Date: Tue, 26 Feb 2008 09:54:10 -0500
Subject: [Activewarehouse-discuss] Where to report bugs?
In-Reply-To: 
References: 	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>		<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>		
	
Message-ID: <47C42812.1010809@mdlogix.com>

Where's the best place to report bugs?

When using the database_destination and truncate == true, only the last 
32 records are actually manifested.

 From database_destination.rb, it looks like the table is repeated 
truncated during processing (every time the buffer is flushed)...

      # Flush the currently buffered data
      def flush
        conn = ETL::Engine.connection(target)
        conn.transaction do
          conn.truncate(table_name) if truncate

Setting truncate to false and emptying table manually allowed me to 
fully load the data.

One other thing:  I, too am experiencing issues with the bulk load 
approach.  The odd thing is that it works sometimes.  For example, 
worked fine for four or five smaller dimension tables, but will not work 
for larger dimension tables.  When it does fail, I see the LOAD DATA sql 
in the etl.log file and pasting this SQL into the MySQL console is 
failing, as well, so not likely Active Record issue.

Regards,

Michael        


Jesper R?nn-Jensen wrote:
> Hi Paul
> Thanks for putting effort into this strange weird behaviour.
>
> The change you suggest, however, makes no difference for me. I still 
> get the same error. I'm including stack trace below signature.
>
> /Jesper
>
>
> On Feb 20, 2008 8:51 AM, Paul Cortens wrote:
>
>     I think I have made some progress. I added one line to
>     mysql_adapter.rb in the adapter extensions plugin:
>     +        @connection = Mysql.new(*@connection_options)
>
>
>
>
> -- 
> ___________________________________________________________
>
> Jesper R?nn-Jensen / Capgemini Denmark
> Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
> www.dk.capgemini.com  / Blog 
> http://justaddwater.dk/
> ___________________________________________________________
>
> Executing screens
> Screens passed
>
> Executing post processes
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in 
> `log': Mysql::Error: #42000The used command is not allowed with this 
> MySQL version: LOAD DATA LOCAL INFILE 'C:/Documents and 
> Settings/Administrator/My 
> Documents/rails/ourpeople/import/output/employees.out.txt' INTO TABLE 
> employees FIELDS TERMINATED BY ',' ENCLOSED BY '"' IGNORE 1 LINES 
> (id,cgid,longid,firstname,lastname,displayname,middleinitial,initials,phone,mobile,fax,subcellphone,email,englishtitle,localtitle,secretary_id,mentor_id,substitute_id,manager_id,comment,licenseplate,employment_cgemployee,employment_cgexternal,employment_cgtemporary,employeework_employeeid,employeework_workpercentage,employeework_actualworkpercentage,company_id,grade_id,location_id,organisation_id,profession_id,resourcetype_id,skillsunit_id,staffingunit_id) 
> (ActiveRecord::StatementInvalid)
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/mysql_adapter.rb:281:in 
> `execute'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/mysql_adapter.rb:49:in 
> `do_bulk_load'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/adapter_extensions-0.4.0/lib/adapter_extensions/connection_adapters/abstract_adapter.rb:18:in 
> `bulk_load'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:73:in 
> `process'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in 
> `transaction'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/processor/bulk_import_processor.rb:62:in 
> `process'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:485:in 
> `post_process'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/../lib/etl/engine.rb:484:in 
> `each'
>      ... 13 levels...
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/dependencies.rb:496:in 
> `require'
>     from 
> c:/ruby/lib/ruby/gems/1.8/gems/activewarehouse-etl-0.9.0/bin/etl:28
>     from c:/ruby/bin/etl:19:in `load'
>     from c:/ruby/bin/etl:19
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Activewarehouse-discuss mailing list
> Activewarehouse-discuss at rubyforge.org
> http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>   

From thibaut.barrere at gmail.com  Tue Feb 26 10:06:48 2008
From: thibaut.barrere at gmail.com (=?ISO-8859-1?Q?Thibaut_Barr=E8re?=)
Date: Tue, 26 Feb 2008 16:06:48 +0100
Subject: [Activewarehouse-discuss] Where to report bugs?
In-Reply-To: <47C42812.1010809@mdlogix.com>
References: 
	<57f29e620802151558w19c76639o6dfa985345ee51a7@mail.gmail.com>
	
	<57f29e620802151611h737e1d23s1d0d4da17a4647a3@mail.gmail.com>
	
	
	
	<47C42812.1010809@mdlogix.com>
Message-ID: <4a68b8cf0802260706j39fa3f65k21bcb8cc9162995f@mail.gmail.com>

Hi Michael

Where's the best place to report bugs?


I'm not sure - but here seems to be fine (someone else ?).

One other thing:  I, too am experiencing issues with the bulk load
> approach.  The odd thing is that it works sometimes.  For example,
> worked fine for four or five smaller dimension tables, but will not work
> for larger dimension tables.  When it does fail, I see the LOAD DATA sql
> in the etl.log file and pasting this SQL into the MySQL console is
> failing, as well, so not likely Active Record issue.


On this specific topic, I remember I had issues caused by MySQL itself
(timeout because of packets too large). We had mail exchanges with Anthony
as well.

The solution I came up with was to split the uploaded file in chunks.

A patch to do this is available here:

http://rubyforge.org/pipermail/activewarehouse-discuss/2007-September/000179.html

hope this helps,

Thibaut Barr?re
--
LoGeek
[blog] http://blog.logeek.fr - learning content for developers
[blog] http://evolvingworker.com - tools for a better day
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080226/30044d71/attachment.html 

From jesperrr at gmail.com  Wed Feb 27 14:49:36 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Wed, 27 Feb 2008 20:49:36 +0100
Subject: [Activewarehouse-discuss] Fwd: Your message to
	Activewarehouse-discuss awaits moderator approval
In-Reply-To: 
References: 
Message-ID: 

Hi there.
Who will moderate my message and approve it? I sent a 55kb patch file with
documentation updates, but nobody has approved it yet.

Does anybody know who have administrative rights on this list??

Cheers,
Jesper



-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
___________________________________________________________

---------- Forwarded message ----------
From: 
Date: Sat, Feb 16, 2008 at 12:24 AM
Subject: Your message to Activewarehouse-discuss awaits moderator approval
To: jesperrr at gmail.com


Your mail to 'Activewarehouse-discuss' with the subject

   Final documentation update for ActiveWarehouse-etl

Is being held until the list moderator can review it for approval.

The reason it is being held:

   Message body is too big: 55436 bytes with a limit of 40 KB

Either the message will get posted to the list, or you will receive
notification of the moderator's decision.  If you would like to cancel
this posting, please visit the following URL: [removed by Jesper]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080227/251cb6e8/attachment.html 

From jesperrr at gmail.com  Fri Feb 15 18:24:36 2008
From: jesperrr at gmail.com (=?UTF-8?Q?Jesper_R=C3=B8nn-Jensen?=)
Date: Sat, 16 Feb 2008 00:24:36 +0100
Subject: [Activewarehouse-discuss] Final documentation update for
	ActiveWarehouse-etl
Message-ID: 

Hi there
(so silent in this forum??? Hope somebody is listening...


Here are my final additions to the documentation (as you know i have
basically just copied from the user guide...)
I am still not satisfied with the level of documentation... I still think
there could be more examples put in, some files/features are still not
documented, etc.

This is what's fixed in patch_04 (which contains all changes including the
earlier patches)
trunk/lib/etl/processor/bulk_import_processor.rb
=> added :ignore flag that was not there

trunk/lib/etl/parser/delimited_parser.rb
=> fixed toggled line/file in error msg


All files: Documentation updates




/Jesper

PS i also encountered some errors that i should probably ask you for help
about... but that would be in other mails...

-- 
___________________________________________________________

Jesper R?nn-Jensen / Capgemini Denmark
Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
www.dk.capgemini.com / Blog http://justaddwater.dk/
jesperrr at gmail.com (Private e-mail and Google Talk IM)
___________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/858bc21d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: documentation_patch_04.patch
Type: application/octet-stream
Size: 38569 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/activewarehouse-discuss/attachments/20080216/858bc21d/attachment-0001.obj 

From anthonyeden at gmail.com  Thu Feb 28 06:26:15 2008
From: anthonyeden at gmail.com (Anthony Eden)
Date: Thu, 28 Feb 2008 06:26:15 -0500
Subject: [Activewarehouse-discuss] Fwd: Your message to
	Activewarehouse-discuss awaits moderator approval
In-Reply-To: 
References: 
	
Message-ID: 

I approved it. In the future you may want to consider uploading
patches to the Patches section of the RubyForge page:

http://rubyforge.org/tracker/?atid=9386&group_id=2435&func=browse

V/r
Anthony

On Wed, Feb 27, 2008 at 2:49 PM, Jesper R?nn-Jensen  wrote:
> Hi there.
> Who will moderate my message and approve it? I sent a 55kb patch file with
> documentation updates, but nobody has approved it yet.
>
> Does anybody know who have administrative rights on this list??
>
>  Cheers,
> Jesper
>
>
>
> --
> ___________________________________________________________
>
> Jesper R?nn-Jensen / Capgemini Denmark
> Tel. +45 3977 8220 / Mob. +45 2373 6220 / Fax +45 7011 2201
>  www.dk.capgemini.com / Blog http://justaddwater.dk/
> ___________________________________________________________
>
> ---------- Forwarded message ----------
> From: 
>  Date: Sat, Feb 16, 2008 at 12:24 AM
> Subject: Your message to Activewarehouse-discuss awaits moderator approval
> To: jesperrr at gmail.com
>
>
> Your mail to 'Activewarehouse-discuss' with the subject
>
>     Final documentation update for ActiveWarehouse-etl
>
>  Is being held until the list moderator can review it for approval.
>
>  The reason it is being held:
>
>     Message body is too big: 55436 bytes with a limit of 40 KB
>
>  Either the message will get posted to the list, or you will receive
>  notification of the moderator's decision.  If you would like to cancel
>  this posting, please visit the following URL: [removed by Jesper]
>
>
>
>
>
>
> _______________________________________________
>  Activewarehouse-discuss mailing list
>  Activewarehouse-discuss at rubyforge.org
>  http://rubyforge.org/mailman/listinfo/activewarehouse-discuss
>
>

