From erik at hollensbe.org  Wed Mar 12 18:29:33 2008
From: erik at hollensbe.org (Erik Hollensbe)
Date: Wed, 12 Mar 2008 15:29:33 -0700
Subject: [Ruby-dbi-devel-new] Oracle vs. OCI8 driver
Message-ID: <200803121529.33872.erik@hollensbe.org>

Hello,

What I've been doing the last few days is attempting to get Oracle working so 
I can test the oracle drivers. Chris Maujean (the oracle guru of the two of 
us) has been strapped for time lately, so he handed it off to me and 
said, "Port it to ruby-oci8". 

FTR, this is the only thing that's holding the 0.2.0 release up, with the 
current constraints on what we're able to maintain.

What I found out where a couple of things unbeknownst to me:

1) OCI8 comes with a DBD that (presumably) is actively maintained. The last 
release was on 11/25/2007.

2) The "Oracle" DBD is supports the Oracle 7 interface on RAA. Since I was 
using 8i when it was "old" about 6-7 years ago, I would be heartily surprised 
if Oracle 7 is still in frequent use. Those who are using the Oracle DBD on 
8i and above would probably be better off using the OCI8 DBD if they aren't.

3) I forgot that oracle is lacking simple things like a boolean type, 
therefore rendering a great deal of the "general" test suite useless without 
a lot of case statements on the dbtype.

So in lieu of all of this, I'm proposing:

1) That we deprecate the Oracle driver, politely asking that those that need 
it maintain it, which is the direction we're heading anyways.

2) We modify the test suite to test against OCI8, outright skipping the types 
testing (but making sure the rest works) until 0.4.0 starts development, 
which is going to have a strong focus on type handling.

Complaints/flames/ideas?

-- 
Erik Hollensbe <erik at hollensbe.org>

From erik at hollensbe.org  Sun Mar 16 16:11:23 2008
From: erik at hollensbe.org (Erik Hollensbe)
Date: Sun, 16 Mar 2008 13:11:23 -0700
Subject: [Ruby-dbi-devel-new] 0.2.0 released
Message-ID: <200803161311.23297.erik@hollensbe.org>

http://rubyforge.org/forum/forum.php?forum_id=22766 for release details, and 
here's the long-form changelog:

http://code.hollensbe.org/?p=ruby-dbi.git;a=summary

Run the tests, break things, fill the tracker up.

Thanks to everyone on these lists for your help!

-- 
Erik Hollensbe <erik at hollensbe.org>

From mjp at pilcrow.madison.wi.us  Thu Mar 20 00:41:25 2008
From: mjp at pilcrow.madison.wi.us (Mike Pomraning)
Date: Wed, 19 Mar 2008 23:41:25 -0500
Subject: [Ruby-dbi-devel-new] BaseStatement#cancel default implementation
Message-ID: <7db9a2550803192141k39a52eafo724b0e0bf63446e9@mail.gmail.com>

Proposal:  DBI::BaseStatement#cancel should simply read and discard
any remaining rows, something like:

    # class BaseStatement
    # Discard all pending rows, if any, making the statement ready for
re-execution.
    # Subclasses are free to offer more efficient implementations, of course,
    def cancel
       row = "not nil"
       until row.nil?
         row = fetch
       end
       nil
    end

    ...
    # class StatementHandle
    def execute(*args)
       cancel if fetchable?  # <--- fetchability check is new
       ...

Having taken a crack at true (server-side) prepared statements in the
Pg driver (Tracker "Patches" #18925), I started toying with a
DatabaseHandle#prepare_cached(), ala "prepare_cached" from perl's
DBI.pm.

Now, DBI.pm raises the question of what to do when a #fetchable? sth
(DBI.pm analog: $sth->{Active}) is pulled from the cache.  DBI.pm by
default complains about this situation, but resets the sth with
$sth->finish, and moves on.  We can't simply imitate that behavior,
since our #finish is more like the DBI.pm's $sth->DESTROY.  Our
#cancel comes closest, however, to a bona fide prepared statement
reset.

Reactions?  There's still the separate question of what
safety-checking #prepare_cached might look like, e.g.:

   dbh.prepare_cached(sql, :if_active => ACTIVE_HANDLE_WARN) #
default, complain but cancel() the cached sth
   # ACTIVE_HANDLE_CANCEL ... silently cancel() the cached sth
   # ACTIVE_HANDLE_IGNORE ... don't bother checking cached sth for
fetchable?ness
   # ACTIVE_HANDLE_REPLACE ... cache a newly prepared sth instead,
without mutating the old one in any way

-Mike

From erik at hollensbe.org  Thu Mar 20 09:58:02 2008
From: erik at hollensbe.org (Erik Hollensbe)
Date: Thu, 20 Mar 2008 06:58:02 -0700
Subject: [Ruby-dbi-devel-new] BaseStatement#cancel default implementation
In-Reply-To: <7db9a2550803192141k39a52eafo724b0e0bf63446e9@mail.gmail.com>
References: <7db9a2550803192141k39a52eafo724b0e0bf63446e9@mail.gmail.com>
Message-ID: <E9B27EAF-9D89-4330-BA20-B81184E3E6E0@hollensbe.org>


On Mar 19, 2008, at 9:41 PM, Mike Pomraning wrote:
> Proposal:  DBI::BaseStatement#cancel should simply read and discard
> any remaining rows, something like:
>
>    # class BaseStatement
>    # Discard all pending rows, if any, making the statement ready for
> re-execution.

I think this could be easily re-written as:

MUST Make the statement ready for re-execution. (using RFC parlance)

And lose nothing, leaving the concept of exhausting a cursor up to the  
driver if need be.

>    ...
>    # class StatementHandle
>    def execute(*args)
>       cancel if fetchable?  # <--- fetchability check is new
>       ...

I'm honestly not sure how to parse this. We especially don't want to  
cancel the statement if it's fetchable, did you mean to use 'unless'  
instead of 'if'?

>
> Having taken a crack at true (server-side) prepared statements in the
> Pg driver (Tracker "Patches" #18925), I started toying with a
> DatabaseHandle#prepare_cached(), ala "prepare_cached" from perl's
> DBI.pm.

I don't want to distract you from improving DBI at all, but I have no  
intention of adding features like this in this release. Right now my  
concern is to get the existing features into an easily maintainable  
and implementable state, which is why type handling is at the top of  
my list. Anyone who's read the guts of a DBD should be able to  
understand why I'm tackling this first.

I'm not saying that a fully specced, fully implemented across all DBDs  
and DBI implementation won't be considered, it's just that I'm not  
ready to discuss it yet, but I will offer my viewpoints for now.

>
> Now, DBI.pm raises the question of what to do when a #fetchable? sth
> (DBI.pm analog: $sth->{Active}) is pulled from the cache.  DBI.pm by
> default complains about this situation, but resets the sth with
> $sth->finish, and moves on.  We can't simply imitate that behavior,
> since our #finish is more like the DBI.pm's $sth->DESTROY.  Our
> #cancel comes closest, however, to a bona fide prepared statement
> reset.
>
> Reactions?  There's still the separate question of what
> safety-checking #prepare_cached might look like, e.g.:
>
>   dbh.prepare_cached(sql, :if_active => ACTIVE_HANDLE_WARN) #
> default, complain but cancel() the cached sth
>   # ACTIVE_HANDLE_CANCEL ... silently cancel() the cached sth
>   # ACTIVE_HANDLE_IGNORE ... don't bother checking cached sth for
> fetchable?ness
>   # ACTIVE_HANDLE_REPLACE ... cache a newly prepared sth instead,
> without mutating the old one in any way

Initially peeking at this it seems like we've got a situation where  
constants are used to define actions on a specific set of states.

Instead of just giving them four options, why not give them a series  
of state reporting methods (e.g., cancelled? which would report if if  
the statement was cancelled prior to a new execution) and let them  
work with it themselves, and allow the option of being able to finish  
cached statements after retrieving them.

Thanks,

-Erik

From erik at hollensbe.org  Thu Mar 20 10:36:04 2008
From: erik at hollensbe.org (Erik Hollensbe)
Date: Thu, 20 Mar 2008 07:36:04 -0700
Subject: [Ruby-dbi-devel-new] Ruby/DBI spec and 0.4.0 development notes
Message-ID: <2E7A1DF6-3AF8-4246-AEE1-4E7AC3FCD50F@hollensbe.org>

I know I've left you guys in the dark for about a week regarding where  
0.4.0 is going, and I apologize for that. Work has been particularly  
demanding on a number of levels. Yep.... that's the best excuse I can  
offer. :)

Anyways, I wanted to make everyone aware of what's going on.

First, I have rubygems rolled and rakefile'd up, but the DBI core  
itself needs to be modified a tad to use the separate DBDs. Notably,  
DBI doesn't use rubygems anywhere (although I did add it in a few of  
the DBDs). Daniel Berger started some work on this process, but dbi.rb  
and the specific DBDs do not actually require rubygems themselves,  
meaning that if you have to perform non-gem installations of all the  
low-level drivers to work with the DBDs.

The other problem is, the DBD directory does not match the gem  
installation path of dbd/SQLite.rb, f.e.

I'm sure I could go through a lot of pain (and believe me, there is  
quite a bit of pain in the DBD include logic) to get the existing  
system working that handles case-insensitivity and a number of other  
odd cases, but frankly, I think this is a lot of overcomplicated code  
that serves no one especially well (f.e., Perl's DBI is case-sensitive  
when it comes to drivers) and de-complicating the logic here seems to  
be the best win for Rubygems support.

The plan for these gems is to go up as DBI 0.3.0 (development)  
supported gems and to allow people to track DBI that way, but I'm  
trying to find a good way to say "Achtung! Stop! This is development!"  
in a `gem install` ecosystem so people who use `gem search` or are  
simply not paying a lot of attention will install 0.2.0 (or perhaps,  
0.1.1) for a stable release.

Once I figure these two things out they will be available. For now,  
you can track the "development" branch of the git repository and build  
your own gems, but I'll be surprised if using them in code requires  
the right files ATM.

Second, I'm working on a little system to annotate the spec so I can  
review it and bring points of contention (and allow others to do the  
same) to the list. This is taking a bit longer than I wanted it to, so  
I may just go with a passworded wiki or something in the same vein.  
Please let me know what you guys think is best and easiest for you to  
work with.

The spec is critical. We're moving to distributing the DBDs separately  
and the OCI8 DBD is already that way. Before we can reliably guarantee  
our users DBDs that are uniform, we need a document and tests that  
ensure that. Ideally DBI's core documentation should be the base of  
the spec, not a copy of the code-specific portions. Look at Perl DBI's  
perldoc for a great example.

I would first like to move to an RFC-style requirements system (MUST,  
MUST NOT, SHOULD, SHOULD NOT, etc) for the spec. I think the spec is  
entirely too vague right now and it's allowing a lot of things through.

After the spec is done but (ideally) before any coding is done, I'm  
going to write 'general' DBD tests against the spec for our supported  
DBDs and DBI. This will hopefully enforce the spec better than words  
in a RDoc document.

As far as features are concerned, I would like to concentrate on type  
handling and ideally lifting a lot more code out of the DBDs and make  
canonical representations in DBI for type coercion a possibility.  
Right now a *lot* of duplicated mappings, coercion systems, and other  
type handling madness is in the DBDs and it makes them extremely hard  
to maintain in a uniform fashion. Abstracting this is one of the  
places that Perl's DBI really doesn't have an equally transferrable  
solution for (anyone familiar with perl knows the perils of its "type  
system") so we're gonna have to be a little more creative than usual.

After that's solved, I would like to direct our focus towards properly  
managing the expectations of database metadata calls (such as tables()  
and columns()) WRT to things like system tables and Pg schemas, and  
describing (in RFC terms) what information should be available here.

After these four things are handled, this would be a good stopping  
point for an 0.4.0 release, along with any 0.2.0 bugfixes that come  
along in that time. I'd talk about 0.6.0 but I don't have anything to  
say just yet. :)

Please feel welcome to argue this with me, especially if I have  
glossed over something you feel is a critical feature that needs to be  
addressed immediately.

Thanks,

-Erik

From mjp at pilcrow.madison.wi.us  Thu Mar 20 20:58:23 2008
From: mjp at pilcrow.madison.wi.us (Mike Pomraning)
Date: Thu, 20 Mar 2008 19:58:23 -0500
Subject: [Ruby-dbi-devel-new] BaseStatement#cancel default implementation
In-Reply-To: <E9B27EAF-9D89-4330-BA20-B81184E3E6E0@hollensbe.org>
References: <7db9a2550803192141k39a52eafo724b0e0bf63446e9@mail.gmail.com>
	<E9B27EAF-9D89-4330-BA20-B81184E3E6E0@hollensbe.org>
Message-ID: <7db9a2550803201758m3e85ca0bm314c07f00554173f@mail.gmail.com>

On Thu, Mar 20, 2008 at 8:58 AM, Erik Hollensbe <erik at hollensbe.org> wrote:
>
>  >    # class StatementHandle
>  >    def execute(*args)
>  >       cancel if fetchable?  # <--- fetchability check is new
>  >       ...
>
>  I'm honestly not sure how to parse this. We especially don't want to
>  cancel the statement if it's fetchable, did you mean to use 'unless'
>  instead of 'if'?

Nope --  we always want to dismiss any queued results from a previous
query.  As it happens, this is the current (and desirable) behavior
and the above was unnecessary:  execute() summarily calls cancel()
before doing anything, and cancel() itself checks for fetchable?ness.

>  > Having taken a crack at true (server-side) prepared statements in the
>  > Pg driver (Tracker "Patches" #18925), I started toying with a
>  > DatabaseHandle#prepare_cached(), ala "prepare_cached" from perl's
>  > DBI.pm.
>
>  I don't want to distract you from improving DBI at all, but I have no
>  intention of adding features like this in this release. Right now my

Do you mean features like externally visible API convenience
extensions (e.g., "prepare_cached"), or features like internal driver
improvements which offer what some would argue is the minimum
functionality required for a useful, safe and correct DB abstraction
(e.g., DBD::Pg::please_please_use_native_prepared_statements)?  As you
might guess, I've an opinion about the relative importance of the two.
 :)

>  >   dbh.prepare_cached(sql, :if_active => ACTIVE_HANDLE_WARN) #
>  > default, complain but cancel() the cached sth
>  >   # ACTIVE_HANDLE_CANCEL ... silently cancel() the cached sth
>  >   # ACTIVE_HANDLE_IGNORE ... don't bother checking cached sth for
>  > fetchable?ness
>  >   # ACTIVE_HANDLE_REPLACE ... cache a newly prepared sth instead,
>  > without mutating the old one in any way
> [....]
>  Instead of just giving them four options, why not give them a series
>  of state reporting methods (e.g., cancelled? which would report if if
>  the statement was cancelled prior to a new execution) and let them
>  work with it themselves, and allow the option of being able to finish
>  cached statements after retrieving them.

Ack.  fetchable? already provides a suitable check (though active?
would be a clearer synonym, IMHO).  The same check, FWIW, could be
used for someone concerned about re-execute()ing an active sth.

-Mike

From erik at hollensbe.org  Mon Mar 24 12:39:45 2008
From: erik at hollensbe.org (Erik Hollensbe)
Date: Mon, 24 Mar 2008 09:39:45 -0700
Subject: [Ruby-dbi-devel-new] some ideas on type conversion
Message-ID: <A797898D-0E95-4B38-96CC-288F6590CCA7@hollensbe.org>

Here's what I've come up with on type conversion, sorry for the ugly  
URL. If you'd prefer, you can get the latest git repo, switch to the  
"development" repo, and look in the "prototypes" directory. Please  
let me know if anything is unclear or requires flames. :)

http://code.hollensbe.org/?p=ruby-dbi.git;a=blob;f=prototypes/ 
types.rb;h=b9ec104118553ec054558c94b51231139a4d44a0;hb=f0c124d920ef6996e 
a63400686a563c6e89cf694

I'm on vacation this week, so I should have plenty of time to work on  
things. One of the other things I plan to do is expand the test suite  
to exercise the full spec. This should work out where our supported  
DBDs for 0.4.0 still need help, and probably where the spec should be  
changed to reflect reality, too.

Mike, how interested would you be in tackling the prepare_cached  
implementation in parallel with the changes above? You'd probably be  
hitting a moving target, and would have to work on more than the Pg  
implementation. :)

--
Erik Hollensbe
erik at hollensbe.org