[Rubytests-devel] Known problems ...
holmberg at iar.se
Wed Dec 1 13:56:47 EST 2004
Thanks for the well-written reply. I can agree with most of what you
said. Some further thoughts:
In C-Ruby I found a function "rb_notimplement". I assume that it is
intended for cases like the "ino" method we talked about. On Windows
we already have:
$ ruby -e 'File.lchmod("foo")'
-e:1:in `lchmod': The lchmod() function is unimplemented on \
this machine (NotImplementedError)
$ ruby -e 'Dir.chroot("foo")'
-e:1:in `chroot': The chroot() function is unimplemented on \
this machine (NotImplementedError)
Throwing the same exception for the File::Stat#ino method would
seems natural to me (I'll try to ask about that on "ruby-core").
Wouldn't this be the right thing for the methods you mentioned as
missing in JRuby too ?
> As far as testing, I think we should write tests according to how a
> function is *supposed* to work (expected to work, documented to work,
> common sense, etc), rather than testing that *how* it works is
> correct. Test-driven development, and etc...write your tests [...]
I _would like_ to fully agree, but I can see some minor "problems":
The Rubicon tests can be used in at least two different ways: A) to
verify an implementation against a specification (how it is supposed
to work), and B) to track changes to the language.
I guess that you are most interested in A). For me it has been B)
that has been in the foreground. I think it would be nice if Rubicon
could accommodate both "use cases".
So, when I run Rubicon I would like to have these inputs:
- a Ruby interpreter to test
- a language version to consider correct.
A problem here is that there is no formal specification of Ruby. The
only thing we have are different versions of C-Ruby that implicitly
defines the language at different times.
As output I would like to have:
- a list of cases where the interpreter under test doesn't
follow the specified language version
If this list gets long, there is a risk that one error _shadows_
another error. Currently most of Rubicon have _one_ test-method for
each method tested. If some test-method have many "asserts", and we
get an error, all we can say is that it has _at least_ one case
where it doesn't behave as expected. But I would often like to know
if there are _one_ or lets say _five_ errors in the method. It makes
a difference in the B) use case above.
I have been trying to figure out a way to handle both the A) and B)
use cases, but I don't have any solution yet.
One thought I've had, is to use the "known_problem" method for all
unresolved cases, but make it work in two different
1) do *nothing special* by default. The enclosed code should work
as if "known_problem" wasn't there at all, i.e. a failed "assert"
will lead to a real failure of the testcase.
2) make "known_problem" sensitive to an environment variable
(or perhaps command line option), and if it is set reclassify the
error to a "known problem" (as it is done today).
This would avoid missing errors by default (since all tests are
really executed). But it would still make it possible to "accept"
that some tests fail, and still go on an find further errors.
Another thought I've had is to split test-methods that have
problems. Maybe one test-method for each method tested is a
_too big_ unit. I've felt uncomfortable with slitting testcase,
since I've felt the splitting would be governed by accidental
circumstances. But now that I'm thinking about it again, maybe this
is the right way to go. It would probably make it easier to handle
language changes (some _test methods_ would be allowed/expected to
fail for a particular language version).
Am I thinking in the wrong direction ?
It's not easy sort all things out at the same time ....
My approach so far has been to take one step at a time,
and fix easy things first. I have been hoping that some good
solution to the trickier issues would pop-up later ...
More information about the Rubytests-devel