[Ironruby-core] State of IronRuby 10/19/07

John Lam (DLR) jflam at microsoft.com
Fri Oct 19 20:28:45 EDT 2007

NOTE: The SVN repository isn't up to date with these changes. It will be sometime over the weekend.

Thanks & happy reading,

286238: (dinov)

Currently ngen will try and produce an image that’s probably around 1gig in size (or larger) for Microsoft.Scripting.dll due to the reflected caller additions.  This cuts back the number of FastCreate’s we’ll do and gets this back to a reasonable level.  It also adds a cache for non-dynamicmethod’s so that we only create one per method.  This should help w/ both the throughput regressions as well as the fact that I broke ngen.

Also included is the fix for the out param reported on the mailing list yesterday.

286963: (tomat) Ruby bug fixes

Fixes the following Ruby bugs (no major change in design needed).

ID            Assigned To        Title
304051  Tomas Matousek             ruby: explicitly set $! in the ensure clause should not be preserved
303812  Tomas Matousek             ruby: $! is not preserved in the ensure clause when the exception is thrown in the else clause
303774  Tomas Matousek             ruby: redo should not re-evaluate the loop condition
303703  Tomas Matousek             ruby: "break" does not break out from the lambda call
298222  Tomas Matousek             ruby: explicitly setting $! in the rescue exception parameter does not preserve $!
297959  Tomas Matousek             ruby: $! in the ensure clause should not be nil when an exception is raised, but no matched rescue clause is found

293040: (dinov) Preparation to fix infinite recursion issue

These are the final set of small fixes which are necessary before we can fix the infinite recursion issue.

    1. Switch from using Type.Missing to our own MissingParameter type.  We can't pass Type.Missing through Reflection.Invoke calls.
    2. Fix Python's DoOperationBinderHelper to avoid casts which box/unbox values (this is just due to an identity of integer test in test_copy)
    3. Add a missing conversion in Ruby
    4. Check IsInstanceOfType in Cast to catch COM objects interface tests correctly
   5. Makes evaluation of addresses work the same as emitting - we can end up evaluating an expression before getting the address and that expression can throw.
   6. Fix an issue w/ return types on properties not going through the binder's converter
   7. Add support for strongly typed delegates in the eval case
   8. Fixed an issue w/ rethrowing exceptions in evaluate mode
   9. Added support for returning from a VoidExpression
  10. Fix an issue w/ arrays in evaluate mode where we try and cast to object[] when we have an array of byte[]

Also fixing this bug:

304475 dir(System.Reflection.Emit.AssemblyBuilder) throw TypeError

Where we just need to check for null in DynamicMixin.

293085: (tomat) Fixes RubyForge bug #14425.

Ruby allows jump statements (return, redo, retry, break and next) to appear in Boolean expressions as right operands.


def foo x
  x and return ‘true’
  x or return ‘false’

foo true               # “true”
foo false              # “false”

This shelveset changes the grammar to support this feature and adds AST node ConditionalJumpExpression that represents <condition> and/or <jump-statement> expression. Also adds a unit test of the node.

293153: (jomes) generics: implements support for constructing generic types

Pretty simple, implements support for creating generic .NET types like List<T>, Dictionary<K,V>, etc.

The syntax in Ruby is something like this:

  require "mscorlib"
  IntList = System::Collections::Generic::List.of(System::Int32)
  a = IntList.new
  a.add 1
  a.add 2
  a.add 3
  assert_equal(a.count, 3)

I also added support for the [] syntax that IronPython uses:

  System::Collections::Generic::Dictionary[String, String]

293329: (mmaly) Bye bye, WeakCondition

This change gets rid of the WeakCondition and updates all former calls to that to generate proper casts. Ruby has factory it wanted which generates the casts and up-casts to object. The cast to bool must be dynamic so that the helper gets called. Alternative would be calling the helper explicitly or doing something yet smarter.

There is one temporary change in the ReferenceArgBuilder.cs The builder generates code:

ElementType tmp;
arg is StrongBox<ElementType> ? tmp = RuntimeHelpers.GetBox<ElementType>((StrongBox<ElementType>)parameter), tmp : RuntimeHelpers.IncorrectBoxType(StrongBox<ElementType>);

Notice the comma in the first expression. If we cast the whole thing to object, the currently not quite complete implementation of calling methods with by ref arguments will do the wrong thing. Temporarily we cast the 2nd expression to ElementType as it will throw inside the helper anyway and the cast will never execute. The ideal solution is for the binder to generate a better tree all around, which will play better with the upcoming “emit as” implementation overhaul.

293941: (tomat) Ruby libraries

Prepares built-ins for a split: the types and functionality that the compiler directly depends on will be kept in Ruby.dll while all Ruby visible methods will be factored out to Ruby.Libraries.dll. Also adds Ruby.Libraries and ClassInitGenerator projects to the Rowan solution – they depend on Ruby.dll and M.S.dll, so they should be rebuilt each time the dependencies are rebuilt.

293960: (jomes) fixes inheriting from a .NET type

Very simple change, fixes inheriting from a .NET type in Ruby

1. We weren’t handling TypeTrackers
2. We weren’t exposing CLS members on the derived type

294389: (mmaly) Switch statement upgrade

I managed to rewrite the switch statement yesterday. While this change doesn’t yet add the missing interpreted mode, it removes all Jscript-isms. The switch statement is true switch. Takes an int and routes to given label. The labels are marked with _constant_ values. Switch will choose dynamically whether to emit as jump table or “if” statements.

If the labels are integer-like constants, it will generate the DLR switch. If not, it will also generate the DLR switch (cool, huh?) except preceded with an if statement which ‘encodes’ the switch labels into a constant integer and then proceeds to the direct switch with that constant. This eliminates the need for goto to do the fall through.

In the second (via encoding) case, you’ll see the if … elseif … elseif pattern first, in which the bodies are only (.bound #2) = … and then switch over the #2 temp.

DLR switch now checks for unique label values, and does it in 3 ways:
• If there are small number of cases (< 10), it uses dumb O(N^2) algorithm
• If there are cases values of which are in reasonable range (max – min < 1024), it uses BitArray to track the unique values
• Else, it falls back to dictionary

Only the generation into the pure switch is affected. When generating the if ..elif ..elif prefix, the generated label numbers are unique by definition so that path remains the same. However, for the switch table code path, the code merges bodies of the case clauses. Essentially each case clause whose label is not unique is merged with the previous clause’s body. The bodies must stay in place because of the fall-through, but no one can ever jump to the label of the non-unique statement so it is safe to merge them with the previous bodies.

294586: (jomes) ruby extension methods on interfaces

Implements extension methods on interfaces for Ruby. What this means it that .NET types will get Ruby methods injected onto them if they implement certain interfaces. For example, types that implement IEnumerable will behave just like a Ruby classes that mix in Enumerable.

I implemented this by treating interfaces as if they were mixed in modules.

These interfaces have extension methods now:
•       IEnumerable
•       IComparable
•       IList<object>
•       IDictionary<object, object>

IList<object> and IDictionary<object, object> were factored out of Array and Hash, respectively. Some investigation remains to see if these can be made to support all generic types or if they could use IList and IDictionary instead, but I didn’t attempt that yet.

Detailed change description:
•       Added a new attribute RubyExtensionInterfaceAttribute, updated ClassInitGenerator & Initializer to understand it
•       Added the MixinInterfaces property to RubyClassAttribute, which tells ClassInitGenerator to inject all valid interface methods it finds onto the type (useful if you’re extending a CLR type and you want everything).
o       StringOps uses this feature to get Enumerable & Comparable support.
•       RubyExecutionContext keeps track of interface mappings & mixes in the interface’s RubyModule onto new .NET types
•       IEnumerableOps implements each & includes Enumerable
•       IComparableOps implements <=> & includes Comparable
•       IDictionaryOps takes methods from HashOps (which were already written to operate on IDictionary<object, object>)
•       IListOps takes methods from ArrayOps
o       Helpers are needed for AddRange, GetRange, InsertRange, and RemoveRange, which IList<T> doesn’t support. Also, several methods are different between ArrayOps and IListOps

294789: (dinov) Infinite recursion fix

This changes rule creation so it only runs the test once and evaluates the target to avoid the test changing and getting infinite recursion.

This is done by pushing the update & invoke out of DynamicSite.UpdateBindingAndInvoke and all the way down into the internal portions of ActionBinder and the rule caching mechanism.  We now pass in the site, the site's target (by-ref), and the site's rule list (by-ref) and let the action binder update these for us.  This reduces a bunch of the duplicated code in DynamicSite.Generated.cs.

Part of the reason to push this in is that the site HAS to have its target delegate set before executing the target of the rule.  If not the site could recurse on its self, not have the target set, and therefore we limit the amount of stack that can be used - as well as increase the # of calls which don't use the optimized target.

294962: (dinov) Updates ExtensionTypeAttribute

This updates ExtensionTypeAttribute to no longer have any DynamicType dependencies.  It also removes some Python specific pieces to it (support for deriving from int, etc…) and removes “using M.S.T;” from everyone using it just for ExtensionTypeAttribute (this is most of the files changed).

294999: (dinov)

This change implements the helper methods on PropertyTracker and switches GetMemberBinderHelper over to using these helper methods.  This change ultimately removes dependencies on ReflectedProperty and ReflectedIndexer.  Mostly this is moving the logic in GetMemberBinderHelper over into PropertyTracker.

Add new ErrorInfo class which is used for reporting information about how an error should be reported to the user.  The error can either be reported as an exception which gets thrown or a value that gets returned.  Added new methods to ActionBinder which enable languages to produce errors.  Eventually all of the error production will move into this format.

Small tweaks:
                Move MakeCallExpression from BinderHelper onto Binder.  The only reason it was on BinderHelper was because it needed to access the Binder to provide conversions.
                Fixed the iteration in MethodBinder over the dictionary (which is bogus, we have a CodePlex bug on this reported by a user)
                Implemented Call on MethodTracker – this is used when PropertyTracker falls back for private binding.
                Exposed IsStatic directly off of PropertyTracker.

295337: (mmaly) Redesigning the DLR AST Walkers

(Walker6 passed SNAP already, Walker7 contains few cosmetic updates such as missing comments, removing few redundant argument checks, adding few extra ones etc. I hope it also passes)

To fix the “non-empty stack upon entry to try” bug, I am going to need a better walker than what is easy to implement in our system (one that I can use to re-generate the AST along the way). The current walker design doesn’t lend itself easily to that purpose (the Walk methods on the nodes do just that – walk, but I’d have to write a type-based switch to implement my walker, rather than an enum-based switch).

Second motivation is LINQ which doesn’t have walker on its nodes so to get closer to that model, our walkers must live completely outside of the AST.

This checkin achieves these two goals in a following ways:

1)      Each node is given an enum (this enum started with LINQ’s values, some are commented out as we don’t use them (yet – CheckedAdd for example), some are added (statements). The UnaryOperator and BinaryOperator enums got swallowed by the big enum which nicely plays that role.
2)      The walker is combination of generated and hand-written code and the Walk methods from the AST nodes are gone. The individual walkers haven’t changed except to call WalkNode(node) instead of node.Walk(this) to perform the node walk. The WalkNode will switch based on the node type (enum), direct cast and perform walk identical to that of the original node’s walk.

What I like about having the walker completely outside of the tree is that the AST nodes don’t dictate the order of walking or don’t specify one way as ‘primary’. This way any walker (and we’ll eventually need a reverse walker) plays by the same rules on the same playing field.

Next steps in this direction:
•       Make all leaf AST nodes sealed
•       Remove codegen from the AST nodes
•       Merge AST node that can be merged (say, after we remove the codegen out, there is no need to have break and continue as separate AST nodes because they’ll hold identical information, same for CodeContextExpression and EnvironmentExpression for example). So we’ll end up with fewer nodes as a result, which is a goodness.

295459: (dinov)

This change is just re-organizing DynamicHelpers and RuntimeHelpers in preparation for the removal of DynamicType.  Previously these two classes had no real strong distinction.  Why does something go into DynamicHelpers?  Why does it go into RuntimeHelpers?  Who knows.  There is now one distinction: Things in DynamicHelpers are going away, things in RuntimeHelpers are staying.  DynamicHelpers its self moves into M.S.Types indicating that it will be removed real soon now (I don’t actually move the file, it’ll be gone in a day or two).

What this is ultimately going to accomplish is that all of the IronPython type system will be able to come up in one chunk w/ hopefully no code changes.  From there it’ll be a few renames away from being integrated into the IronPython code base.  Because there are a whole lot more references to DynamicHelpers.GetDynamicType* then there are to the other functionality in DynamicHelpers it seems natural to keep DynamicHelpers as the home of GetDynamicType and friends.

So most of this change is just moving things from DH to RH and adding “using M.S.Types;” to IronPython.  There’s one additional tweak in that I remove LanguageContext.DeleteMember.  JS was using DynamicType for its implementation and the only real caller was IronPython.  This gets replaced by a PythonOps.DeleteAttr which closely mimics the TryGetAttr method we already have.

Also some code only used by Python (CheckTypeVersion) gets pulled up into PythonOps and a couple of places where we stop getting type names from DynamicType.

295467: (jomes) inject onto IList vs. IList<object>

Pretty simple, just changes the IList<object> member injectors to inject onto IList instead, because a lot more things implement IList (especially notable are ArrayList and List<T> for any type T).

I needed a couple of minor bug fixes—the interface injection was injecting things onto non-concrete generic types. I also changed the generator back to using a sorted dictionary so Initializer.Generated.cs doesn’t change so much each time (which you can see if you look at the diff, some stuff changed that shouldn’t have).

295500: (dinov)

This starts exposing MethodGroup’s instead of BuiltinFunction’s.

First off one “big” change is FastCallable is getting moved into M.S.Types (which will get pulled up).  This also causes a bunch of files to get using M.S.Types for references to FastCallable.  CallType remains in Microsoft.Scripting as it’s a component of the MethodBinder.

This change alters some MemberTracker setup.  I’ve made MemberGroup’s no longer a MemberTracker.  Instead MemberGroup’s are just ways that we talk about groups of members.   I’ve added a MethodGroup which is a collection of methods.  MethodGroup’s are also unique per actual .NET method (and generic instantiation for generic methods).  That allows us to use a simple comparison on them to check for identity (this is enforced by the ReflectionCache).

From here it’s basically pretty straight-forward: The DLR gets MemberGroup’s, for method binding it creates MethodGroup’s (once we’re resolved to just a bunch of methods).  All languages except for Python currently expose MethodGroup’s directly to the user.  Python transforms these into BuiltinFunction’s.

There’s still more work to be done: This change doesn’t remove CallBinderHelper’s recognition of BuiltinFunction/BuiltinMethodDescriptor’s.  We should also more strictly base Python’s identity of BuiltinFunction’s on MethodGroup’s and then we can also use simple object identity for tests against BuiltinFunction’s – currently Python is sharing the hash-key w/ ReflectionCache which isn’t necessary.  Finally we don’t have perfect one-to-one mappings between methods and MethodTracker’s.  This is because we use the MethodInfo as the key which can be duplicated based upon from what type you got the MethodInfo.  But this change is already big enough that it seemed prudent to wait on those next steps.

296822: (jomes) Ruby Class Variables

Implements support for Ruby class variables (aka static fields)

Mostly, this is straightforward. We add a dictionary to RubyModule for storing the values, and add methods for getting and setting the variables. The class variable AST node turns into get/set calls. Ruby resolves class variables in method resolution order, so it searches all base classes & mixins. Setting variables is interesting—the MRO is searched, and if the variable is found it’s modified, otherwise, it’s added to the current Module/Class.

A few AST changes are needed—unlike instance variables (which are resolved on the current “self”), class variables are lexically bound to the enclosing module/class. So we need scoping for modules, and methods need the ability to be closures. We close over the module’s “self”, but because the DLR merges variables by name, we need to generate a new name for each module. Also, DLR closures don’t work unless you flow in the correct CodeContext, so RubyMethodInfo’s need to hold on to their CodeContext, and flow in the right one.

297031: (jomes) ruby monkey patching of .NET types

A few minor fixes to allow Ruby to add methods to .NET types, aka “monkey patch”. The reason this wasn’t working is that Ruby exposes .NET namespaces as NamespaceTrackers & .NET types as TypeTrackers. We were already converting TypeTrackers to RubyClass under certain circumstances; now we also convert NamespaceTrackers to RubyModules. For this to work, a RubyModule can have a NamespaceTracker backing it. When we go to lookup a constant on the module, we check the module’s constants, and then fall back to the namespace.

This design is not optimal... we need to come up with a better way for language’s type objects to interoperate. But it’s intended to unblock some of our .NET interop scenarios in the short term.

297060: (jomes) fix scoping of constant variables

Ruby constants are supposed to bind lexically to the enclosing class/module, but this wasn’t working. Pretty simple change, ConstantVariable now binds to the lexically enclosing module, or Object if an enclosing module isn’t present.

There was already a test for this, but it wasn’t running against IronRuby, so I enabled it & added a few more test cases.

297066: (jomes) Ruby instance_eval/class_eval/module_eval support (for the non-string overloads)

Implements basic support for the instance_eval, class_eval, module_eval overloads that take a block (not the ones that take strings, that requires full “eval” support). This is pretty easy to do, because we just create a new Proc with a different “self” and call the block.

instance_eval on a Module/Class seems to work a bit differently, so I’m throwing NotImplemented until we can make the necessary infrastructure changes to handle it correctly.

297179: (dinov)

Next change in the line of changes to remove DynamicType.  This one removes ReflectedEvent/BoundEvent from the parlance of the DLR:

ReflectionCache.GetReflectedEvent moves to DynamicTypeOps and callers are updated.
Python gets a PythonSetMemberBinderHelper for dealing w/ the assignment logic back to a ReflectedEvent.
We now pass the type we’re doing the lookup from for ReturnMemberTracker (this is our version of the .NET ReflectedType property which shows where each member came from)
The DLR currently generates calls to static helper methods for doing the addition/subtraction to the event object.
Fixed an issue where getting a TypeGroup’s DeclaringType/Name could throw
And finally manifest the in-place add/subtract methods onto EventTracker’s and BoundEventTrackers (these aren’t directly defined to deal w/ typing issues where we need the delegate to be strongly typed).

2975790: (jflam) Enables support for Rubinius spec suite

Adds Rubinius spec suite for Array, Hash and String. Dir is also added, but tests are not enabled for this yet.

Adds a modified version of the Rubinius spec runner that uses only features implemented by IronRuby. These are the mini_mock, mini_rspec, mspec, mspec_helper, rspec_helper, simple_mock, and spec_helper files in \Tests.

Adds/fixes features to IronRuby to support running the test suite:
  - Fixes require behavior in loader.cs to allow files to be loaded more than once
  - Fixes __FILE__
  - Adds File.dirname() method
  - Adds RubyExtensionModuleAttribute and marks IListOps, IComparableOps, IDictionaryOps, IEnumerableOps using this attribute.
  - Adds implementation of const_defined?, alias_method and remove_method

Adds a new utility (IronRuby.Libraries.Scanner) that generates a YAML file that contains a list of all implemented singleton and instance methods (note that this is written in C# 3.0).

297638: (mmaly) Removing callwiththis

I’ve had CallWithThisExpression in my sight for a while now and as Nandan and Jitu made progress on the binder, it is no longer used so it can go now.
Additionally, MSAst.Arg was used only as a temporary information storage and was never actually part of the AST (CallSignature would throw it away) so I got rid of it too. In Python it meant even cleaner code.

297654: (jomes) splat and send

This change enables splatting of argument lists when calling Ruby library methods that are implemented in C#. Before this change, splatting only worked if you’re calling methods defined in Ruby. Fixing this is pretty straightforward: I just flatten out any splatted lists before passing them to the MethodBinder. (The method binder expects things flattened down to Types—it doesn’t use ArgumentKinds).

I also implement Ruby’s send and __send__ methods, which are used by a lot of tests that John Lam wants to enable. The implementations are very simple & slow: they just create a new site with the correct InvokeMemberAction and Invoke it. Not optimal, but it should be okay until we integrate send directly into the method binder (Tomas is already planning on refactoring the method binder).

297839: (tomat) Ruby Singletons and Method Lookup

Adds support for singleton classes and fixes method lookup. A singleton class is a class that adds additional members to object instances. In the following example, “class << x” defines a singleton class for instance x.

x = Object.new
class << x
  def foo

Object.new.foo # error

Singleton class is a first class object, you can store it to a variable and even create another singleton class for it. The class itself is also defining some methods. Those are implemented as extension methods in SingletonOps.

There are many peculiarities involved. I’ve also found at least 2 bugs in MRI during my investigation.
I’ll write a spec and present the internals of singletons and method lookup (as I understand them) on Friday’s IronRuby meeting.

The shelveset also implementation rescue expression, rescue statement and symbol constructors, which are handy for testing.

297943: (dinov) Remove references to built-in functions

This change removes the remaining references to BuiltinFunction’s (and friends).  Instead these objects now implement IDynamicObject and use the CallBinderHelper to produce a call rule.

Supporting changes:
1.      Needed to split RuntimeHelpers into 2 parts (RuntimeHelpers and BinderOps).  This is due to a FxCop error where RuntimeHelpers is getting too complex and including things from too many classes.  BinderOps are just the methods that are called from generated code to do binding (calls, create instance, get member, etc…)
2.      Moved StrongBox helper functions to CompilerHelpers as these need to be accessed outside the binder.
3.      Added support for passing isBinaryOperator/isReversedOperator on CallBinderHelper as well as ability to set Instance.  The hope here is that these flags go away from the MethodBinder (they’re awfully Python specific) and that we have a more consumable way to do call binding in the future but this is a reasonable intermediate step.  The CallBinderHelper also no longer stores the instance type instead getting it from the _instance expression.

297991: (dinov)

This moves the Microsoft.Scripting.Types namespace to IronPython.  No namespaces are renamed during this remove (that touches over 100 files, so it’s next).  Appropriate resource strings also get moved to IronPython, and much of the DynamicType machinery is marked as internal.

This reduces a release build of MS.Scripting to 868,352 bytes from 958,464 saving 90,112 bytes or 9.4%.
This reduces a debug build of MS.Scripting to 995,328 bytes from 1,118,208 saving 122,880 bytes or 10.9%.

298014: (jomes) Case Expressions in Ruby

This change implements case expressions in  Ruby. Case expressions look like this:

case x
  when a0, … aN, *splat0: <body0>
  when b0, … bN, *splat1: <body1>
  else: <bodyN>

It translates roughly into:

if (a0 === x || … || aN === x || <testarray>) {
} else if ( … ) {
} else {

Where <testarray> is:

result = false;
foreach (object obj in splat0) {
                if (obj === x) { result = true; break; }

And of course, === is Ruby’s case equality method 

It’s pretty straightforward—the only tricky part is cracking open the splatted array, if present.

More information about the Ironruby-core mailing list