[Ironruby-core] Unicode Source Files

Ted Milker tmilker at gmail.com
Sun Oct 26 12:37:37 EDT 2008


Is the DLR going to be fixed so that it properly supports Unicode
source files or is this an issue with IronRuby?  If you attempt to
create a new Code File with Visual Studio 2008 and call it test.rb and
then execute it with:

ScriptRuntime runtime = IronRuby.Ruby.CreateRuntime();
runtime.ExecuteFile( "test.rb" );

it blows up on the Unicode byte-order marker with:

Unhandled Exception: Microsoft.Scripting.SyntaxErrorException: Invalid
character 'ï' in expression
   at Microsoft.Scripting.ErrorSink.Add(SourceUnit source, String
message, SourceSpan span, Int32 errorCode, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line
34
   at Microsoft.Scripting.ErrorCounter.Add(SourceUnit source, String
message, SourceSpan span, Int32 errorCode, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line
92
   at IronRuby.Compiler.Tokenizer.Report(String message, Int32
errorCode, SourceSpan location, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
430
   at IronRuby.Compiler.Tokenizer.ReportError(ErrorInfo info, Object[]
args) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
442
   at IronRuby.Compiler.Tokenizer.Tokenize(Boolean whitespaceSeen,
Boolean cmdState) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
966
   at IronRuby.Compiler.Tokenizer.Tokenize() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
739
   at IronRuby.Compiler.Tokenizer.GetNextToken() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
711
   at IronRuby.Compiler.Parser.GetNextToken() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line
99
   at IronRuby.Compiler.ShiftReduceParser`2.Parse() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\GPPG.cs:line
310
   at IronRuby.Compiler.Parser.Parse(SourceUnit sourceUnit,
RubyCompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line
158
   at IronRuby.Runtime.RubyContext.ParseSourceCode(SourceUnit
sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line
203
   at IronRuby.Runtime.RubyContext.CompileSourceCode(SourceUnit
sourceUnit, CompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line
179
   at Microsoft.Scripting.SourceUnit.Compile(CompilerOptions options,
ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line
215
   at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink
errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line
225
   at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope
scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptSource.cs:line
129
   at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String
path, ScriptScope scope) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line
159
   at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String
path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line
148
   at Microsoft.Scripting.Hosting.ScriptRuntime.ExecuteFile(String
path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptRuntime.cs:line
257
   at HostingDLRConsole.Program.Main(String[] args) in
C:\Users\ted\Documents\Visual Studio 2008\Projects\Books\IronRuby in
Action\HostingDLRConsole\HostingDLRConsole\Program.cs:line 14
Press any key to continue . . .

I know I can fix this by using the Advanced Save Options but the DLR
spec talks about Unicode support, so I assume this means that
ScriptRuntime.ExecuteFile() should also support Unicode source files.


More information about the Ironruby-core mailing list