[Ironruby-core] Unicode Source Files

Curt Hagenlocher curth at microsoft.com
Sun Oct 26 12:52:59 EDT 2008


We do this for compatibility with Ruby 1.8.6, though as you can see, we don't have the error message quite right:

PS F:\> C:\ruby\bin\ruby.exe x.rb
x.rb:1: Invalid char `\377' in expression
x.rb:1: Invalid char `\376' in expression

:)

I believe you'll need to save as UTF-8 and then manually strip the BOM in order to use Unicode source files -- hopefully Tomas will tell me if I'm wrong.

Source encoding for Ruby is extremely tricky, and (from what I can tell) hasn't even yet been finalized for 1.9.x.  We will eventually support whatever the Ruby standards are.

-----Original Message-----
From: ironruby-core-bounces at rubyforge.org [mailto:ironruby-core-bounces at rubyforge.org] On Behalf Of Ted Milker
Sent: Sunday, October 26, 2008 9:38 AM
To: ironruby-core at rubyforge.org
Subject: [Ironruby-core] Unicode Source Files

Is the DLR going to be fixed so that it properly supports Unicode
source files or is this an issue with IronRuby?  If you attempt to
create a new Code File with Visual Studio 2008 and call it test.rb and
then execute it with:

ScriptRuntime runtime = IronRuby.Ruby.CreateRuntime();
runtime.ExecuteFile( "test.rb" );

it blows up on the Unicode byte-order marker with:

Unhandled Exception: Microsoft.Scripting.SyntaxErrorException: Invalid
character 'ï' in expression
   at Microsoft.Scripting.ErrorSink.Add(SourceUnit source, String
message, SourceSpan span, Int32 errorCode, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line
34
   at Microsoft.Scripting.ErrorCounter.Add(SourceUnit source, String
message, SourceSpan span, Int32 errorCode, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\ErrorSink.cs:line
92
   at IronRuby.Compiler.Tokenizer.Report(String message, Int32
errorCode, SourceSpan location, Severity severity) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
430
   at IronRuby.Compiler.Tokenizer.ReportError(ErrorInfo info, Object[]
args) in C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
442
   at IronRuby.Compiler.Tokenizer.Tokenize(Boolean whitespaceSeen,
Boolean cmdState) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
966
   at IronRuby.Compiler.Tokenizer.Tokenize() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
739
   at IronRuby.Compiler.Tokenizer.GetNextToken() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Tokenizer.cs:line
711
   at IronRuby.Compiler.Parser.GetNextToken() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line
99
   at IronRuby.Compiler.ShiftReduceParser`2.Parse() in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\GPPG.cs:line
310
   at IronRuby.Compiler.Parser.Parse(SourceUnit sourceUnit,
RubyCompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Compiler\Parser\Parser.cs:line
158
   at IronRuby.Runtime.RubyContext.ParseSourceCode(SourceUnit
sourceUnit, RubyCompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line
203
   at IronRuby.Runtime.RubyContext.CompileSourceCode(SourceUnit
sourceUnit, CompilerOptions options, ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\ironruby\Runtime\RubyContext.cs:line
179
   at Microsoft.Scripting.SourceUnit.Compile(CompilerOptions options,
ErrorSink errorSink) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line
215
   at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink
errorSink) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\SourceUnit.cs:line
225
   at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope
scope) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptSource.cs:line
129
   at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String
path, ScriptScope scope) in
C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line
159
   at Microsoft.Scripting.Hosting.ScriptEngine.ExecuteFile(String
path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptEngine.cs:line
148
   at Microsoft.Scripting.Hosting.ScriptRuntime.ExecuteFile(String
path) in C:\Users\ted\Desktop\IronRuby\src\Microsoft.Scripting\Hosting\ScriptRuntime.cs:line
257
   at HostingDLRConsole.Program.Main(String[] args) in
C:\Users\ted\Documents\Visual Studio 2008\Projects\Books\IronRuby in
Action\HostingDLRConsole\HostingDLRConsole\Program.cs:line 14
Press any key to continue . . .

I know I can fix this by using the Advanced Save Options but the DLR
spec talks about Unicode support, so I assume this means that
ScriptRuntime.ExecuteFile() should also support Unicode source files.
_______________________________________________
Ironruby-core mailing list
Ironruby-core at rubyforge.org
http://rubyforge.org/mailman/listinfo/ironruby-core



More information about the Ironruby-core mailing list