[Rubygems-developers] [ rubygems-Bugs-28551 ] UTF-8 files break Gem builds under Ruby 1.9.0

Thu Sep 9 11:10:09 EDT 2010

Bugs item #28551, was opened at 2010-09-08 21:34
You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=575&aid=28551&group_id=126

Category: `gem install` command
>Group: v1.3.x
>Status: Closed
>Resolution: Rejected
Priority: 3
Submitted By: chocolate boy (chocolateboy)
>Assigned to: Eric Hodel (drbrain)
Summary: UTF-8 files break Gem builds under Ruby 1.9.0

Initial Comment:
I've spent the past couple of hours trying to get "rake gem" to build a valid gem for this project:

http://github.com/nricciar/wikicloth

Each time, gem install pkg/wikicloth-0.2.0.gem would fail with:

ERROR:  While executing gem ... (ArgumentError)
    string contains null byte

gem install --backtrace pkg/wikicloth-0.2.0.gem gives:

ERROR:  While executing gem ... (ArgumentError)
    string contains null byte
	/usr/lib/ruby/1.9.0/rubygems/package/tar_reader/entry.rb:66:in `join'
	...
	/usr/lib/ruby/1.9.0/rubygems/gem_runner.rb:58:in `run'
	/usr/bin/gem1.9:21:in `<main>'

Placing a puts statement in the join method revealed a corrupt header:

Good header:

#<Gem::Package::TarHeader:0x97f9064 @checksum=7592, @devmajor=0, @devminor=0, @gid=0, @gname="wheel", @linkname="", @magic="ustar", @mode=420, @mtime=0, @name="sample_documents/lists.wiki", @prefix="", @size=12390, @typeflag="0", @uid=0, @uname="wheel", @version=0, @empty=false>

Bad header:

#<Gem::Package::TarHeader:0x97f7c64 @checksum=95, @devmajor=0, @devminor=0, @gid=420, @gname="\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00wheel", @linkname="000000\x00020307\x00 0", @magic="", @mode=0, @mtime=0, @name="\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00sample_documents/air_force_one.wiki", @prefix="0000000\x000000000", @size=0, @typeflag="0", @uid=0, @uname="\x00\x00\x00\x00\x00\x00\x00\x00ustar\x0000wheel", @version=0, @empty=false>

Although the "file" command flags both of these as UTF-8, it seems that the first UTF-8 entry corrupts the following entry. Removing UTF-8 files from the sample_documents directory solves the problem (a valid data.tar gzipped inside the gem) and placing any of the UTF-8 files in the samples directory, even on its own, produces an invalid tarball.

After a couple of hours of tracing gem creation through the maze of:

/usr/lib/ruby/1.9.0/rubygems/builder.rb
/usr/lib/ruby/1.9.0/rubygems/package.rb
/usr/lib/ruby/1.9.0/rubygems/package/tar_header.rb
/usr/lib/ruby/1.9.0/rubygems/package/tar_output.rb
/usr/lib/ruby/1.9.0/rubygems/package/tar_reader/entry.rb
/usr/lib/ruby/1.9.0/rubygems/package/tar_writer.rb

- trying to find the bug, I finally relented and installed rubygems1.8 (just to build the gem) and it worked.

The rubygems1.8 package on Ubuntu (10.04) is exactly the same version as rubygems1.9: 1.3.5. All the files above are the same. They just differ in the path and the Ruby version: 1.8.7 vs 1.9.0.

This may be fixed in 1.9.1 or 1.9.2, but as it stands: RubyGems can't build a valid tarball on 1.9.0 when confronted with UTF-8 files.

Ubuntu 10.04
Ruby 1.9.0
RubyGems 1.3.5
(Rake 0.8.7)

RubyGems Environment:
  - RUBYGEMS VERSION: 1.3.5
  - RUBY VERSION: 1.9.0 (2008-10-04 patchlevel 0) [i486-linux]
  - INSTALLATION DIRECTORY: ~/.gem/ruby/1.9.0
  - RUBY EXECUTABLE: /usr/bin/ruby1.9
  - EXECUTABLE DIRECTORY: ~/.gem/ruby/1.9.0/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-linux
  - GEM PATHS:
     - ~/.gem/ruby/1.9.0
     - /var/lib/gems/1.9.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://gems.rubyforge.org/

wikicloth$ gem --debug install pkg/wikicloth-0.2.0.gem 
Exception `NameError' at /usr/lib/ruby/1.9.0/rubygems/command_manager.rb:161 - uninitialized constant Gem::Commands::InstallCommand
Exception `LoadError' at /usr/lib/ruby/1.9.0/rubygems/gem_openssl.rb:40 - no such file to load -- openssl
Exception `Gem::LoadError' at /usr/lib/ruby/1.9.0/rubygems.rb:827 - Could not find RubyGem test-unit (>= 0)

Exception `NoMethodError' at /usr/lib/ruby/1.9.0/rubygems/package/tar_input.rb:48 - undefined method `readpartial' for #<Gem::Package::TarReader::Entry:0x9f9e508>
Exception `ArgumentError' at /usr/lib/ruby/1.9.0/rubygems/package/tar_reader/entry.rb:66 - string contains null byte
Exception `IOError' at /usr/lib/ruby/1.9.0/rubygems/format.rb:50 - closed stream

----------------------------------------------------------------------

>Comment By: Eric Hodel (drbrain)
Date: 2010-09-09 08:10

Message:
RubyGems is not supported on Ruby 1.9.0.

Ruby 1.9.0 is not a production-ready release.

You will need to upgrade your ruby.

----------------------------------------------------------------------

You can respond by visiting: 
http://rubyforge.org/tracker/?func=detail&atid=575&aid=28551&group_id=126