[Rubygems-developers] pure-ruby vs. compile-on-intsall

Trans transfire at gmail.com
Wed Nov 7 09:08:28 EST 2007

On 7 Nov 2007 13:37:30 +0100, Florian Frank <flori at nixe.ping.de> wrote:
> Trans wrote:
> >>>> I'm a bit confused. If I have a pure-ruby version of my lib, but also
> >>>> have some extensions that can speed things up, should the pure-ruby
> >>>> gem be named plainly? Eg. 'foo-1.0.0.gem'.  But then what platform is
> >>>> the compile-on-install gem? Am I going about this wrong? I'm starting
> >>>> to think it would be easier to create two packages, one for the
> >>>> pure-ruby gem and another for the optional extensions.
> >>>>
> >
> > I need to provide these:
> >
> >   * pure ruby version
> >   * compile-on-install version
> >   * win32 pre-built version
> >
> > I would like to make the first two a single gem, rather then two
> > separate gems. To do that I need RubyGems not to abort if make fails.
> > I am going to try to figure out a way to do that. But if someone
> > already has an idea how, please let me know.
> >
> If you figure this out, please post the solution to the list. I had/have
> exactly the same problem with my json gem, which offers a pure ruby
> implementation variant and a faster c extension variant.
> I have used another name (json_pure) for the pure ruby variant, which
> solves one problem: You now can pick the json variant, if your platform
> supports compilation, and you can pick json_pure if you have to use the
> pure version. Ironically my json gem would automatically fallback to the
> pure variant anyway, if the shared objects couldn't be loaded. So the
> json_pure variant contains almost the same files as the json variant
> does, and the json variant includes the pure ruby version, which can be
> used by requiring 'json/pure' instead of requiring 'json'.
> Another problem isn't solved by this workaround, though. If some other
> developer adds json as a dependency, it will resolve to the compile
> variant, but the resolution will fail altogether on a platform like
> jruby, which doesn't support compilation.
> I have learned that merb ran into this problem, they now use json_pure
> as a dependency. This is far from optimal. I have made made a RubyGems
> feature request for a pure ruby platform
> (http://rubyforge.org/tracker/index.php?func=detail&aid=13290&group_id=126&atid=578)
> a while ago, in order to address this problem, but it was closed without
> a real solution.
> It's unfortunate, that RubyGems conflates gems, that are pure ruby with
> those that will trigger a compilation process, but I guess it's too late
> to change this now. In the case of json, it would be indeed be enough,
> if RubyGems would soldier on and just install the pure ruby stuff
> anyway, if compilation fails. But this isn't guaranteed to work for
> other gems, that don't have a pure ruby fallback.
> The gem command could do a better job to resolve dependencies as well:
> It could first figure out, if the platform under which is running,
> supports compilation. If the json dependency is being resolved on
> platform like jruby, that doesn't support compilation like jruby, it
> should first look for a binary platform specific gem, e. g. jruby
> platform. If none is found, it should look for a pure platform variant
> and try to install that one. If there is no pure platform found, it
> should assume, that the ruby platform gem is actually a pure ruby
> implementation and try to install that one. This could still fail, but
> this would be the same behaviour, that is happening now.

It can get tricky. Gems really needs to offer a switch as well so the
installer can select the variant.

I managed a partial solution. There did not seem to be a way to rescue
the compile failure b/c Rubygems is shelling out the execution of
extconf.rb. So instead I made prints a message when compiling fails
that tells the user they can still use the library in pure-ruby mode
by setting an environment variable and installing again.

My extconf.rb looks like this:

  require 'mkmf'
  require 'rbconfig'

  extension_name = 'base64'

  arch = Config::CONFIG['arch']

  FailedMessage.replace("Could not create Makefile, probably for the
lack of necessary libraries and/or headers. Check the mkmf.log file
for more details. You may need configuration options (see below).
TMail has a pure-ruby fallback mode, so you can still use this
library. To do so, set the environment variable, export
NORUBYEXT='true', and gem install again.\n\n")

  if ENV['NORUBYEXT'] == 'true'
    # Rubygems is sending all output to dev/null :(
    STDOUT << "NORUBYEXT option is set to true. Native extension will
be omitted."
    File.open('Makefile', 'w') do |f|
      f << "all:\n"
      f << "install:\n"
    if (/mswin/ =~ RUBY_PLATFORM) and ENV['make'].nil?
      $LIBS += " msvcprt.lib"
      create_makefile(extension_name, "tmail/#{arch}")
      $CFLAGS += " -D_FILE_OFFSET_BITS=64"  #???
      create_makefile(extension_name, "tmail/#{arch}")

As you can see I just made a dummy Makefile that does nothing if
RUBYNOEXT is set to 'true' (Hmm... I should probably give the variable
a mode specific name, anyway...) You'll also notice I use 'arch' for
the lib destination. I use a #require_arch method in my code to first
attempt to load the native code:

  def require_arch(fname)
    arch = Config::CONFIG['arch']
      path = File.join("tmail", arch, fname)
      require path
    rescue LoadError => e
      # try pre-built Windows binaries
      if arch =~ /mswin/
        require File.join("tmail", 'mswin32', fname)
        raise e

I'm still running it through some paces but it seems to be working
fine. It's not the ideal solution, but it's workable for now.

The ideal solution would be for RubyGems to add a 'ruby_ext_fallback'
option to the gemspec. I have a quick patch if the RubyGem team is
interested. I believe it's as simple adding an

   if @spec.ruby_ext_fallback

to Installer#build_extensions method.


More information about the Rubygems-developers mailing list