From luislavena at gmail.com Sat Jan 7 14:00:29 2012 From: luislavena at gmail.com (Luis Lavena) Date: Sat, 7 Jan 2012 16:00:29 -0300 Subject: There is no more annoucements on releases? Message-ID: Hey guys, Don't want to be a PITA but since RubyGems 1.8.10 there has been no announcement to this list of any of the releases: 1.8.11 - 1.8.15 (which happen yesterday) Is not easy to point users when they encounter a problem to upgrade RubyGems because this was fixed (because there is no announcement that states something was fixed) I understand that we are all busy, but since RubyGems uses Hoe, sending the email shouldn't be a problem. Thank you for your time. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From evan at fallingsnow.net Sat Jan 7 17:01:39 2012 From: evan at fallingsnow.net (Evan Phoenix) Date: Sat, 7 Jan 2012 14:01:39 -0800 Subject: [ANN] rubygems-update 1.8.15 Released Message-ID: <5437AF2BBACE42B29742F88B7E80F0D9@phx.io> rubygems-update version 1.8.15 has been released! * * * * * RubyGems is a package management framework for Ruby. This gem is an update for the RubyGems software. You must have an installation of RubyGems before this update can be applied. See Gem for information on RubyGems (or `ri Gem`) To upgrade to the latest RubyGems, run: $ gem update --system # you might need to be an administrator or root See UPGRADING.rdoc for more details and alternative instructions. ----- If you don't have RubyGems installed, your can still do it manually: * Download from: https://rubygems.org/pages/download * Unpack into a directory and cd there * Install with: ruby setup.rb # you may need admin/root privilege For more details and other options, see: ruby setup.rb --help Changes: ### 1.8.15 / 2012-01-06 * 1 bug fix: * Don't eager load yaml, it creates a bad loop. Fixes #256 Evan Phoenix // evan at fallingsnow.net From evan at fallingsnow.net Sat Jan 7 17:02:59 2012 From: evan at fallingsnow.net (Evan Phoenix) Date: Sat, 7 Jan 2012 14:02:59 -0800 Subject: There is no more annoucements on releases? In-Reply-To: References: Message-ID: <58CF20636E65456B8F61B4D1A155DC87@phx.io> That's all on me. I haven't been sending them out for the last releases. I promise I'll start doing so again. I've just sent out the one for 1.8.15. - Evan Evan Phoenix // evan at fallingsnow.net On Saturday, January 7, 2012 at 11:00 AM, Luis Lavena wrote: > Hey guys, > > Don't want to be a PITA but since RubyGems 1.8.10 there has been no > announcement to this list of any of the releases: 1.8.11 - 1.8.15 > (which happen yesterday) > > Is not easy to point users when they encounter a problem to upgrade > RubyGems because this was fixed (because there is no announcement that > states something was fixed) > > I understand that we are all busy, but since RubyGems uses Hoe, > sending the email shouldn't be a problem. > > Thank you for your time. > > -- > Luis Lavena > AREA 17 > - > Perfection in design is achieved not when there is nothing more to add, > but rather when there is nothing more to take away. > Antoine de Saint-Exup?ry > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > http://rubyforge.org/mailman/listinfo/rubygems-developers From luislavena at gmail.com Sat Jan 7 17:09:03 2012 From: luislavena at gmail.com (Luis Lavena) Date: Sat, 7 Jan 2012 19:09:03 -0300 Subject: There is no more annoucements on releases? In-Reply-To: <58CF20636E65456B8F61B4D1A155DC87@phx.io> References: <58CF20636E65456B8F61B4D1A155DC87@phx.io> Message-ID: On Sat, Jan 7, 2012 at 7:02 PM, Evan Phoenix wrote: > That's all on me. I haven't been sending them out for the last releases. > > I promise I'll start doing so again. I've just sent out the one for 1.8.15. > Thank you, I really don't want to be a pain but the release announcements are important for me to keep track of packages that are part of RubyInstaller or Ruby itself. For example, we agreed on heads up on releases so I can run tests against Windows and ensure nothing mayor is going to broke for us. Once again, thank you and have a nice weekend. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From headius at headius.com Mon Jan 9 13:59:20 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Mon, 9 Jan 2012 12:59:20 -0600 Subject: Dealing with JRuby and jars Message-ID: Ok, we need to figure out a solution here :) Here's the deal: JRuby users are using JVM libraries. That's obviously a big selling point of JRuby. They're also living in the Ruby world, and use RubyGems, Bundler, etc. They don't want to have to manage dependencies multiple ways. Ruby world revolves around RubyGems and Bundler. Java world revolves around Maven. We want to integrate the two for JRuby users in a way that RubyGems, RubyGems.org, and Bundler folks can accept. JRuby currently supports installing jars from Maven at the "gem" command line, but that's the only place it works. RubyGems.org does not allow us to push gems with maven dependencies, and Bundler does not recognize maven dependencies. What we need: * A way for RubyGems to understand Maven artifacts and dependency graphs and fetch them. We have that at the command line, but it's via our own patches. * A way for RubyGems.org to allow gems that have Maven dependencies. The alternatives to this are pushing gems that *just* wrap a jar file (some of them very large) or hosting a JRuby-specific RubyGems.org. I don't like either of those options. * A way for Bundler to recognize Maven dependencies as though they are gems. I'm looking for thoughts and suggestions on how we can make this easier. It's one of the biggest headaches for JRuby users right now, since they have to use multiple mechanisms to track dependencies. I also don't want to flood RubyGems.org with a bunch of giant jar-only gems when we can simply source them from Maven mirrors. So...what do you think? - Charlie From evan at phx.io Mon Jan 9 15:04:43 2012 From: evan at phx.io (Evan Phoenix) Date: Mon, 9 Jan 2012 12:04:43 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: <547022838FF642F8A5A2BB763E531486@phx.io> See below. -- Evan Phoenix // evan at phx.io On Monday, January 9, 2012 at 10:59 AM, Charles Oliver Nutter wrote: > Ok, we need to figure out a solution here :) > > Here's the deal: > > JRuby users are using JVM libraries. That's obviously a big selling > point of JRuby. They're also living in the Ruby world, and use > RubyGems, Bundler, etc. They don't want to have to manage dependencies > multiple ways. > > Ruby world revolves around RubyGems and Bundler. Java world revolves > around Maven. We want to integrate the two for JRuby users in a way > that RubyGems, RubyGems.org (http://RubyGems.org), and Bundler folks can accept. > > JRuby currently supports installing jars from Maven at the "gem" > command line, but that's the only place it works. RubyGems.org (http://RubyGems.org) does > not allow us to push gems with maven dependencies, and Bundler does > not recognize maven dependencies. > > What we need: > > * A way for RubyGems to understand Maven artifacts and dependency > graphs and fetch them. We have that at the command line, but it's via > our own patches. > > * A way for RubyGems.org (http://RubyGems.org) to allow gems that have Maven dependencies. > The alternatives to this are pushing gems that *just* wrap a jar file > (some of them very large) or hosting a JRuby-specific RubyGems.org (http://RubyGems.org). I > don't like either of those options. This currently fails because rubygems.org makes sure that there is an existing gem on rubygems.org for each requested dependency. It only checks the presence, not the versions required by the dependency. Given that it's just checking the name and thusly not checking if the gem is actually usable with the requested dependencies, I think we should go ahead and remove the requirement all together. If we want to be nice to the user and indicate to them that their may be a problem with their gem, it's best to do that at 'gem push' time rather than enforcing the policy on rubygems.org. > > * A way for Bundler to recognize Maven dependencies as though they are gems. > > I'm looking for thoughts and suggestions on how we can make this > easier. It's one of the biggest headaches for JRuby users right now, > since they have to use multiple mechanisms to track dependencies. I > also don't want to flood RubyGems.org (http://RubyGems.org) with a bunch of giant jar-only > gems when we can simply source them from Maven mirrors. > > So...what do you think? > > - Charlie > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > http://rubyforge.org/mailman/listinfo/rubygems-developers From luislavena at gmail.com Mon Jan 9 15:07:53 2012 From: luislavena at gmail.com (Luis Lavena) Date: Mon, 9 Jan 2012 17:07:53 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 3:59 PM, Charles Oliver Nutter wrote: > Ok, we need to figure out a solution here :) > Let's see, Bear with me my naive answers... > Here's the deal: > > JRuby users are using JVM libraries. That's obviously a big selling > point of JRuby. They're also living in the Ruby world, and use > RubyGems, Bundler, etc. They don't want to have to manage dependencies > multiple ways. > > Ruby world revolves around RubyGems and Bundler. Java world revolves > around Maven. We want to integrate the two for JRuby users in a way > that RubyGems, RubyGems.org, and Bundler folks can accept. > > JRuby currently supports installing jars from Maven at the "gem" > command line, but that's the only place it works. RubyGems.org does > not allow us to push gems with maven dependencies, and Bundler does > not recognize maven dependencies. > What about a extension to Gem::Installer provided by JRuby that uses Gem::Specification metadata field? (of Course, RubyGems master at this time) > What we need: > > * A way for RubyGems to understand Maven artifacts and dependency > graphs and fetch them. We have that at the command line, but it's via > our own patches. > Gem::Installer provides pre and post install hooks. You can't use rubygems/defaults/#{RUBY_ENGINE}.rb ? > * A way for RubyGems.org to allow gems that have Maven dependencies. > The alternatives to this are pushing gems that *just* wrap a jar file > (some of them very large) or hosting a JRuby-specific RubyGems.org. I > don't like either of those options. > I vote for metadata usage. > * A way for Bundler to recognize Maven dependencies as though they are gems. > Dunno how Bundler installer gems, so can't comment. I was guess it uses Gem::Installer and didn't reinvent the wheel. (Isolate at least uses it) > I'm looking for thoughts and suggestions on how we can make this > easier. It's one of the biggest headaches for JRuby users right now, > since they have to use multiple mechanisms to track dependencies. I > also don't want to flood RubyGems.org with a bunch of giant jar-only > gems when we can simply source them from Maven mirrors. > > So...what do you think? > On a different project I've started to work on a plugin that uses this metadata information to install missing binaries, similar to what you want to accomplish for JRuby. Since metadata is not available yet, what if you use spec.requirements array? Something like: Gem::Specification.new do |s| s.requirements = [ "maven:org.something.foo.whatever", "maven:junit.junit-4.10" ] end $ gem spec --local pkg/testext-0.0.1.gem requirements: - maven:org.something.foo.whatever - maven:junit.junit-4.10 rubyforge_project: That would work? -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From luislavena at gmail.com Mon Jan 9 15:14:49 2012 From: luislavena at gmail.com (Luis Lavena) Date: Mon, 9 Jan 2012 17:14:49 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 3:59 PM, Charles Oliver Nutter wrote: > > * A way for RubyGems.org to allow gems that have Maven dependencies. > The alternatives to this are pushing gems that *just* wrap a jar file > (some of them very large) or hosting a JRuby-specific RubyGems.org. I > don't like either of those options. > Forgot to mention, in current implementation, dependencies are not resolved across sources, so gems in rubygems.org will not find gems in jars.jruby.org (or whatever) -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From hgs at dmu.ac.uk Mon Jan 9 16:01:07 2012 From: hgs at dmu.ac.uk (Hugh Sasse) Date: Mon, 9 Jan 2012 21:01:07 +0000 (GMT) Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Mon, 9 Jan 2012, Luis Lavena wrote: > > Forgot to mention, in current implementation, dependencies are not > resolved across sources, so gems in rubygems.org will not find gems in > jars.jruby.org (or whatever) That raises an interesting question. I don't want to prematurely optimize, but if the servers get to ask any servers, how do we avoid this becoming an implementation of "Happy Families"? "gems.example1.com, do you have mr_bun_the_baker.jar?" "Sorry, not at home." "jars.example2.com, do you have mr_bun_the_baker.jar?" It works as a game because it doesn't scale. Would we need hints about where to look? This feels like a problem that must have been solved before, but I don't know how to classify it to search... Hugh From headius at headius.com Mon Jan 9 16:48:39 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Mon, 9 Jan 2012 15:48:39 -0600 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 2:14 PM, Luis Lavena wrote: > Forgot to mention, in current implementation, dependencies are not > resolved across sources, so gems in rubygems.org will not find gems in > jars.jruby.org (or whatever) The maven-based gems are synthetic; we ship a patched version that catches gems prefixed with "mvn" and handles them using our maven-gem logic. There's no requirement for RubyGems to change to support that, and dependencies appear to resolve just fine: system ~/projects/cloby $ gem install cloby-0.0.3-java.gem Successfully installed mvn:org.clojure:clojure-1.4.0.a.3-java Successfully installed cloby-0.0.3-java 2 gems installed I have a separate task on my list to get some small patches to RubyGems to support plugging in other types of gem sources, such as custom impls like ours. - Charlie From luislavena at gmail.com Mon Jan 9 18:52:03 2012 From: luislavena at gmail.com (Luis Lavena) Date: Mon, 9 Jan 2012 20:52:03 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: My comment was about using another rubygems.org-like to host these thin wrapper gems for maven. Please see my other comments about using requirements or metadata. Sorry for top posting. Sent from mobile. On Jan 9, 2012 8:16 PM, "Charles Oliver Nutter" wrote: > On Mon, Jan 9, 2012 at 2:14 PM, Luis Lavena wrote: > > Forgot to mention, in current implementation, dependencies are not > > resolved across sources, so gems in rubygems.org will not find gems in > > jars.jruby.org (or whatever) > > The maven-based gems are synthetic; we ship a patched version that > catches gems prefixed with "mvn" and handles them using our maven-gem > logic. There's no requirement for RubyGems to change to support that, > and dependencies appear to resolve just fine: > > system ~/projects/cloby $ gem install cloby-0.0.3-java.gem > Successfully installed mvn:org.clojure:clojure-1.4.0.a.3-java > Successfully installed cloby-0.0.3-java > 2 gems installed > > I have a separate task on my list to get some small patches to > RubyGems to support plugging in other types of gem sources, such as > custom impls like ours. > > - Charlie > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org > http://rubyforge.org/mailman/listinfo/rubygems-developers > From luislavena at gmail.com Wed Jan 11 19:55:01 2012 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 11 Jan 2012 21:55:01 -0300 Subject: A failure that escapes my comprehension Message-ID: In master, I've the following failure: ?12) Failure: test_check_executable_overwrite_default_bin_dir(TestGemInstaller) [C:/Users/Luis/Projects/oss/rubygems/rubygems/test/rubygems/test_gem_installer.rb:149]: --- expected +++ actual @@ -1 +1 @@ -"\"executable\" from a conflicts with C:/Users/Luis/Projects/oss/rubygems/rubygems/tmp/test/test_rubygems_3892/tmp/test/test_rubygems_3892/gemhome/bin/executable" +"\"executable\" from a conflicts with installed executable from conflict" The error message comes from Gem::Installer#check_executable_overwrite What I don't understand is why is triggering a ruby_executable (conflict = "installed executable...") PS: some single quotes will make more easy to spot what is the message and what represents an executable or a gem. Gem named "a" confuses in the generated language for non-english native speakers (at least it does for me) -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From headius at headius.com Thu Jan 12 13:11:49 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Thu, 12 Jan 2012 12:11:49 -0600 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 2:07 PM, Luis Lavena wrote: >> JRuby users are using JVM libraries. That's obviously a big selling >> point of JRuby. They're also living in the Ruby world, and use >> RubyGems, Bundler, etc. They don't want to have to manage dependencies >> multiple ways. >> >> Ruby world revolves around RubyGems and Bundler. Java world revolves >> around Maven. We want to integrate the two for JRuby users in a way >> that RubyGems, RubyGems.org, and Bundler folks can accept. >> >> JRuby currently supports installing jars from Maven at the "gem" >> command line, but that's the only place it works. RubyGems.org does >> not allow us to push gems with maven dependencies, and Bundler does >> not recognize maven dependencies. >> > > What about a extension to Gem::Installer provided by JRuby that uses > Gem::Specification metadata field? > (of Course, RubyGems master at this time) We want the external libraries to be fully functional gems. JRuby already supports installing them like this (I know colons are bad, we'll figure out something else): gem install mvn:org.clojure:clojure This creates a gem on the fly and installs it, and any scripts, tools, etc that use RubyGems see it as just another gem. >> What we need: >> >> * A way for RubyGems to understand Maven artifacts and dependency >> graphs and fetch them. We have that at the command line, but it's via >> our own patches. >> > > Gem::Installer provides pre and post install hooks. > > You can't use rubygems/defaults/#{RUBY_ENGINE}.rb ? I do not want to use a hook to fire a separate install process; we want those jars to be gems and have their dependencies resolved like gems. Maven artifacts come with a list of dependencies that we map into RubyGems dependencies. For example: system ~/projects/jruby $ gem install mvn:junit:junit Successfully installed mvn:org.hamcrest:hamcrest-core-1.3.r.2-java Successfully installed mvn:junit:junit-4.10.0-java 2 gems installed The junit gemspec looks like this: https://gist.github.com/1602109 >> * A way for RubyGems.org to allow gems that have Maven dependencies. >> The alternatives to this are pushing gems that *just* wrap a jar file >> (some of them very large) or hosting a JRuby-specific RubyGems.org. I >> don't like either of those options. >> > > I vote for metadata usage. I don't understand why we can't just push gems that have maven dependencies. Because they depend on Java libs, they only work on JRuby anyway. Because they're used on JRuby, the maven dependencies will resolve at install time. >> * A way for Bundler to recognize Maven dependencies as though they are gems. >> > > Dunno how Bundler installer gems, so can't comment. I was guess it > uses Gem::Installer and didn't reinvent the wheel. > (Isolate at least uses it) Yeah, I need to investigate how it works. I know it does its own dependency resolution cycle. >> I'm looking for thoughts and suggestions on how we can make this >> easier. It's one of the biggest headaches for JRuby users right now, >> since they have to use multiple mechanisms to track dependencies. I >> also don't want to flood RubyGems.org with a bunch of giant jar-only >> gems when we can simply source them from Maven mirrors. >> >> So...what do you think? >> > > On a different project I've started to work on a plugin that uses this > metadata information to install missing binaries, similar to what you > want to accomplish for JRuby. > > Since metadata is not available yet, what if you use spec.requirements array? > > Something like: > > Gem::Specification.new do |s| > ?s.requirements = [ > ? ?"maven:org.something.foo.whatever", > ? ?"maven:junit.junit-4.10" > ?] > end > > $ gem spec --local pkg/testext-0.0.1.gem > > requirements: > - maven:org.something.foo.whatever > - maven:junit.junit-4.10 > rubyforge_project: > > That would work? I don't see that it works better than us just installing them as gems. The issue I have is that by having it as a separate post-install hook, every tool that uses RubyGems or Bundler to pull in dependencies has to be modified to also know how to fetch those libraries and include them. If they're just gems, the normal logic just works... - Charlie From luislavena at gmail.com Thu Jan 12 13:29:42 2012 From: luislavena at gmail.com (Luis Lavena) Date: Thu, 12 Jan 2012 15:29:42 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Thu, Jan 12, 2012 at 3:11 PM, Charles Oliver Nutter wrote: > > We want the external libraries to be fully functional gems. JRuby > already supports installing them like this (I know colons are bad, > we'll figure out something else): > > gem install mvn:org.clojure:clojure > So what you're suggesting, even you stated differently is that you want to pollute rubygems.org with maven artifacts wrapped as gems? > > I don't understand why we can't just push gems that have maven > dependencies. Because they depend on Java libs, they only work on > JRuby anyway. Because they're used on JRuby, the maven dependencies > will resolve at install time. > Simply because a maven dependency is not Ruby nor a RubyGem? >> >> $ gem spec --local pkg/testext-0.0.1.gem >> >> requirements: >> - maven:org.something.foo.whatever >> - maven:junit.junit-4.10 >> rubyforge_project: >> >> That would work? > > I don't see that it works better than us just installing them as gems. > The issue I have is that by having it as a separate post-install hook, > every tool that uses RubyGems or Bundler to pull in dependencies has > to be modified to also know how to fetch those libraries and include > them. If they're just gems, the normal logic just works... > As long those tools use gem command line or Gem::Installer, those hooks will be fired up automatically. I have the feeling that no matter what solution we can come up with will never be enough to you. I hint that you want rubygems.org understand and host your maven dependencies. What I suggested is a way that you can add a maven-logic layer to RubyGems and be able to install those dependencies for the user, but seems clearly discarded by you. What you really want is be able to manage maven as gems from "gem" command, something that I believe goes beyond the scope of RubyGems (or Ruby) and that solves the needs of one single Ruby implementation: jRuby. Perhaps you need a way to plugin and extend Gem::Installer to fulfill jRuby needs. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From evan at phx.io Thu Jan 12 13:46:42 2012 From: evan at phx.io (Evan Phoenix) Date: Thu, 12 Jan 2012 10:46:42 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: I feel like the crux of the request, removing the dependency name check on push, has gotten lost in the discussion. Given that it currently doesn't provide a true safeguard that the gem's deps are actually satisfied by gems currently on rubygems.org, I think we should go ahead and simply remove the check all together. If users create gems that have unresolvable deps (something they can easily do now) thats on them to do. We don't have a reason to police this aspect of gem availability. Any thoughts? -- Evan Phoenix // evan at phx.io From luislavena at gmail.com Thu Jan 12 18:56:13 2012 From: luislavena at gmail.com (Luis Lavena) Date: Thu, 12 Jan 2012 20:56:13 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Thu, Jan 12, 2012 at 3:46 PM, Evan Phoenix wrote: > I feel like the crux of the request, removing the dependency name check on push, has gotten lost in the discussion. > You are right, I completely missed that point, my bad. > Given that it currently doesn't provide a true safeguard that the gem's deps are actually satisfied by gems currently on rubygems.org, I think we should go ahead and simply remove the check all together. > Couldn't find where that check is applied but if that means solving the issue for JRuby and not breaking anything else then go ahead :-) > If users create gems that have unresolvable deps (something they can easily do now) thats on them to do. We don't have a reason to police this aspect of gem availability. > People will do stupid stuff no matter what you put to block, limit or safeguard them. Cheers, -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From jeremy at hinegardner.org Thu Jan 12 23:52:57 2012 From: jeremy at hinegardner.org (Jeremy Hinegardner) Date: Thu, 12 Jan 2012 21:52:57 -0700 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: <20120113045257.GY31450@hinegardner.org> On Thu, Jan 12, 2012 at 08:56:13PM -0300, Luis Lavena wrote: > On Thu, Jan 12, 2012 at 3:46 PM, Evan Phoenix wrote: > > I feel like the crux of the request, removing the dependency name check on push, has gotten lost in the discussion. > > > > You are right, I completely missed that point, my bad. > > > Given that it currently doesn't provide a true safeguard that the gem's deps are actually satisfied by gems currently on rubygems.org, I think we should go ahead and simply remove the check all together. > > > > Couldn't find where that check is applied but if that means solving > the issue for JRuby and not breaking anything else then go ahead :-) https://github.com/rubygems/rubygems.org/blob/master/app/models/dependency.rb#L90 Currently when you push a gem, rubygems.org checks to see that the gem name that is in the dependency exists on rubygems.org as a before_validation check. I say we go ahead and remove this before_validation check. -jeremy -- ======================================================================== Jeremy Hinegardner jeremy at hinegardner.org From noreply at rubyforge.org Tue Jan 17 13:39:04 2012 From: noreply at rubyforge.org (noreply at rubyforge.org) Date: Tue, 17 Jan 2012 13:39:04 -0500 (EST) Subject: [ rubygems-Bugs-27169 ] unable to resolve transitive dependencies? Message-ID: <20120117183904.7F081121826E@rubyforge.org> Bugs item #27169, was opened at 2009-09-23 18:53 You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=575&aid=27169&group_id=126 Category: `gem install` command Group: None Status: Closed Resolution: None Priority: 3 Submitted By: Roger Pack (rogerdpack) Assigned to: Nobody (None) Summary: unable to resolve transitive dependencies? Initial Comment: Appears that if you have a gem on github which depends on gems from rubyforge, which gems from rubyforge also have dependencies on gems from rubyforge, the install fails. Ex: (a fresh install of ruby) $ gem install rogerdpack-desc_method -v0.1.5 ERROR: Error installing rogerdpack-desc_method: ruby2ruby requires ruby_parser (~> 2.0, runtime) Thanks. -r [1] http://osdir.com/ml/ruby-talk/2009-08/msg00955.html ---------------------------------------------------------------------- >Comment By: Roger Pack (rogerdpack) Date: 2012-01-17 18:39 Message: refiled this at https://github.com/rubygems/rubygems/issues/96 ---------------------------------------------------------------------- Comment By: Roger Pack (rogerdpack) Date: 2010-01-19 04:38 Message: I believe this is a duplicate of http://rubyforge.org/tracker/? func=detail&atid=575&aid=27608&group_id=126 (now fixed) Thanks. -r ---------------------------------------------------------------------- Comment By: Luis Lavena (luislavena) Date: 2009-09-23 18:55 Message: That is a known issue: there is no cross repository gem dependency resolution. Patches are welcome. ---------------------------------------------------------------------- You can respond by visiting: http://rubyforge.org/tracker/?func=detail&atid=575&aid=27169&group_id=126 From drbrain at segment7.net Tue Jan 17 17:02:53 2012 From: drbrain at segment7.net (Eric Hodel) Date: Tue, 17 Jan 2012 14:02:53 -0800 Subject: A failure that escapes my comprehension In-Reply-To: References: Message-ID: <8E9ACAA6-BAF3-4F46-83A0-A8E6CE821CE9@segment7.net> On Jan 11, 2012, at 4:55 PM, Luis Lavena wrote: > In master, I've the following failure: > > 12) Failure: > test_check_executable_overwrite_default_bin_dir(TestGemInstaller) > [C:/Users/Luis/Projects/oss/rubygems/rubygems/test/rubygems/test_gem_installer.rb:149]: > --- expected > +++ actual > @@ -1 +1 @@ > -"\"executable\" from a conflicts with > C:/Users/Luis/Projects/oss/rubygems/rubygems/tmp/test/test_rubygems_3892/tmp/test/test_rubygems_3892/gemhome/bin/executable" > +"\"executable\" from a conflicts with installed executable from conflict" > > The error message comes from Gem::Installer#check_executable_overwrite > > What I don't understand is why is triggering a ruby_executable > (conflict = "installed executable?") I think it is because the windows executables have different lines in them. Look line 142 in rubygems/installer.rb, it is explicitly looking for /^#!/, not what a windows wrapper uses. > PS: some single quotes will make more easy to spot what is the message > and what represents an executable or a gem. Gem named "a" confuses in > the generated language for non-english native speakers (at least it > does for me) If you have time to tackle the above, feel free to add them, otherwise I can fix it. From evan at phx.io Tue Jan 17 18:10:32 2012 From: evan at phx.io (Evan Phoenix) Date: Tue, 17 Jan 2012 15:10:32 -0800 Subject: Gem file size limits Message-ID: <15BC7AC4FB97479BA529B366D21452F8@phx.io> I believe that rubygems.org needs to limit the max size of a .gem file which will be allowed. This serves two purposes: 1) It protects users from themselves. The top 19 of 20 gems sorted by size are all huge because they accidentally packaged all previous versions within themselves. This issue needs to be fixed on the gem build side also, but there is no reason to allow these gems. 2) Cost. Rubygems.org is becoming increasingly expensive to run and thus we need to begin thinking of ways to keep it mean and lean. I think we can all agree that some kind of limit makes sense. At the moment, there is nothing from preventing a user from using rubygems.org as their personal backup and pushing terabytes in a .gem file. Clearly we can't operate if people do that. So the natural question I have for all of you is: what makes sense as the size limit? To help you with this decision, here is some data for you to chew on: 1) The top 1000 gems, sorted by size: https://gist.github.com/1629309 2) A histogram of gem sizes by megabyte: https://gist.github.com/1629435 You can see from the histogram that 96% of gems are less than one megabyte, and 98% are 3 megs or less. It seems like that fact should inform our decision. To start the decision, let me throw out a starting point: 10 megs. Looking at the biggest non-accidental gems, they're almost all jruby related and contain huge .jar files. We've pinged others about removing the impediment to pushing gems with maven deps and thusly devs would use that functionality rather than packaging the jars within the gems themselves. Comments and Criticisms Required. - Evan -- Evan Phoenix // evan at phx.io From luislavena at gmail.com Tue Jan 17 18:29:10 2012 From: luislavena at gmail.com (Luis Lavena) Date: Tue, 17 Jan 2012 20:29:10 -0300 Subject: A failure that escapes my comprehension In-Reply-To: <8E9ACAA6-BAF3-4F46-83A0-A8E6CE821CE9@segment7.net> References: <8E9ACAA6-BAF3-4F46-83A0-A8E6CE821CE9@segment7.net> Message-ID: On Tue, Jan 17, 2012 at 7:02 PM, Eric Hodel wrote: > > On Jan 11, 2012, at 4:55 PM, Luis Lavena wrote: > >> In master, I've the following failure: >> >> ?12) Failure: >> test_check_executable_overwrite_default_bin_dir(TestGemInstaller) >> [C:/Users/Luis/Projects/oss/rubygems/rubygems/test/rubygems/test_gem_installer.rb:149]: >> --- expected >> +++ actual >> @@ -1 +1 @@ >> -"\"executable\" from a conflicts with >> C:/Users/Luis/Projects/oss/rubygems/rubygems/tmp/test/test_rubygems_3892/tmp/test/test_rubygems_3892/gemhome/bin/executable" >> +"\"executable\" from a conflicts with installed executable from conflict" >> >> The error message comes from Gem::Installer#check_executable_overwrite >> >> What I don't understand is why is triggering a ruby_executable >> (conflict = "installed executable?") > > I think it is because the windows executables have different lines in them. > bin/executable contains the shebang, I added a debug line to check_executable_overwrite which resulted in: {:generated_bin=>"#!C:/Users/Luis/Projects/oss/rubygems/rubygems/tmp/test/test_rubygems_6340/gemhome/bin/ruby.exe\n#\n# This file was generated by RubyGems.\n#\n# The application 'conflict' is installed as part of a gem, and\n# this file is here to facilitate running it.\n#\n\nrequire 'rubygems'\n\nversion = \">= 0\"\n\nif ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then\n version = $1\n ARGV.shift\nend\n\ngem 'conflict', version\nload Gem.bin_path('conflict', 'executable', version)\n"} > Look line 142 in rubygems/installer.rb, it is explicitly looking for /^#!/, not what a windows wrapper uses. > >From above output seems to be that first line does match with the regexp, so why is not valid? Continue to investigate, will report later :-) >> PS: some single quotes will make more easy to spot what is the message >> and what represents an executable or a gem. Gem named "a" confuses in >> the generated language for non-english native speakers (at least it >> does for me) > > > If you have time to tackle the above, feel free to add them, otherwise I can fix it. > Will do, thank you. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From alex at stinky.com Tue Jan 17 18:36:54 2012 From: alex at stinky.com (Alex Chaffee) Date: Tue, 17 Jan 2012 15:36:54 -0800 Subject: Rails Takes Ten Seconds (!) Message-ID: I'm teaching a class right now and I have a room full of students. Half of them are on Macs and the other half are on Windows and used RailsInstaller (latest version, downloaded this morning) to install on fresh systems. Whenever the Windows users type any rails command from the terminal -- including "rails -v" -- there is a 10 second delay before anything starts happening. Same with "gem -v" (though some students say "gem -v" takes 3-4 seconds), though "ruby -v" returns immediately. The Mac users, of couse, are experiencing no such delays; the Windows users are learning that Ruby is slow :-) Is this a known issue? Is there anything we can do about it? It is *seriously* impeding flow. We are using rails 3.1.1 and gem 1.7.2 which are the versions Railsinstaller installed. If we hit control-C during the pause we get a stack trace somewhere inside the gem require library, e.g. custom_require.rb:36 (which calls gem_original_require). Could rubygems be hitting the network or timing out for some reason? -- Alex Chaffee - alex at stinky.com http://alexchaffee.com http://twitter.com/alexch From luislavena at gmail.com Tue Jan 17 18:51:23 2012 From: luislavena at gmail.com (Luis Lavena) Date: Tue, 17 Jan 2012 20:51:23 -0300 Subject: A failure that escapes my comprehension In-Reply-To: References: <8E9ACAA6-BAF3-4F46-83A0-A8E6CE821CE9@segment7.net> Message-ID: On Tue, Jan 17, 2012 at 8:29 PM, Luis Lavena wrote: > >> Look line 142 in rubygems/installer.rb, it is explicitly looking for /^#!/, not what a windows wrapper uses. >> > > From above output seems to be that first line does match with the > regexp, so why is not valid? > > Continue to investigate, will report later :-) > So, regexp matches, things works, but here is the thing: =begin ruby_executable = false existing = nil open generated_bin, 'rb' do |io| next unless io.gets =~ /^#!/ # shebang io.gets # blankline # TODO detect a specially formatted comment instead of trying # to run a regexp against ruby code. next unless io.gets =~ /This file was generated by RubyGems/ ruby_executable = true existing = io.read.slice(/^gem (['"])(.*?)(\1),/, 2) end ... if ruby_executable then question << existing return if ask_yes_no "#{question}\nOverwrite the executable?", false conflict = "installed executable from #{existing}" else =end ruby_executable is set to true, meaning the generated_bin was installed from another gem. Now, the spec is attempting to compare that with an executable that was not installed from another gem, and is failing. util_conflict_executable is being used to generate the 'conflict' gem and its executable. I'm missing something? because the test is not very descriptive. Thank you for your time. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From luislavena at gmail.com Tue Jan 17 19:13:13 2012 From: luislavena at gmail.com (Luis Lavena) Date: Tue, 17 Jan 2012 21:13:13 -0300 Subject: Gem file size limits In-Reply-To: <15BC7AC4FB97479BA529B366D21452F8@phx.io> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: On Tue, Jan 17, 2012 at 8:10 PM, Evan Phoenix wrote: > I believe that rubygems.org needs to limit the max size of a .gem file which will be allowed. > > ... > > You can see from the histogram that 96% of gems are less than one megabyte, and 98% are 3 megs or less. It seems like that fact should inform our decision. > > To start the decision, let me throw out a starting point: 10 megs. > > Looking at the biggest non-accidental gems, they're almost all jruby related and contain huge .jar files. We've pinged others about removing the impediment to pushing gems with maven deps and thusly devs would use that functionality rather than packaging the jars within the gems themselves. > That leaves out gems with native dependencies like qtbindings. QT is massive and to ask users to compile it to be able to install it on Windows is very problematic. qtbindings pre-compiled for Windows weight ~43MB, the limit you comment will block them from publishing gems. There are other gems with pre-compiled bindings like qtbindings or gtk2 (12MB) out. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From nick at quaran.to Tue Jan 17 19:18:59 2012 From: nick at quaran.to (Nick Quaranto) Date: Tue, 17 Jan 2012 19:18:59 -0500 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: I'll be honest, I don't remember why we added this. I think it was basically to ensure that gems that could be pushed could *always* be installed. I'm suprised metadata hasn't been brought up here. It would solve this entire problem and let jruby gems specify whatever they want. On Thu, Jan 12, 2012 at 1:46 PM, Evan Phoenix wrote: > I feel like the crux of the request, removing the dependency name check on > push, has gotten lost in the discussion. > > Given that it currently doesn't provide a true safeguard that the gem's > deps are actually satisfied by gems currently on rubygems.org, I think we > should go ahead and simply remove the check all together. > > If users create gems that have unresolvable deps (something they can > easily do now) thats on them to do. We don't have a reason to police this > aspect of gem availability. > > Any thoughts? > > -- > Evan Phoenix // evan at phx.io > > > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org > http://rubyforge.org/mailman/listinfo/rubygems-developers > From nick at quaran.to Tue Jan 17 19:21:08 2012 From: nick at quaran.to (Nick Quaranto) Date: Tue, 17 Jan 2012 19:21:08 -0500 Subject: Gem file size limits In-Reply-To: <15BC7AC4FB97479BA529B366D21452F8@phx.io> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: For those willing to do more data analysis, here's everything we have on S3 right now with file sizes. https://gist.github.com/1629174 On Tue, Jan 17, 2012 at 6:10 PM, Evan Phoenix wrote: > I believe that rubygems.org needs to limit the max size of a .gem file > which will be allowed. > > This serves two purposes: > 1) It protects users from themselves. The top 19 of 20 gems sorted by size > are all huge because they accidentally packaged all previous versions > within themselves. This issue needs to be fixed on the gem build side also, > but there is no reason to allow these gems. > 2) Cost. Rubygems.org is becoming increasingly expensive to run and thus > we need to begin thinking of ways to keep it mean and lean. > > I think we can all agree that some kind of limit makes sense. At the > moment, there is nothing from preventing a user from using rubygems.orgas their personal backup and pushing terabytes in a .gem file. Clearly we > can't operate if people do that. > > So the natural question I have for all of you is: what makes sense as the > size limit? To help you with this decision, here is some data for you to > chew on: > > 1) The top 1000 gems, sorted by size: https://gist.github.com/1629309 > 2) A histogram of gem sizes by megabyte: https://gist.github.com/1629435 > > You can see from the histogram that 96% of gems are less than one > megabyte, and 98% are 3 megs or less. It seems like that fact should inform > our decision. > > To start the decision, let me throw out a starting point: 10 megs. > > Looking at the biggest non-accidental gems, they're almost all jruby > related and contain huge .jar files. We've pinged others about removing the > impediment to pushing gems with maven deps and thusly devs would use that > functionality rather than packaging the jars within the gems themselves. > > Comments and Criticisms Required. > > - Evan > > -- > Evan Phoenix // evan at phx.io > > > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org > http://rubyforge.org/mailman/listinfo/rubygems-developers > From luislavena at gmail.com Tue Jan 17 19:34:14 2012 From: luislavena at gmail.com (Luis Lavena) Date: Tue, 17 Jan 2012 21:34:14 -0300 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Tue, Jan 17, 2012 at 9:18 PM, Nick Quaranto wrote: > I'll be honest, I don't remember why we added this. I think it was > basically to ensure that gems that could be pushed could *always* be > installed. > > I'm suprised metadata hasn't been brought up here. It would solve this > entire problem and let jruby gems specify whatever they want. > Err, didn't my email get your inbox? I did mention metadata several times... -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From ryand-ruby at zenspider.com Tue Jan 17 20:35:08 2012 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Tue, 17 Jan 2012 17:35:08 -0800 Subject: Rails Takes Ten Seconds (!) In-Reply-To: References: Message-ID: On Jan 17, 2012, at 15:36 , Alex Chaffee wrote: > The Mac users, of couse, are experiencing no such delays; the Windows > users are learning that Ruby is slow :-) > > Is this a known issue? Is there anything we can do about it? It is > *seriously* impeding flow. This is a known issue with ruby itself (specific to file/dir/glob? access on windows) and was fixed (iirc) for 1.9.3. From ryand-ruby at zenspider.com Tue Jan 17 20:38:51 2012 From: ryand-ruby at zenspider.com (Ryan Davis) Date: Tue, 17 Jan 2012 17:38:51 -0800 Subject: Gem file size limits In-Reply-To: References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: <5B535262-E3D7-4B92-B382-7D4D40982B6C@zenspider.com> On Jan 17, 2012, at 16:13 , Luis Lavena wrote: > That leaves out gems with native dependencies like qtbindings. > > QT is massive and to ask users to compile it to be able to install it > on Windows is very problematic. > > qtbindings pre-compiled for Windows weight ~43MB, the limit you > comment will block them from publishing gems. > > There are other gems with pre-compiled bindings like qtbindings or > gtk2 (12MB) out. Maybe we should have a second gem source that doesn't have this restriction but isn't run by rubycentral and is sponsored by someone else (even if just by bandwidth)? After seeing a number of exponentially growing projects I do think _something_ needs to be done. My insistence on using hand checked manifests will only go so far. :P From drbrain at segment7.net Tue Jan 17 20:41:53 2012 From: drbrain at segment7.net (Eric Hodel) Date: Tue, 17 Jan 2012 17:41:53 -0800 Subject: Gem file size limits In-Reply-To: <15BC7AC4FB97479BA529B366D21452F8@phx.io> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: On Jan 17, 2012, at 3:10 PM, Evan Phoenix wrote: > I believe that rubygems.org needs to limit the max size of a .gem file which will be allowed. > > This serves two purposes: > 1) It protects users from themselves. The top 19 of 20 gems sorted by size are all huge because they accidentally packaged all previous versions within themselves. This issue needs to be fixed on the gem build side also, but there is no reason to allow these gems. > 2) Cost. Rubygems.org is becoming increasingly expensive to run and thus we need to begin thinking of ways to keep it mean and lean. > > I think we can all agree that some kind of limit makes sense. At the moment, there is nothing from preventing a user from using rubygems.org as their personal backup and pushing terabytes in a .gem file. Clearly we can't operate if people do that. > > So the natural question I have for all of you is: what makes sense as the size limit? To help you with this decision, here is some data for you to chew on: > > 1) The top 1000 gems, sorted by size: https://gist.github.com/1629309 > 2) A histogram of gem sizes by megabyte: https://gist.github.com/1629435 For additional data, here's the sum of space consumed by each gem for all its releases: http://paste.segment7.net/qh.html > You can see from the histogram that 96% of gems are less than one megabyte, and 98% are 3 megs or less. It seems like that fact should inform our decision. > > To start the decision, let me throw out a starting point: 10 megs. Most of the gems listed in the top 10 contain embedded third-party code, SDKs, etc. At position 726 in the list the total consumption for a gem reaches 10MB, so 98% of authors use less than 10MB total. > Looking at the biggest non-accidental gems, they're almost all jruby related and contain huge .jar files. We've pinged others about removing the impediment to pushing gems with maven deps and thusly devs would use that functionality rather than packaging the jars within the gems themselves. For some perspective on possible limits, here's the list of gems that used more than 5MB for any release: http://paste.segment7.net/qi.html For 34264 releases: A 10MB limit would block 0.7% of gems A 5MB limit would block 1.3% of gems On IRC Gregory Brown suggested we cross-reference this list with the popularity of the gem, but I don't have download counts handy. From the most-downloaded-today list on rubygems.org: mime-types uses 238KB total multi_json uses 134KB total treetop uses 2MB total json uses 32MB total thor uses 20MB total neither thor nor json use > 5MB for any release Some other known-popular gems: rake uses 29MB total rails uses 20MB total activerecord uses 48MB total actionpack uses 68MB total actionmailer uses 9MB total activeresource uses 5MB total activesupport uses 28MB total bundler uses 10MB total There was one anomolous release of action pack, 2.3.6, which was 17MB due to garbage in tmp/test. 2.3.10 and 2.3.11 were 1MB, the rest below. The largest release in the 3 series is 600KB and the 3.2 RC is 374272 bytes. From drbrain at segment7.net Tue Jan 17 20:46:14 2012 From: drbrain at segment7.net (Eric Hodel) Date: Tue, 17 Jan 2012 17:46:14 -0800 Subject: Gem file size limits In-Reply-To: <5B535262-E3D7-4B92-B382-7D4D40982B6C@zenspider.com> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> <5B535262-E3D7-4B92-B382-7D4D40982B6C@zenspider.com> Message-ID: On Jan 17, 2012, at 5:38 PM, Ryan Davis wrote: > On Jan 17, 2012, at 16:13 , Luis Lavena wrote: >> That leaves out gems with native dependencies like qtbindings. >> >> QT is massive and to ask users to compile it to be able to install it >> on Windows is very problematic. >> >> qtbindings pre-compiled for Windows weight ~43MB, the limit you >> comment will block them from publishing gems. >> >> There are other gems with pre-compiled bindings like qtbindings or >> gtk2 (12MB) out. > > Maybe we should have a second gem source that doesn't have this restriction but isn't run by rubycentral and is sponsored by someone else (even if just by bandwidth)? > > After seeing a number of exponentially growing projects I do think _something_ needs to be done. My insistence on using hand checked manifests will only go so far. :P Four of the exponential growth gems have been traced back to jeweler not cleaning the pkg/ dir before building gems and building out of the source directory, not a copy in the pkg/ directory. I filed the following issue: https://github.com/technicalpickles/jeweler/issues/216 actionpack-2.3.6 had a similar issue where test/tmp was not cleaned before packaging resulting in a 17MB gem. I will double check the package code to ensure that only files listed in the spec are added to a packaged gem which should limit this problem, provided the build-tool authors do the right thing. From drbrain at segment7.net Tue Jan 17 20:51:01 2012 From: drbrain at segment7.net (Eric Hodel) Date: Tue, 17 Jan 2012 17:51:01 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> On Jan 12, 2012, at 10:29 AM, Luis Lavena wrote: > On Thu, Jan 12, 2012 at 3:11 PM, Charles Oliver Nutter > wrote: >> We want the external libraries to be fully functional gems. JRuby >> already supports installing them like this (I know colons are bad, >> we'll figure out something else): >> >> gem install mvn:org.clojure:clojure > > So what you're suggesting, even you stated differently is that you > want to pollute rubygems.org with maven artifacts wrapped as gems? The following line is: >> This creates a gem on the fly and installs it, and any scripts, tools, >> etc that use RubyGems see it as just another gem. So I take this to mean that the clojure gem only exists on the user's machine while installed and not on rubygems.org nor anywhere else. From drbrain at segment7.net Tue Jan 17 20:59:06 2012 From: drbrain at segment7.net (Eric Hodel) Date: Tue, 17 Jan 2012 17:59:06 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: On Jan 12, 2012, at 10:46 AM, Evan Phoenix wrote: > I feel like the crux of the request, removing the dependency name check on push, has gotten lost in the discussion. > > Given that it currently doesn't provide a true safeguard that the gem's deps are actually satisfied by gems currently on rubygems.org, I think we should go ahead and simply remove the check all together. > > If users create gems that have unresolvable deps (something they can easily do now) thats on them to do. We don't have a reason to police this aspect of gem availability. > > Any thoughts? The biggest users of S3 space on rubygems.org embed other software within the gem. Many of the embedded packages are jar files. Having an alternate way of downloading the java packages will be an extreme help in reducing the size consumed by this tiny handful of gems. Not every gem packages jar files, but it seems that gem authors need a way to ship third-party code their gems need to be useful for the users. For jruby users the alternative is to embed all the jars needed which increases the costs on rubygems.org. We should find an alternative mechanism for other authors that create large gems. In the mean time, I think we should remove the dependency name check from rubygems.org. We can possibly add it as a gem package time check to maintain most of the benefits. From evan at phx.io Tue Jan 17 21:14:46 2012 From: evan at phx.io (Evan Phoenix) Date: Tue, 17 Jan 2012 18:14:46 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: Message-ID: Based on everyones feedback, I'm going to lift the dependency name check. This will allow jruby specific gems to specify maven dependencies directly. I'll be commit the change to rubygems.org shortly and coordinating with Nick to get it deployed. - Evan -- Evan Phoenix // evan at phx.io From evan at phx.io Tue Jan 17 21:17:54 2012 From: evan at phx.io (Evan Phoenix) Date: Tue, 17 Jan 2012 18:17:54 -0800 Subject: Dealing with JRuby and jars In-Reply-To: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> References: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> Message-ID: See below. -- Evan Phoenix // evan at phx.io On Tuesday, January 17, 2012 at 5:51 PM, Eric Hodel wrote: > On Jan 12, 2012, at 10:29 AM, Luis Lavena wrote: > > On Thu, Jan 12, 2012 at 3:11 PM, Charles Oliver Nutter > > wrote: > > > We want the external libraries to be fully functional gems. JRuby > > > already supports installing them like this (I know colons are bad, > > > we'll figure out something else): > > > > > > gem install mvn:org.clojure:clojure > > > > So what you're suggesting, even you stated differently is that you > > want to pollute rubygems.org (http://rubygems.org) with maven artifacts wrapped as gems? > > > > The following line is: > > > > This creates a gem on the fly and installs it, and any scripts, tools, > > > etc that use RubyGems see it as just another gem. > > > > > > So I take this to mean that the clojure gem only exists on the user's machine while installed and not on rubygems.org (http://rubygems.org) nor anywhere else. Yes, thats correct. It works similar to how bundler synthesizes specs for git repos and raw directories. > > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > http://rubyforge.org/mailman/listinfo/rubygems-developers From luislavena at gmail.com Wed Jan 18 11:07:56 2012 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 18 Jan 2012 13:07:56 -0300 Subject: Rails Takes Ten Seconds (!) In-Reply-To: References: Message-ID: On Tue, Jan 17, 2012 at 10:35 PM, Ryan Davis wrote: > > On Jan 17, 2012, at 15:36 , Alex Chaffee wrote: > >> The Mac users, of couse, are experiencing no such delays; the Windows >> users are learning that Ruby is slow :-) >> >> Is this a known issue? Is there anything we can do about it? It is >> *seriously* impeding flow. > > This is a known issue with ruby itself (specific to file/dir/glob? access on windows) and was fixed (iirc) for 1.9.3. 1.9.3 improve it, yet still is far from fixed. I've linked to the user to my post, in case you want to read about it: http://blog.mmediasys.com/2011/11/26/rubyconf-argentina-and-fenix/ Slides are in spanish but you might get the idea. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From evan at phx.io Wed Jan 18 12:49:24 2012 From: evan at phx.io (Evan Phoenix) Date: Wed, 18 Jan 2012 09:49:24 -0800 Subject: "Federating" rubygems.org Message-ID: While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. What if the following were possible: gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. This allows rubygems.org to be a sort of head-in or primary federation server for gems. 99% of people will just push their gem like normal, but for the people that have very large gems that no longer fit within our size limit, they can simply host the gem file themselves. This provides a best case scenario where users can continue to simply install a gem like normal, but the costs of hosting the large gems is offset by the creator. I was worried about security slightly, since the file at the url could be replaced with different code after the fact. But this is not dissimilar from someone pushing a new version that introduces a security issue. Since most people use ~> or > for their deps, they'd pick up that new version. Thoughts? Comments and Criticisms Required. -- Evan Phoenix // evan at phx.io From drbrain at segment7.net Wed Jan 18 13:16:58 2012 From: drbrain at segment7.net (Eric Hodel) Date: Wed, 18 Jan 2012 10:16:58 -0800 Subject: "Federating" rubygems.org In-Reply-To: References: Message-ID: <8912A440-80CC-409E-AA3D-5F31576014E0@segment7.net> On Jan 18, 2012, at 9:49 AM, Evan Phoenix wrote: > While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. > > What if the following were possible: > > gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem > > This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. > > This allows rubygems.org to be a sort of head-in or primary federation server for gems. 99% of people will just push their gem like normal, but for the people that have very large gems that no longer fit within our size limit, they can simply host the gem file themselves. This provides a best case scenario where users can continue to simply install a gem like normal, but the costs of hosting the large gems is offset by the creator. > > I was worried about security slightly, since the file at the url could be replaced with different code after the fact. But this is not dissimilar from someone pushing a new version that introduces a security issue. Since most people use ~> or > for their deps, they'd pick up that new version. I think a secure hash of the gem should be uploaded along with the spec to ensure the destination file cannot be changed. Offhand, I can't think of anything terrible about this so long as rubygems can verify the foreign file is correct. From jon.forums at gmail.com Wed Jan 18 15:50:39 2012 From: jon.forums at gmail.com (Jon) Date: Wed, 18 Jan 2012 15:50:39 -0500 Subject: "Federating" rubygems.org In-Reply-To: <8912A440-80CC-409E-AA3D-5F31576014E0@segment7.net> References: <8912A440-80CC-409E-AA3D-5F31576014E0@segment7.net> Message-ID: <20120118155039.6c2bbfdd4fa8338c8eaf65bc@gmail.com> > On Jan 18, 2012, at 9:49 AM, Evan Phoenix wrote: > > While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. > > > > What if the following were possible: > > > > gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem > > > > This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. Nice. Would the --custom-url impl do a size check and could it easily be hacked to allow pushing whale gems? Don't what you're thinking re: implementation, but wondering about this hypothetical: 1) create a 1024MB whopper-1.0.0.gem 2) fail on `gem push` due to limits and can't figure out how to locally hack RG to allow push 3) figure how to hack `gem push --custom-url` and push the whopper anyway I'm guessing API key revocation could help in this scenario, but I'm curious on how and where size limits would get enforced. From evan at phx.io Wed Jan 18 16:22:04 2012 From: evan at phx.io (Evan Phoenix) Date: Wed, 18 Jan 2012 13:22:04 -0800 Subject: "Federating" rubygems.org In-Reply-To: <20120118155039.6c2bbfdd4fa8338c8eaf65bc@gmail.com> References: <8912A440-80CC-409E-AA3D-5F31576014E0@segment7.net> <20120118155039.6c2bbfdd4fa8338c8eaf65bc@gmail.com> Message-ID: See below. -- Evan Phoenix // evan at phx.io On Wednesday, January 18, 2012 at 12:50 PM, Jon wrote: > > On Jan 18, 2012, at 9:49 AM, Evan Phoenix wrote: > > > While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. > > > > > > What if the following were possible: > > > > > > gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem > > > > > > This would, rather than uploading the gx gem to rubygems.org (http://rubygems.org), it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. > > Nice. > > Would the --custom-url impl do a size check and could it easily be hacked to allow pushing whale gems? The whole point of custom-url is that the gem is not uploaded to rubygems.org, so the size doesn't matter at all. > > Don't what you're thinking re: implementation, but wondering about this hypothetical: > > 1) create a 1024MB whopper-1.0.0.gem > 2) fail on `gem push` due to limits and can't figure out how to locally hack RG to allow push Again, no check here because custom-url makes rubygems.org not care since the gem isn't uploaded. > 3) figure how to hack `gem push --custom-url` and push the whopper anyway > > I'm guessing API key revocation could help in this scenario, but I'm curious on how and where size limits would get enforced. > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > http://rubyforge.org/mailman/listinfo/rubygems-developers From headius at headius.com Wed Jan 18 17:36:35 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Wed, 18 Jan 2012 16:36:35 -0600 Subject: Gem file size limits In-Reply-To: <15BC7AC4FB97479BA529B366D21452F8@phx.io> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: Checking in on this one... On Tue, Jan 17, 2012 at 5:10 PM, Evan Phoenix wrote: > I believe that rubygems.org needs to limit the max size of a .gem file which will be allowed. I agree, but having a fairly important gem that's over the 10MB limit (jruby-jars, fetched by anyone deploying JRuby to a Java webapp server) I have concerns :) > This serves two purposes: > 1) It protects users from themselves. The top 19 of 20 gems sorted by size are all huge because they accidentally packaged all previous versions within themselves. This issue needs to be fixed on the gem build side also, but there is no reason to allow these gems. > 2) Cost. Rubygems.org is becoming increasingly expensive to run and thus we need to begin thinking of ways to keep it mean and lean. Limiting gem sizes may stall the cost issue, but it's not going to eliminate it. Eventually you'll be right back up there with "well behaved" gems that have released many versions over time. > To start the decision, let me throw out a starting point: 10 megs. > > Looking at the biggest non-accidental gems, they're almost all jruby related and contain huge .jar files. We've pinged others about removing the impediment to pushing gems with maven deps and thusly devs would use that functionality rather than packaging the jars within the gems themselves. What about a Github approach? Everyone can get an account with some amount of free space, and if you go beyond that you have to pay or move stuff off to your own server (I will get to the "federated" thread in a minute). The problem with choosing a limit based on individual gem size is that the worst offenders may be below that limit but push lots of revisions, compared to JRuby which has maybe a dozen jruby-jars versions around 10MB each. Basing the limit on a per-account "free" tier makes more sense to me. It's also possible (if not likely) that people interested in pushing big gems and not hosting them would happily pay for larger tiers of service. That would immediately start blunting costs without any complicated multi-home or federation work. Another mad-cap idea: multihome *using* github. Make it possible for people to host their gems on github and use that as the store for gems associated with their account. Then you don't even have to manage the paid tiers; they just do that themselves. There's many better options than just shutting the door at 10MB. - Charlie From headius at headius.com Wed Jan 18 17:49:32 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Wed, 18 Jan 2012 16:49:32 -0600 Subject: Dealing with JRuby and jars In-Reply-To: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> References: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> Message-ID: Answering below and then summarizing *exactly* how our maven gem stuff works. On Tue, Jan 17, 2012 at 7:51 PM, Eric Hodel wrote: > On Jan 12, 2012, at 10:29 AM, Luis Lavena wrote: >> So what you're suggesting, even you stated differently is that you >> want to pollute rubygems.org with maven artifacts wrapped as gems? ... > So I take this to mean that the clojure gem only exists on the user's machine while installed and not on rubygems.org nor anywhere else. That is correct. Maven artifacts are managed much like RubyGems, with a central server (really many "centrals"). The main namespace is "group", which would be something like org.jruby. Under group there's "artifact" which would be "jruby" for the main JRuby library, "jruby-complete" for the jruby jar that includes stdlib, and so on. All this is managed in an XML-based specification file (pom.xml) which additionally contains other library dependencies, library information (desc, authors, home repo, web page), testing information, build information, and so on. Maven is like RubyGems + Rake (+ 5000 other things) all in one. JRuby's support for "maven gems" basically just uses Maven to fetch the given library's pom.xml, convert that into a gemspec, and then pass that off to RubyGems to install. We hook into spec retrieval and file download in both cases, looking for "mvn" prefix to indicate that we need to look at Maven Central instead of RubyGems.org. The mechanism is fairly elegant, since RubyGems does all the version and dependency walking. At install time, the process goes like this: * Our RubyGems recognizes that this is an mvn: gem and uses our Maven logic to build a gemspec * The gemspec is returned and RubyGems processes it as normal * Dependencies that are also mvn: gems go through the same process * Finally, when it comes time to fetch the actual gem, our RubyGems again recognizes that it's a mvn: gem, fetches the jar file (or whatever) for the Maven artifact, and wraps it up in a rough gem structure. The resulting gem only exists on the user's local system; it is "virtual" in nature. The gem mostly just contains the jar file, but we also throw in a few simple .rb files for easy loading, version numbers, and getting the real path to the jar file on disk. If the dependency restriction is dropped, that's 90% of our issues solved. I fully endorse that change :) The other piece that would be "nice to have" is making it possible for gems to either: a) automatically source pieces of them from another location; e.g. pull the gem from RubyGems.org but the jar from somewhere else OR b) make it easier to hook the spec retrieval + gem fetch process I prefer the latter, since it allows us to build the gems any way we want, so long as we feed RubyGems a valid spec and provide it smarts to get a .gem file in hand. This thread is fine to discuss the additional support, I think. Questions? Comments? - Charlie From luislavena at gmail.com Wed Jan 18 17:50:25 2012 From: luislavena at gmail.com (Luis Lavena) Date: Wed, 18 Jan 2012 19:50:25 -0300 Subject: "Federating" rubygems.org In-Reply-To: References: Message-ID: On Wed, Jan 18, 2012 at 2:49 PM, Evan Phoenix wrote: > While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. > > What if the following were possible: > > ?gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem > > This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. > The only problem I see with that is the usability issue: it will require manual (or automated) upload of the gem to the custom server. > Thoughts? Comments and Criticisms Required. In my experience with S3+CloudFront, costs increases not on storage but traffic. I have gigs of data stored on Amazon and pay pretty much nothing while on CloudFront is another thing. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exup?ry From headius at headius.com Wed Jan 18 18:09:57 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Wed, 18 Jan 2012 17:09:57 -0600 Subject: "Federating" rubygems.org In-Reply-To: References: Message-ID: On Wed, Jan 18, 2012 at 11:49 AM, Evan Phoenix wrote: > While thinking about the gem size and dependency naming issues last night, I had a thought for a feature. > > What if the following were possible: > > ?gem push --custom-url http://phx.io/gems/gx-1.0.0.gem gx-1.0.0.gem > > This would, rather than uploading the gx gem to rubygems.org, it would upload just the spec and set a "url" field that the user has specified. When the gem is requested for download, we'd redirect to that custom url. > > This allows rubygems.org to be a sort of head-in or primary federation server for gems. 99% of people will just push their gem like normal, but for the people that have very large gems that no longer fit within our size limit, they can simply host the gem file themselves. This provides a best case scenario where users can continue to simply install a gem like normal, but the costs of hosting the large gems is offset by the creator. I wonder if Github or someone else with a tiered service would be interested in partnering with RubyGems.org? My concern about an arbitrary federation system is that there's a lot of details involved. We're pushing the storage onto an arbitrary service that may be unreliable, may be insecure, may allow the files to change (more on security below). If we partnered directly with someone like Github or folks that host Maven artifacts, we'd have a directly integrated service that already manages sizes, tiering, authentication, and so on. Being able to push a gem with its guts on an arbitrary server is nice, but I'd prefer to see something that just say "offload my gems to Github" and figures out the rest for you. > I was worried about security slightly, since the file at the url could be replaced with different code after the fact. But this is not dissimilar from someone pushing a new version that introduces a security issue. Since most people use ~> or > for their deps, they'd pick up that new version. Security is definitely a tricky bit. Perhaps what actually gets pushed to the store is not the raw gem but a gem plus a hash. Basically, you've pushed the gem based on a particular file and it must remain that file if you want the gem to keep working. Then the onus is on the user to not replace files they've pushed before. If course if you have a direct tie-in with something like GH, RG can mangle the filename any way it wants, like rake-1.2.3.gem.SOMELONGHASH. If the hash doesn't match the contents, it won't be used, and if you're just using RG to do all the pushes anyway you'll never care. - Charlie From headius at headius.com Wed Jan 18 18:55:55 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Wed, 18 Jan 2012 17:55:55 -0600 Subject: "Federating" rubygems.org In-Reply-To: References: Message-ID: On Wed, Jan 18, 2012 at 5:09 PM, Charles Oliver Nutter wrote: > I wonder if Github or someone else with a tiered service would be > interested in partnering with RubyGems.org? After chatting a bit with Evan on IRC, I'm on board with a simple option. Summarizing my understanding of the idea here: Basic multi-home support: * gem push (or something like that) will fetch your pre-uploaded gem, hash it, and submit gemspec + hash + URL to RG.org * gem install X (where X is a remote-sourced gem) will get gemspec + hash + URL from RG.org and fetch the gem file on its own It does make pushing these gems a two-step process, which could perhaps be mitigated by RG supporting push --upload plugins in the future that can work with your S3, GH, whatever account. But for now, I think people pushing big gems would be more than happy to deal with the two-step process. For JRuby, we'd just add the gem upload to our S3 push, and then the gem push just references that URL. No extra work, really. >> I was worried about security slightly, since the file at the url could be replaced with different code after the fact. But this is not dissimilar from someone pushing a new version that introduces a security issue. Since most people use ~> or > for their deps, they'd pick up that new version. Security is still tricky. Newer RubyGems versions would validate the downloaded gem client-side based on the hash from RG.org, so that's ok. But older RubyGems versions would have to either reject remote-sourced gems entirely (not great) or just accept them as ok. Evan suggested that RG.org could periodically cull remote-sourced gems that no longer match their hashes, to reduce the window of opportunity for an attack. My position: * I don't care of old RG can download, but having RG.org still mirror remote-sourced gems for older clients (for a while) would be a good compromise. * I really like the gem push form. * JRuby is on board to offload our gems and provide info to all JRuby users on how to offload theirs (e.g. Torquebox, Jenkins). - Charlie From drbrain at segment7.net Wed Jan 18 23:42:07 2012 From: drbrain at segment7.net (Eric Hodel) Date: Wed, 18 Jan 2012 20:42:07 -0800 Subject: Dealing with JRuby and jars In-Reply-To: References: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> Message-ID: On Jan 18, 2012, at 2:49 PM, Charles Oliver Nutter wrote: > b) make it easier to hook the spec retrieval + gem fetch process > > I prefer the latter, since it allows us to build the gems any way we > want, so long as we feed RubyGems a valid spec and provide it smarts > to get a .gem file in hand. in master I've unified Gem::Package::TarInput, Gem::Package::TarOutput, Gem::Format and Gem::Builder into Gem::Package which handles all the work of reading and writing .gem files. There's also Gem::Package::Old for the YAML-ish gems. While there's no hook to replace Gem::Package with an alternate implementation, the required API is: #spec= to provide a Gem::Specification for building #build to build a .gem file for the spec #spec to retrieve a Gem::Specification from a .gem file #extract_files(dir) to extract the data.tar.gz Additionally, #verify is called from #spec for Gem::Package and ensures the gem is correctly formatted and complete. From transfire at gmail.com Thu Jan 19 02:48:50 2012 From: transfire at gmail.com (Trans) Date: Wed, 18 Jan 2012 23:48:50 -0800 (PST) Subject: Gem file size limits In-Reply-To: <15BC7AC4FB97479BA529B366D21452F8@phx.io> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> Message-ID: <28347700.873.1326959330815.JavaMail.geo-discussion-forums@yqfd34> Allow developers to mark old versions as "unmaintained" and then delete such versions after a specific period of time without download. If there is a desire to keep them for posterity or perhaps for just-in-case support reasons, these deleted gems could be archived to someone's dedicated local machine for an extended period. From headius at headius.com Thu Jan 19 10:57:39 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Thu, 19 Jan 2012 09:57:39 -0600 Subject: Gem file size limits In-Reply-To: <28347700.873.1326959330815.JavaMail.geo-discussion-forums@yqfd34> References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> <28347700.873.1326959330815.JavaMail.geo-discussion-forums@yqfd34> Message-ID: It would also be good to allow re-pushing an existing (identical) gem with a new remote URL, to allow users to actively offload current giants rather than waiting for new releases. That could immediately reduce the size and bandwidth load for RG.org. For example, JRuby team could re-push all jruby-jars gems using S3 URLs at a moment's notice. - Charlie On Thu, Jan 19, 2012 at 1:48 AM, Trans wrote: > Allow developers to mark old versions as "unmaintained" and then delete > such versions after a specific period of time without download. > > If there is a desire to keep them for posterity or perhaps for just-in-case > support reasons, these deleted gems could be archived to someone's > dedicated local machine for an extended period. > > > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org > http://rubyforge.org/mailman/listinfo/rubygems-developers From headius at headius.com Thu Jan 19 11:03:50 2012 From: headius at headius.com (Charles Oliver Nutter) Date: Thu, 19 Jan 2012 10:03:50 -0600 Subject: Dealing with JRuby and jars In-Reply-To: References: <829F4B16-7219-468E-B78B-42D58F99B8D5@segment7.net> Message-ID: On Wed, Jan 18, 2012 at 10:42 PM, Eric Hodel wrote: > in master I've unified Gem::Package::TarInput, Gem::Package::TarOutput, Gem::Format and Gem::Builder into Gem::Package which handles all the work of reading and writing .gem files. ?There's also Gem::Package::Old for the YAML-ish gems. > > While there's no hook to replace Gem::Package with an alternate implementation, the required API is: > > #spec= to provide a Gem::Specification for building > #build to build a .gem file for the spec > > #spec to retrieve a Gem::Specification from a .gem file > #extract_files(dir) to extract the data.tar.gz > > Additionally, #verify is called from #spec for Gem::Package and ensures the gem is correctly formatted and complete. I'll try to have a look soon. Is fetching part of this at all? What you describe above would be useful for us packaging the maven gems on the fly, but that's mostly a solved problem right now (we just do it manually in a tmp dir). The real trouble is hooking how RG retrieves a spec for a given gem name and how it later fetches the .gem file. We have to patch both paths. Compare this with your 1.8.13 tag to see our changes: https://github.com/jruby/rubygems/tree/jruby-rubygems_1_8_13 - Charlie From dan at kallistec.com Thu Jan 19 12:04:15 2012 From: dan at kallistec.com (Daniel DeLeo) Date: Thu, 19 Jan 2012 09:04:15 -0800 Subject: Gem file size limits In-Reply-To: References: <15BC7AC4FB97479BA529B366D21452F8@phx.io> <28347700.873.1326959330815.JavaMail.geo-discussion-forums@yqfd34> Message-ID: <4B878D19C9874673B8FA5F5646DAB3CB@kallistec.com> The chef-solr gem, which embeds Solr, is also one of the offenders. We wouldn't have a problem hosting the gem on S3 ourselves. The only issue I foresee is that many of our users are new to Ruby and its ecosystem, so I hope that the transition can be made as painless as possible for end users. Thanks, -- Dan DeLeo On Thursday, January 19, 2012 at 7:57 AM, Charles Oliver Nutter wrote: > It would also be good to allow re-pushing an existing (identical) gem > with a new remote URL, to allow users to actively offload current > giants rather than waiting for new releases. That could immediately > reduce the size and bandwidth load for RG.org (http://RG.org). > > For example, JRuby team could re-push all jruby-jars gems using S3 > URLs at a moment's notice. > > - Charlie > > On Thu, Jan 19, 2012 at 1:48 AM, Trans wrote: > > Allow developers to mark old versions as "unmaintained" and then delete > > such versions after a specific period of time without download. > > > > If there is a desire to keep them for posterity or perhaps for just-in-case > > support reasons, these deleted gems could be archived to someone's > > dedicated local machine for an extended period. > > > > > > _______________________________________________ > > RubyGems-Developers mailing list > > http://rubyforge.org/projects/rubygems > > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > > http://rubyforge.org/mailman/listinfo/rubygems-developers > > > _______________________________________________ > RubyGems-Developers mailing list > http://rubyforge.org/projects/rubygems > RubyGems-Developers at rubyforge.org (mailto:RubyGems-Developers at rubyforge.org) > http://rubyforge.org/mailman/listinfo/rubygems-developers