From emarthinsen at novemberconsulting.com  Tue May  1 20:44:17 2007
From: emarthinsen at novemberconsulting.com (Eric Marthinsen)
Date: Tue, 1 May 2007 20:44:17 -0400
Subject: [s3-dev] Error Deleting Bucket
Message-ID: <20070502004416.1A6281CC3DE@eq2.spamarrest.com>

Hello-

 

I'm trying to delete a bucket and I keep getting this error:

 

c:/ruby/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/error.rb:38:in
`raise': We encountered an internal error. Please try again.
(AWS::S3::InternalError)

        from
c:/ruby/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb:72:in
`request'

        from
c:/ruby/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb:84:in `get'

        from
c:/ruby/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/bucket.rb:102:in
`find'

        from
c:/ruby/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/bucket.rb:161:in
`delete'

        from aws_test.rb:42

 

My code at line 42 is pretty simple:

 

AWS::S3::Bucket.delete(

            bucket_name,

            :force => true

)

 

I've been running the script for a while now with the same error. Right
before I attempt the delete, I create the bucket and load a file into it.

 

I'm running this from a Windows machine. Has anyone seen this before?

 

-Eric

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070501/a3dbd6ab/attachment.html 

From marcel at vernix.org  Fri May  4 22:21:04 2007
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Sat, 5 May 2007 02:21:04 +0000
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
In-Reply-To: <20070405152027.GV83368@comox.textdrive.com>
References: <1175770726.v2.fusewebmail-9566@f>
	<20070405152027.GV83368@comox.textdrive.com>
Message-ID: <20070505022104.GG83368@comox.textdrive.com>

On Thu, Apr 05, 2007 at 03:20:27PM +0000, Marcel Molina Jr. wrote:
> On Thu, Apr 05, 2007 at 05:58:46AM -0500, Simon Horne wrote:
> > it does not appear to be possible to access S3 with aws-s3 via a proxy.
> > I've had a look in connection.rb and there is a direct call to
> > Net::HTTP.new().
> > It looks like it would be simple enough to use Net::HTTP::Proxy().new()
> > and to use options to specify the proxy
> > settings.
> > If no :proxy_host is specified then it will safely default to no proxy.
> > 
> > Would this be feasible?
> > 
> > See below for a patch (from 0.3.0 release) -
> > 
> > 82c82
> > <           @http             = Net::HTTP.new(options[:server],
> > options[:port])
> > ---
> > >           @http             = Net::HTTP::Proxy(options[:proxy_addr],
> > options[:proxy_port], options[:proxy_user],
> > options[:proxy_pass]).new(options[:server], options[:port])
> > 211c211
> > <             [:access_key_id, :secret_access_key, :server, :port,
> > :use_ssl, :persistent]
> > ---
> > >             [:access_key_id, :secret_access_key, :server, :port,
> > :use_ssl, :persistent, :proxy_addr, :proxy_port,
> > :proxy_user, :proxy_pass]
> 
> Hey Simon, that looks pretty reasonable. I'll shoe horn it in. Thanks.

Hey Simon, et al. I've gotten around to adding an initial version of proxy
support.

Beta gem is available here:
http://amazon.rubyforge.org/beta/aws-s3-0.3.1178330006.gem

I've attached a diff for those interested in the details. 

I've implemented tests for the wiring, but have not set up any remote tests
for it, so I haven't verified it actually hitting the s3 servers. 

For those who have a need for this, please give it a whirl and let me know if
you have any problems with it. 

Documentation is included for those wondering how to actually use the new
:proxy option :)

marcel
-- 
Marcel Molina Jr. <marcel at vernix.org>
-------------- next part --------------
Index: test/test_helper.rb
===================================================================
--- test/test_helper.rb	(revision 163)
+++ test/test_helper.rb	(revision 164)
@@ -79,4 +79,8 @@
 
 class Test::Unit::TestCase
   include AWS::S3
+  
+  def sample_proxy_settings
+    {:host => 'http://google.com', :port => 8080, :user => 'marcel', :password => 'secret'}
+  end
 end
\ No newline at end of file
Index: test/connection_test.rb
===================================================================
--- test/connection_test.rb	(revision 163)
+++ test/connection_test.rb	(revision 164)
@@ -95,6 +95,14 @@
     assert authenticated[connection.url_for('/foo', :authenticated => true)]
     assert !authenticated[connection.url_for('/foo', :authenticated => false)]
   end
+  
+  def test_connecting_through_a_proxy
+    connection = nil
+    assert_nothing_raised do
+      connection = Connection.new(@keys.merge(:proxy => sample_proxy_settings))
+    end
+    assert connection.http.proxy?
+  end
 end
 
 class ConnectionOptionsTest < Test::Unit::TestCase
@@ -136,8 +144,35 @@
     end
   end
   
+  def test_not_specifying_all_required_proxy_settings_raises
+    assert_raises(ArgumentError) do
+      generate_options(:proxy => {})
+    end
+  end
+  
+  def test_not_specifying_proxy_option_at_all_does_not_raise
+    assert_nothing_raised do
+      generate_options
+    end
+  end
+  
+  def test_specifying_all_required_proxy_settings
+    assert_nothing_raised do
+      generate_options(:proxy => sample_proxy_settings)
+    end
+  end
+  
+  def test_proxy_settings_are_extracted
+    options = generate_options(:proxy => sample_proxy_settings)
+    assert_equal sample_proxy_settings.values.map {|value| value.to_s}.sort, options.proxy_settings.map {|value| value.to_s}.sort
+  end
+  
+  def test_recognizing_that_the_settings_want_to_connect_through_a_proxy
+    options = generate_options(:proxy => sample_proxy_settings)
+    assert options.connecting_through_proxy?
+  end
+  
   private
-  
     def assert_key_transfered(key, value, options)
       assert_equal value, options[key]
       assert !options.instance_variable_get('@options').has_key?(key)
Index: lib/aws/s3/connection.rb
===================================================================
--- lib/aws/s3/connection.rb	(revision 163)
+++ lib/aws/s3/connection.rb	(revision 164)
@@ -88,12 +88,20 @@
         end
         
         def create_connection
-          http             = Net::HTTP.new(options[:server], options[:port])
+          http             = http_class.new(options[:server], options[:port])
           http.use_ssl     = !options[:use_ssl].nil? || options[:port] == 443
           http.verify_mode = OpenSSL::SSL::VERIFY_NONE
           http
         end
         
+        def http_class
+          if options.connecting_through_proxy?
+            Net::HTTP::Proxy(*options.proxy_settings)
+          else
+            Net::HTTP
+          end
+        end
+        
         def connect
           extract_keys!
           @http = create_connection
@@ -171,6 +179,13 @@
           # * <tt>:persistent</tt> - Whether to use a persistent connection to the server. Having this on provides around a two fold 
           # performance increase but for long running processes some firewalls may find the long lived connection suspicious and close the connection.
           # If you run into connection errors, try setting <tt>:persistent</tt> to false. Defaults to true.
+          # * <tt>:proxy</tt> - If you need to connect through a proxy, you can specify your proxy settings by specifying a <tt>:host</tt>, <tt>:port</tt>, <tt>:user</tt>, and <tt>:password</tt>
+          # with the <tt>:proxy</tt> option.
+          # All settings are required. 
+          #   
+          #   AWS::S3::Bucket.established_connection!(:proxy => {
+          #     :host => '...', :port => 8080, :user => 'marcel', :password => 'secret'
+          #   })
           def establish_connection!(options = {})
             # After you've already established the default connection, just specify 
             # the difference for subsequent connections
@@ -227,7 +242,7 @@
       class Options < Hash #:nodoc:
         class << self
           def valid_options
-            [:access_key_id, :secret_access_key, :server, :port, :use_ssl, :persistent]
+            [:access_key_id, :secret_access_key, :server, :port, :use_ssl, :persistent, :proxy]
           end
         end
         
@@ -236,17 +251,45 @@
           super()
           @options = options
           validate!
+          extract_proxy_settings!
           extract_persistent!
           extract_server!
           extract_port!
           extract_remainder!
         end
         
+        def connecting_through_proxy?
+          !self[:proxy].nil?
+        end
+        
+        def proxy_settings
+          proxy_setting_keys.map do |proxy_key| 
+            self[:proxy][proxy_key]
+          end
+        end
+        
         private
+          def proxy_setting_keys
+            [:host, :port, :user, :password]
+          end
+          
+          def missing_proxy_settings?
+            !missing_proxy_settings.empty?
+          end
+          
+          def missing_proxy_settings
+            proxy_setting_keys - self[:proxy].keys
+          end
+          
           def extract_persistent!
             self[:persistent] = options.has_key?(:persitent) ? options[:persitent] : true
           end
           
+          def extract_proxy_settings!
+            self[:proxy] = options.delete(:proxy) if options.include?(:proxy)
+            validate_proxy_settings!
+          end
+          
           def extract_server!
             self[:server] = options.delete(:server) || DEFAULT_HOST
           end
@@ -263,6 +306,12 @@
             invalid_options = options.keys.select {|key| !self.class.valid_options.include?(key)}
             raise InvalidConnectionOption.new(invalid_options) unless invalid_options.empty?
           end
+          
+          def validate_proxy_settings!
+            if connecting_through_proxy? && missing_proxy_settings?
+              raise ArgumentError, "Missing proxy settings: #{(proxy_setting_keys - self[:proxy].keys).join(', ')}"
+            end
+          end
       end
     end
   end
Index: CHANGELOG
===================================================================
--- CHANGELOG	(revision 163)
+++ CHANGELOG	(revision 164)
@@ -1,5 +1,7 @@
 trunk:
 
+- Add ability to connect through a proxy using the :proxy option when establishing a connection. Suggested by [Simon Horne <simon at soulware.co.uk>]
+
 - Add :authenticated option to url_for. When passing false, don't generate signature parameters for query string.
 
 - Make url_for accept custom port settings. [Rich Olson] 

From simon at soulware.co.uk  Sun May  6 11:05:19 2007
From: simon at soulware.co.uk (Simon Horne)
Date: Sun, 6 May 2007 16:05:19 +0100
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
In-Reply-To: <20070505022104.GG83368@comox.textdrive.com>
References: <1175770726.v2.fusewebmail-9566@f>
	<20070405152027.GV83368@comox.textdrive.com>
	<20070505022104.GG83368@comox.textdrive.com>
Message-ID: <D2C9A166-CC48-4967-90F9-2CE953A5D886@soulware.co.uk>

Hi,

cool -
just tried this out and it looks like it works fine.
One suggestion - could the proxy user and password be made optional,  
just for convenience if not needed?

Thanks,

Simon


On 5 May 2007, at 03:21, Marcel Molina Jr. wrote:

> On Thu, Apr 05, 2007 at 03:20:27PM +0000, Marcel Molina Jr. wrote:
>> On Thu, Apr 05, 2007 at 05:58:46AM -0500, Simon Horne wrote:
>>> it does not appear to be possible to access S3 with aws-s3 via a  
>>> proxy.
>>> I've had a look in connection.rb and there is a direct call to
>>> Net::HTTP.new().
>>> It looks like it would be simple enough to use Net::HTTP::Proxy 
>>> ().new()
>>> and to use options to specify the proxy
>>> settings.
>>> If no :proxy_host is specified then it will safely default to no  
>>> proxy.
>>>
>>> Would this be feasible?
>>>
>>> See below for a patch (from 0.3.0 release) -
>>>
>>> 82c82
>>> <           @http             = Net::HTTP.new(options[:server],
>>> options[:port])
>>> ---
>>>>           @http             = Net::HTTP::Proxy(options 
>>>> [:proxy_addr],
>>> options[:proxy_port], options[:proxy_user],
>>> options[:proxy_pass]).new(options[:server], options[:port])
>>> 211c211
>>> <             [:access_key_id, :secret_access_key, :server, :port,
>>> :use_ssl, :persistent]
>>> ---
>>>>             [:access_key_id, :secret_access_key, :server, :port,
>>> :use_ssl, :persistent, :proxy_addr, :proxy_port,
>>> :proxy_user, :proxy_pass]
>>
>> Hey Simon, that looks pretty reasonable. I'll shoe horn it in.  
>> Thanks.
>
> Hey Simon, et al. I've gotten around to adding an initial version  
> of proxy
> support.
>
> Beta gem is available here:
> http://amazon.rubyforge.org/beta/aws-s3-0.3.1178330006.gem
>
> I've attached a diff for those interested in the details.
>
> I've implemented tests for the wiring, but have not set up any  
> remote tests
> for it, so I haven't verified it actually hitting the s3 servers.
>
> For those who have a need for this, please give it a whirl and let  
> me know if
> you have any problems with it.
>
> Documentation is included for those wondering how to actually use  
> the new
> :proxy option :)
>
> marcel
> -- 
> Marcel Molina Jr. <marcel at vernix.org>
> <connection-through-proxy-for-aws-s3.diff>
> _______________________________________________
> amazon-s3-dev mailing list
> amazon-s3-dev at rubyforge.org
> http://rubyforge.org/mailman/listinfo/amazon-s3-dev


From marcel at vernix.org  Sun May  6 11:34:48 2007
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Sun, 6 May 2007 15:34:48 +0000
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
In-Reply-To: <D2C9A166-CC48-4967-90F9-2CE953A5D886@soulware.co.uk>
References: <1175770726.v2.fusewebmail-9566@f>
	<20070405152027.GV83368@comox.textdrive.com>
	<20070505022104.GG83368@comox.textdrive.com>
	<D2C9A166-CC48-4967-90F9-2CE953A5D886@soulware.co.uk>
Message-ID: <20070506153448.GQ83368@comox.textdrive.com>

On Sun, May 06, 2007 at 04:05:19PM +0100, Simon Horne wrote:
> cool -
> just tried this out and it looks like it works fine.
> One suggestion - could the proxy user and password be made optional,  
> just for convenience if not needed?

I've made it so that only the :host option is required. Try it out:
http://amazon.rubyforge.org/beta/aws-s3-0.3.1178465623.gem

marcel
-- 
Marcel Molina Jr. <marcel at vernix.org>

From simon at soulware.co.uk  Sun May  6 12:14:27 2007
From: simon at soulware.co.uk (Simon Horne)
Date: Sun, 6 May 2007 17:14:27 +0100
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
In-Reply-To: <20070506153448.GQ83368@comox.textdrive.com>
References: <1175770726.v2.fusewebmail-9566@f>
	<20070405152027.GV83368@comox.textdrive.com>
	<20070505022104.GG83368@comox.textdrive.com>
	<D2C9A166-CC48-4967-90F9-2CE953A5D886@soulware.co.uk>
	<20070506153448.GQ83368@comox.textdrive.com>
Message-ID: <4A308932-7050-43B5-BF5B-D11E6010C68B@soulware.co.uk>

looks good.

Simon

On 6 May 2007, at 16:34, Marcel Molina Jr. wrote:

> On Sun, May 06, 2007 at 04:05:19PM +0100, Simon Horne wrote:
>> cool -
>> just tried this out and it looks like it works fine.
>> One suggestion - could the proxy user and password be made optional,
>> just for convenience if not needed?
>
> I've made it so that only the :host option is required. Try it out:
> http://amazon.rubyforge.org/beta/aws-s3-0.3.1178465623.gem
>
> marcel
> -- 
> Marcel Molina Jr. <marcel at vernix.org>
> _______________________________________________
> amazon-s3-dev mailing list
> amazon-s3-dev at rubyforge.org
> http://rubyforge.org/mailman/listinfo/amazon-s3-dev


From simon at soulware.co.uk  Thu May 10 18:34:03 2007
From: simon at soulware.co.uk (Simon Horne)
Date: Thu, 10 May 2007 23:34:03 +0100
Subject: [s3-dev] listing keys through prefix/delimiter
Message-ID: <25E70CD0-E1BD-4131-B40B-CB7983319FAF@soulware.co.uk>

Hi Marcel,

have you considered exposing the prefix/delimiter mechanism for  
listing keys?
It might be useful to provide this functionality to query and  
traverse the keys in a bucket.

S3Object might not be suitable for this so maybe there is a need for  
S3Key or something similar - with a way of accessing the sub-keys  
beneath it.

For the 2 keys -

/path/abc -> some_data
/path/def -> more_data

with a delimiter of "/", the subkeys of "/path" would be "/path/abc"  
and "/path/def".
"/path" itself may contain no data but it would be useful to be able  
to traverse through it to "/path/abc" which does contain data (and  
could be represented by an S3Object instance).


Thanks,

Simon