From ivan at ivaner.com Tue Jul 1 15:43:50 2008 From: ivan at ivaner.com (Ivan) Date: Tue, 1 Jul 2008 15:43:50 -0400 Subject: [s3-dev] ArgumentError using AWS-S3 Message-ID: I'm having trouble getting my rails app to use AWS/S3. I believe that I'm pretty close, but can't seem to get past finding a bucket. I've enclosed the function I'm trying to use, and the resulting output error... I have this running on a shared host, and I have the following declared in the environment.rb: require 'rubygems' require 'aws/s3' Thanks in advance for any help provided, Ivan def upload_AWS # Connect to S3 logger.info "*** Connecting to S3 ***" conn = AWS::S3::Base.establish_connection!(:access_key_id => 'XXmyaccesskeyidXX', :secret_access_key => 'XXmysecretaccesskeyXX', :persistent => true) unless conn logger.info "*** NO CONNECTION :***" else logger.info " *** CONNECTION! ***" end # find bucket logger.info "*** Finding out bucket ***" bucket = AWS::S3::Bucket.find('XXmybucketnameXX') unless bucket logger.info "*** Couldn't find bucket ***" end # upload file logger.info "*** Uploading our file ***" logger.info "*** The file name is: " + filename if AWS::S3::S3Object.store(filename, open(filename), 'XXmybucketnameXX') logger.info "*** File upload successful ***" end end OUTPUT: *** Connecting to S3 *** *** CONNECTION! *** *** Finding out bucket *** [4;35;1mSQL (0.012269) [0m [0mROLLBACK [0m ArgumentError (wrong number of arguments (1 for 0)): /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/extensions.rb:109:in `__method__' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/extensions.rb:109:in `memoize' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/extensions.rb:144:in `canonical_string' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/authentication.rb:72:in `encoded_canonical' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/authentication.rb:94:in `initialize' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/connection.rb:130:in `new' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/connection.rb:130:in `authenticate!' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/connection.rb:34:in `request' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/connection.rb:50:in `call' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/connection.rb:50:in `request' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/base.rb:69:in `request' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/base.rb:83:in `get' /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/bucket.rb:102:in `find' /app/models/song.rb:129:in `upload_AWS' /app/models/song.rb:98:in `process' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:307:in `send' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:307:in `callback' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:304:in `each' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:304:in `callback' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:214:in `create_or_update' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:1972:in `save_without_validation' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/validations.rb:934:in `save_without_transactions' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:80:in `transaction' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:100:in `transaction' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:120:in `rollback_active_record_state!' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:572:in `create' /app/controllers/songs_controller.rb:9:in `create' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `send' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `perform_action_without_filters' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:697:in `call_filters' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:689:in `perform_action_without_benchmark' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue' /usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/rescue.rb:199:in `perform_action_without_caching' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:678:in `perform_action' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache' /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/query_cache.rb:8:in `cache' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:677:in `perform_action' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `send' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `process_without_filters' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:685:in `process_without_session_management_support' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/session_management.rb:123:in `process' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:388:in `process' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:171:in `handle_request' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:115:in `dispatch' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:126:in `dispatch_cgi' /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:9:in `dispatch' dispatch.cgi:10 -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcel at vernix.org Tue Jul 1 16:33:28 2008 From: marcel at vernix.org (Marcel Molina Jr.) Date: Tue, 1 Jul 2008 15:33:28 -0500 Subject: [s3-dev] ArgumentError using AWS-S3 In-Reply-To: References: Message-ID: <8C5776C2-2A0A-4F3F-A585-590FBB63CC6E@vernix.org> Sorry you're having this error. Kernel#__method__ didn't exist in Ruby back when I added it to the S3 library. There is now a native implementation in both 1.9 and one backported to 1.8.7. I only define my version if the version of Ruby is less then 1.8.7. The problem is though that my version takes two arguments and the native version takes none. I pass 1 argument almost everywhere in the library that I use it. The right way for the library to be compatible with version of Ruby that defined Kernel#__method__ and those who don't is to have add something like Kernel#name_of_current_method which checks the version number and either calls __method__ with the appropriate arguments or simply delegates to the native implementation. I'm don't have the cycles just this moment to work on this. If anyone is willing to submitted a tested patch I'd gladly apply it quickly if it's a good fit. On Jul 1, 2008, at 2:43 PM, Ivan wrote: > I'm having trouble getting my rails app to use AWS/S3. > I believe that I'm pretty close, but can't seem to get past finding > a bucket. > I've enclosed the function I'm trying to use, and the resulting > output error... > > I have this running on a shared host, and I have the following > declared in the environment.rb: > require 'rubygems' > require 'aws/s3' > > Thanks in advance for any help provided, > Ivan > > > def upload_AWS > > # Connect to S3 > logger.info "*** Connecting to S3 ***" > conn = AWS::S3::Base.establish_connection!(:access_key_id => > 'XXmyaccesskeyidXX', :secret_access_key => 'XXmysecretaccesskeyXX', > :persistent => true) > > unless conn > logger.info "*** NO CONNECTION :***" > else > logger.info " *** CONNECTION! ***" > end > > # find bucket > logger.info "*** Finding out bucket ***" > bucket = AWS::S3::Bucket.find('XXmybucketnameXX') > unless bucket > logger.info "*** Couldn't find bucket ***" > end > > # upload file > logger.info "*** Uploading our file ***" > logger.info "*** The file name is: " + filename > if AWS::S3::S3Object.store(filename, open(filename), > 'XXmybucketnameXX') > logger.info "*** File upload successful ***" > end > end > > > OUTPUT: > > *** Connecting to S3 *** > *** CONNECTION! *** > *** Finding out bucket *** > [4;35;1mSQL (0.012269) [0m [0mROLLBACK [0m > > > ArgumentError (wrong number of arguments (1 for 0)): > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > extensions.rb:109:in > `__method__' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > extensions.rb:109:in > `memoize' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > extensions.rb:144:in > `canonical_string' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > authentication.rb:72:in > `encoded_canonical' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > authentication.rb:94:in > `initialize' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > connection.rb:130:in > `new' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > connection.rb:130:in > `authenticate!' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > connection.rb:34:in > `request' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > connection.rb:50:in > `call' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > connection.rb:50:in > `request' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/base.rb: > 69:in > `request' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/base.rb: > 83:in > `get' > /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.0/lib/aws/s3/ > bucket.rb:102:in > `find' > /app/models/song.rb:129:in `upload_AWS' > /app/models/song.rb:98:in `process' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/callbacks.rb:307:in > `send' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/callbacks.rb:307:in > `callback' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/callbacks.rb:304:in > `each' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/callbacks.rb:304:in > `callback' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/callbacks.rb:214:in > `create_or_update' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/base.rb:1972:in > `save_without_validation' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/validations.rb:934:in > `save_without_transactions' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:108:in > `save' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/connection_adapters/abstract/database_statements.rb: > 66:in > `transaction' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:80:in > `transaction' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:100:in > `transaction' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:108:in > `save' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:120:in > `rollback_active_record_state!' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/transactions.rb:108:in > `save' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/base.rb:572:in > `create' > /app/controllers/songs_controller.rb:9:in `create' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/base.rb:1158:in > `send' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/base.rb:1158:in > `perform_action_without_filters' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/filters.rb:697:in > `call_filters' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/filters.rb:689:in > `perform_action_without_benchmark' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/benchmarking.rb:68:in > `perform_action_without_rescue' > /usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/benchmarking.rb:68:in > `perform_action_without_rescue' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/rescue.rb:199:in > `perform_action_without_caching' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/caching.rb:678:in > `perform_action' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/connection_adapters/abstract/query_cache.rb:33:in > `cache' > /usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/ > active_record/query_cache.rb:8:in > `cache' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/caching.rb:677:in > `perform_action' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/base.rb:524:in > `send' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/base.rb:524:in > `process_without_filters' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/filters.rb:685:in > `process_without_session_management_support' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/session_management.rb:123:in > `process' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/base.rb:388:in > `process' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/dispatcher.rb:171:in > `handle_request' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/dispatcher.rb:115:in > `dispatch' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/dispatcher.rb:126:in > `dispatch_cgi' > /usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/ > action_controller/dispatcher.rb:9:in > `dispatch' > dispatch.cgi:10 > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev marcel -- Marcel Molina Jr. marcel at vernix.org From jerod.santo at gmail.com Wed Jul 2 11:57:34 2008 From: jerod.santo at gmail.com (Jerod Santo) Date: Wed, 2 Jul 2008 10:57:34 -0500 Subject: [s3-dev] ArgumentError using AWS-S3 Message-ID: Wouldn't the fastest/simplest fix for this be to simply remove the version check on the __method__ define so yours is used everytime and the native implementation is ignored? That would get those of us running >= 1.8.7 back up and running until a more elegant solution is put in place. -- Jerod -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcel at vernix.org Thu Jul 3 14:38:40 2008 From: marcel at vernix.org (Marcel Molina Jr.) Date: Thu, 3 Jul 2008 13:38:40 -0500 Subject: [s3-dev] ArgumentError using AWS-S3 In-Reply-To: References: Message-ID: <24D2EA2E-C779-40E2-BAA0-831AB2060CB2@vernix.org> On Jul 2, 2008, at 10:57 AM, Jerod Santo wrote: > Wouldn't the fastest/simplest fix for this be to simply remove the > version check on the __method__ define so yours is used everytime > and the native implementation is ignored? > > That would get those of us running >= 1.8.7 back up and running > until a more elegant solution is put in place. Agreed. Good call. I've made this change and pushed out a bugfix release version 0.5.1. Once the Rubyforge gem mirrors have been replicated to in the next few hours it should be available for download via the gem command. You can of course always grab a gem or tgz straight from Rubyforge. marcel -- Marcel Molina Jr. marcel at vernix.org From ivan at ivaner.com Thu Jul 3 14:48:06 2008 From: ivan at ivaner.com (Ivan) Date: Thu, 3 Jul 2008 14:48:06 -0400 Subject: [s3-dev] ArgumentError using AWS-S3 In-Reply-To: <24D2EA2E-C779-40E2-BAA0-831AB2060CB2@vernix.org> References: <24D2EA2E-C779-40E2-BAA0-831AB2060CB2@vernix.org> Message-ID: Thank you Marcel! I look forward to using this new version. Ivan. On 7/3/08, Marcel Molina Jr. wrote: > > On Jul 2, 2008, at 10:57 AM, Jerod Santo wrote: > > Wouldn't the fastest/simplest fix for this be to simply remove the version >> check on the __method__ define so yours is used everytime and the native >> implementation is ignored? >> >> That would get those of us running >= 1.8.7 back up and running until a >> more elegant solution is put in place. >> > > Agreed. Good call. I've made this change and pushed out a bugfix release > version 0.5.1. Once the Rubyforge gem mirrors have been replicated to in > the next few hours it should be available for download via the gem command. > > You can of course always grab a gem or tgz straight from Rubyforge. > > marcel > -- > Marcel Molina Jr. > marcel at vernix.org > > > > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunkybill at gmail.com Mon Jul 7 18:07:05 2008 From: hunkybill at gmail.com (Dave Lazar) Date: Mon, 7 Jul 2008 18:07:05 -0400 Subject: [s3-dev] Bucket to Bucket copy Message-ID: Hi, I see the Amazon docs show bucket to bucket copy of files... but the latest implementation of copy for the library simply copies a file within the same bucket. Are there any short term plans to implement a full bucket to bucket copy for this library? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcel at vernix.org Mon Jul 7 18:10:17 2008 From: marcel at vernix.org (Marcel Molina Jr.) Date: Mon, 7 Jul 2008 17:10:17 -0500 Subject: [s3-dev] Bucket to Bucket copy In-Reply-To: References: Message-ID: On Jul 7, 2008, at 5:07 PM, Dave Lazar wrote: > I see the Amazon docs show bucket to bucket copy of files... but the > latest implementation of copy for the library simply copies a file > within the same bucket. Are there any short term plans to implement > a full bucket to bucket copy for this library? > > Thanks! Hey Dave. I didn't know about this feature of the new copy API until you brought it up so I wasn't personally planning on adding that but as always patches are welcome if anyone is keen to give it a crack. marcel -- Marcel Molina Jr. marcel at vernix.org From james at floppy.org.uk Tue Jul 8 03:08:42 2008 From: james at floppy.org.uk (James Smith) Date: Tue, 8 Jul 2008 08:08:42 +0100 Subject: [s3-dev] Bucket to Bucket copy In-Reply-To: References: Message-ID: <6e9ea1180807080008g5b0a25f2rb6c18e950a627ee9@mail.gmail.com> Hi all, I actually wrote a bucket-to-bucket copy function a couple of weeks ago, been meaning to send it in for ages: module AWS module S3 class S3Object def self.remote_copy(src_object, src_bucket, tgt_object = src_object, tgt_bucket = src_bucket) # Work out paths target_path = S3Object.path!(tgt_bucket, tgt_object, nil) source_path = S3Object.path!(src_bucket, src_object, nil) # Set copy header options = {:x_amz_copy_source => source_path} # Send PUT to do the copy put(target_path, options, nil) # Copy ACL from source object copied_object = S3Object.find(tgt_object, tgt_bucket) copied_object.acl(S3Object.acl(src_object, src_bucket)) end end end end Hope this helps! cheers, James Smith http://www.floppy.org.uk From james at floppy.org.uk Tue Jul 8 04:03:28 2008 From: james at floppy.org.uk (James Smith) Date: Tue, 8 Jul 2008 09:03:28 +0100 Subject: [s3-dev] Bucket to Bucket copy In-Reply-To: <81D5C332-FF8A-4126-AB59-B3E18A6A7548@gmail.com> References: <6e9ea1180807080008g5b0a25f2rb6c18e950a627ee9@mail.gmail.com> <81D5C332-FF8A-4126-AB59-B3E18A6A7548@gmail.com> Message-ID: <6e9ea1180807080103r1d44b665y406123e1242f4311@mail.gmail.com> Morning, 2008/7/8 Francesc Esplugas : >> I actually wrote a bucket-to-bucket copy function a couple of weeks >> ago, been meaning to send it in for ages: > > If you include tests it will be easier that Marcel includes your code into > the gem. Ah yes - now I remember why I hadn't sent it in :) At the time I was in a hell of a hurry, and couldn't get to grips with how the existing tests worked quickly enough (mocks and so on if I remember correctly). And yes, I know that's a poor, poor excuse :) I will take another look and send the patch through GitHub if I can get it sorted. Dave Lazar might find it useful in the meantime though ;) cheers, James http://www.floppy.org.uk From tiberiu.motoc at gmail.com Thu Jul 10 17:12:48 2008 From: tiberiu.motoc at gmail.com (Tiberiu Motoc) Date: Thu, 10 Jul 2008 14:12:48 -0700 Subject: [s3-dev] logging doesn't work for me Message-ID: <84ec458f0807101412n6c006020n59b96be1275bf447@mail.gmail.com> Hi, I attempted to enabled logging on my bucket and got a 200 OK response. I then tried to access the logs and got []. I uploaded some files to the bucket and deleted some older files, I reconnected, I made sure that logging is enabled on the bucket and still got []. I then created another bucket for logging purposes and enabled logging to this bucket. I'm still getting []. What am I doing wrong? Thanks, Tiberiu From marcel at vernix.org Thu Jul 10 17:16:11 2008 From: marcel at vernix.org (Marcel Molina Jr.) Date: Thu, 10 Jul 2008 16:16:11 -0500 Subject: [s3-dev] logging doesn't work for me In-Reply-To: <84ec458f0807101412n6c006020n59b96be1275bf447@mail.gmail.com> References: <84ec458f0807101412n6c006020n59b96be1275bf447@mail.gmail.com> Message-ID: I believe in the Amazon docs it says that logs aren't written to immediately. So there may be some lag between when you do some activity on your account and when that activity shows up in the logs. If it's only been under 20 or so minutes you might want to wait a bit and try to access the logs again. On Jul 10, 2008, at 4:12 PM, Tiberiu Motoc wrote: > Hi, > > I attempted to enabled logging on my bucket and got a 200 OK response. > I then tried to access the logs and got []. I uploaded some files to > the bucket and deleted some older files, I reconnected, I made sure > that logging is enabled on the bucket and still got []. > I then created another bucket for logging purposes and enabled logging > to this bucket. I'm still getting []. What am I doing wrong? > > Thanks, > Tiberiu > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev marcel -- Marcel Molina Jr. marcel at vernix.org From hunkybill at gmail.com Tue Jul 15 16:01:12 2008 From: hunkybill at gmail.com (Dave Lazar) Date: Tue, 15 Jul 2008 16:01:12 -0400 Subject: [s3-dev] COPY command and utf-8 Message-ID: Hi, I have had success uploading files to a bucket like this: image/foo.??.jpg Now, I am using a simple copy implementation to transfer this file to another bucket. The copy fails with S3 reporting the error: SignatureDoesNotMatch The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method The only time this occurs is when the filename in the bucket contains accented characters. Could someone kindly suggest a way to ensure when moving objects around that the encoding does not cause this kind of problem? I was trying to see how the aws::s3 put method dealt with the content-type but I failed to locate much. I am pretty sure that is part of the Net::HTTP libraries responsibility, but so far in my digging around I have not run accross how it is all hooked up... for example in object.rb. Any Tips most appreciated!! Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunkybill at gmail.com Thu Jul 17 19:51:32 2008 From: hunkybill at gmail.com (Dave Lazar) Date: Thu, 17 Jul 2008 19:51:32 -0400 Subject: [s3-dev] Copy command problems with accented characters Message-ID: Hi, I am testing this code snippet from James Smith (Thanks James!!) module AWS module S3 class S3Object def self.remote_copy(src_object, src_bucket, tgt_object = src_object, tgt_bucket = src_bucket) # Work out paths target_path = S3Object.path!(tgt_bucket, tgt_object, nil) source_path = S3Object.path!(src_bucket, src_object, nil) # Set copy header options = {:x_amz_copy_source => source_path} # Send PUT to do the copy put(target_path, options, nil) # Copy ACL from source object copied_object = S3Object.find(tgt_object, tgt_bucket) copied_object.acl(S3Object.acl(src_object, src_bucket)) end end end end This code works fine almost all cases I have tried except one really snarky one... If I try and copy any file with accented characters... the copy fails: A simple use case is illustrated here, source: /foo/foo???.bar target: /bar/foo???.bar Amazon S3 Error code 403 , The request signature we calculated does not match the signature you provided. Check your key and signing method. So, I am not sure why this is happening but it might be due to the authentication and signing algorithm screwing up on the key??? Any insights into how to go about debugging this would be much appreciated.... Thanks Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From edeleflie at gmail.com Thu Jul 17 21:58:32 2008 From: edeleflie at gmail.com (e deleflie) Date: Fri, 18 Jul 2008 11:58:32 +1000 Subject: [s3-dev] Does AWS::S3 support signed URLs for storing? Message-ID: <9a471d320807171858h16b5f733j2eb6a58aaacfc277@mail.gmail.com> Hi all, I have a client/server app I'm building. To avoid bundling the AWS secret keys with my little client app, I'm using S3's signed URL feature. My server (running python) is successfully generating signed URL's (signed for 'PUT', not 'GET'... I'm uploading) that look like this: https://myBucket.s3.amazonaws.com:443/somename%2Ftesting.jpg?Signature=YuyOUOuho87858gjguy65865kgh%3D&Expires=1216301872&AWSAccessKeyId=U878JAN7HUYFOU76HGBKMHF I'm wondering if S3Object.store() can use that signed URL (by setting 'options' perhaps?) Etienne From tiberiu.motoc at gmail.com Mon Jul 28 18:00:16 2008 From: tiberiu.motoc at gmail.com (Tiberiu Motoc) Date: Mon, 28 Jul 2008 15:00:16 -0700 Subject: [s3-dev] change default object list size Message-ID: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> Hi, I see that if you have over 1000 elements in your bucket, Bucket.objects and Bucket.logs will return the first 1000 elements. How can you retrieve all elements in a bucket? If you cannot, what is the best algorithm to parse through all of them? Thanks, Tiberiu From mdchaney at michaelchaney.com Mon Jul 28 19:43:37 2008 From: mdchaney at michaelchaney.com (Michael Chaney) Date: Mon, 28 Jul 2008 18:43:37 -0500 Subject: [s3-dev] change default object list size In-Reply-To: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> References: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> Message-ID: <3b456edd0807281643o5742ae13j36dbca27fbc22532@mail.gmail.com> On Mon, Jul 28, 2008 at 5:00 PM, Tiberiu Motoc wrote: > Hi, > > I see that if you have over 1000 elements in your bucket, > Bucket.objects and Bucket.logs will return the first 1000 elements. > How can you retrieve all elements in a bucket? If you cannot, what is > the best algorithm to parse through all of them? bucket = AWS::S3::Bucket.find(BUCKET_NAME) if bucket items = bucket.objects( :marker => '', :max_keys => 1000) while items.length > 0 STDERR.printf "%d amazon objects (partial)...\n", items.length if debug items.each do |item| puts "#{item.key} - #{item.about{'content-length'}}" # process items end last_object = items.last.key STDERR.printf "Getting at %s...\n", last_object if debug items = bucket.objects( :marker => last_object, :max_keys => 1000) end else STDERR.puts "Cannot find bucket #{BUCKET_NAME}" end You use "marker" to start at the beginning, then to continue finding everything past the last item, until you get an empty list. Michael -- Michael Darrin Chaney, Sr. mdchaney at michaelchaney.com http://www.michaelchaney.com/ From tiberiu.motoc at gmail.com Mon Jul 28 19:55:00 2008 From: tiberiu.motoc at gmail.com (Tiberiu Motoc) Date: Mon, 28 Jul 2008 16:55:00 -0700 Subject: [s3-dev] change default object list size In-Reply-To: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> References: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> Message-ID: <84ec458f0807281655l21759afemb473e0a156172d86@mail.gmail.com> Hi, One more thing: how do people delete the logs after they've read them? I see that log objects are not S3Object objects, therefore you couldn't call delete on them. What I did, was to extend the log class: class AWS::S3::Logging::Log def get_key @log.key end end So then as I go through each log, I do something like: AWS::S3::S3Object.find(log.get_key, bucket_name).delete And one last thing. if you delete a (log) file, it still shows up in the bucket objects list (if you run the Bucket.objects command immediately). I repeated the command an hour later and I noticed that some files that I have previously deleted did not show up any more - but not all. I then ran the delete_all_objects code over and over, and eventually none were listed. How long do you have to wait until they are all deleted? I don't want to run the delete method repeatedly. Thanks, Tiberiu On Mon, Jul 28, 2008 at 3:00 PM, Tiberiu Motoc wrote: > Hi, > > I see that if you have over 1000 elements in your bucket, > Bucket.objects and Bucket.logs will return the first 1000 elements. > How can you retrieve all elements in a bucket? If you cannot, what is > the best algorithm to parse through all of them? > > Thanks, > Tiberiu > From mdchaney at michaelchaney.com Mon Jul 28 20:08:01 2008 From: mdchaney at michaelchaney.com (Michael Chaney) Date: Mon, 28 Jul 2008 19:08:01 -0500 Subject: [s3-dev] change default object list size In-Reply-To: <84ec458f0807281655l21759afemb473e0a156172d86@mail.gmail.com> References: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> <84ec458f0807281655l21759afemb473e0a156172d86@mail.gmail.com> Message-ID: <3b456edd0807281708x2af4f73fv30cff323540588f0@mail.gmail.com> On Mon, Jul 28, 2008 at 6:55 PM, Tiberiu Motoc wrote: > Hi, > > One more thing: how do people delete the logs after they've read them? > I see that log objects are not S3Object objects, therefore you > couldn't call delete on them. What I did, was to extend the log class: > > class AWS::S3::Logging::Log > def get_key > @log.key > end > end > > So then as I go through each log, I do something like: > AWS::S3::S3Object.find(log.get_key, bucket_name).delete > > And one last thing. if you delete a (log) file, it still shows up in > the bucket objects list (if you run the Bucket.objects command > immediately). I repeated the command an hour later and I noticed that > some files that I have previously deleted did not show up any more - > but not all. I then ran the delete_all_objects code over and over, and > eventually none were listed. How long do you have to wait until they > are all deleted? I don't want to run the delete method repeatedly. To make your life easier, put the logs in a separate bucket, then don't worry about deleting them. They'll drop off after a couple of months if I remember correctly. Michael -- Michael Darrin Chaney, Sr. mdchaney at michaelchaney.com http://www.michaelchaney.com/ From marcel at vernix.org Tue Jul 29 14:04:04 2008 From: marcel at vernix.org (Marcel Molina Jr.) Date: Tue, 29 Jul 2008 18:04:04 +0000 Subject: [s3-dev] change default object list size In-Reply-To: <84ec458f0807281655l21759afemb473e0a156172d86@mail.gmail.com> References: <84ec458f0807281500s3b10825ere5e1e022707bec1b@mail.gmail.com> <84ec458f0807281655l21759afemb473e0a156172d86@mail.gmail.com> Message-ID: <20080729180404.GN10333@comox.textdrive.com> On Mon, Jul 28, 2008 at 04:55:00PM -0700, Tiberiu Motoc wrote: > And one last thing. if you delete a (log) file, it still shows up in > the bucket objects list (if you run the Bucket.objects command > immediately). I repeated the command an hour later and I noticed that > some files that I have previously deleted did not show up any more - > but not all. I then ran the delete_all_objects code over and over, and > eventually none were listed. How long do you have to wait until they > are all deleted? I don't want to run the delete method repeatedly. Bucket lists are cached as are a lot of network based / xml parsing based operations in the library. All memoized methods can be force cleared for the most up to date results by passing additional arguments. Check the code for details. marcel -- Marcel Molina Jr.