[s3-dev] patch for 1000 objects limitation

[Metalhead]

Hi all,

 I've included a patch against 0.2.1 that works around the limit of not more
than 1000 objectss being returned by the service. I've successfully tested it with
a bucket containing 50000+ items.

I'm not entirely sure that I've added the patch at the right method because I
don't understand the code completely, so please feel free to comment on it and
suggest improvements.

I've also added '+' to the list of unsafe characters for URL escaping because S3
converts '+' to ' ' when used in the value of the marker option.

Regards,

Lars


diff -ru aws-s3-0.2.1/lib/aws/s3/bucket.rb aws-s3-0.2.1-mine/lib/aws/s3/bucket.rb
--- aws-s3-0.2.1/lib/aws/s3/bucket.rb	2006-12-04 08:29:30.000000000 +0100
+++ aws-s3-0.2.1-mine/lib/aws/s3/bucket.rb	2006-12-16 21:00:09.000000000 +0100
@@ -103,7 +103,15 @@
         # There are several options which allow you to limit which objects are retrieved. The list of object filtering options
         # are listed in the documentation for Bucket.objects.
         def find(name = nil, options = {})
-          new(get(path(name, options)).bucket)
+	    response = get(path(name, options)).bucket
+	    if not options.has_key?(:max_keys) and response['is_truncated']
+		begin
+		    options[:marker] = response['contents'].last['key']
+		    temp_response = get(path(name, options)).bucket
+		    response['contents'] += temp_response['contents']
+		end while(temp_response['is_truncated'])
+	    end
+	    new(response)
         end
         
         # Return just the objects in the bucket named <tt>name</tt>.
diff -ru aws-s3-0.2.1/lib/aws/s3/extensions.rb aws-s3-0.2.1-mine/lib/aws/s3/extensions.rb
--- aws-s3-0.2.1/lib/aws/s3/extensions.rb	2006-12-04 07:36:21.000000000 +0100
+++ aws-s3-0.2.1-mine/lib/aws/s3/extensions.rb	2006-12-16 21:00:09.000000000 +0100
@@ -1,5 +1,9 @@
 #:stopdoc:
 
+module URI
+    UNSAFE = Regexp.new(URI::UNSAFE.to_s.sub('+', ''))
+end
+
 class Hash
   def to_query_string(include_question_mark = true)
     return '' if empty?

-- 
4th Law of Hacking:  you will find the exit at the entrance.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20061216/d2cf609d/attachment.bin