From tiberiu.motoc at gmail.com  Fri May  1 15:32:50 2009
From: tiberiu.motoc at gmail.com (Tiberiu Motoc)
Date: Fri, 1 May 2009 12:32:50 -0700
Subject: [s3-dev] Errno::ECONNRESET: Connection reset by peer
Message-ID: <84ec458f0905011232h26293889pda977d49fb46beea@mail.gmail.com>

Hi,

I keep getting this error in my production environment for large
files. Initially I thought it was because I was calling the store
method from Rails, but then I tried it with s3sh, and I still get it.
25% of upload attempts are successful. Again, this error happens for
large files only. The file I'm currently testing with is 100MB.
Did anyone encounter this problem? Am I supposed to handle large file
uploads differently?
Btw, this is my call to the store method:

S3Object.store('fileset.zip', File.open('fileset.zip'), BUCKET_NAME,
:access => :public_read)

Thanks,
Tiberiu

Errno::ECONNRESET: Connection reset by peer
        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write0'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:151:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:166:in `writing'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:150:in `write'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/extensions.rb:292:in
`send_request_with_body_stream'
        from /usr/local/lib/ruby/1.8/net/http.rb:1524:in `exec'
        from /usr/local/lib/ruby/1.8/net/http.rb:1045:in `request'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/connection.rb:45:in
`request'
        from /usr/local/lib/ruby/1.8/net/http.rb:543:in `start'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/connection.rb:52:in
`request'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/base.rb:69:in
`request'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/base.rb:83:in
`put'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.5.1/bin/../lib/aws/s3/object.rb:241:in
`store'
        from (irb):5

From marcel at vernix.org  Fri May 22 17:38:54 2009
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Fri, 22 May 2009 14:38:54 -0700
Subject: [s3-dev] Optimizing file downloads - AWS::S3::S3Object.stream
In-Reply-To: <fb2fc0360905221404l7c4a1251l3108f67b50a63970@mail.gmail.com>
References: <fb2fc0360905221404l7c4a1251l3108f67b50a63970@mail.gmail.com>
Message-ID: <05FA70F2-8508-4B6C-B9CD-A245E0F58ADE@vernix.org>

On May 22, 2009, at 2:04 PM, Robert Jones wrote:

> I'm downloading files from S3 into a Rails app using S3Object.stream  
> like this:
>
>         open(local_path, 'w') do |file|
>           AWS::S3::S3Object.stream(remote_path, @aws_bucket_name) do  
> |chunk|
>             file.write chunk
>           end
>         end
>
> The files are not big - 1 or 2 MB - but sometimes I'm seeing very  
> slow downloads (over a minute)
> The intermittent delays suggest either an S3 or a network issue, but  
> I'm wondering if AWS::S3 is playing a part.
> Downloading a 1.6MB file resulted in 880 'chunks' being written - so  
> a chunk is ~2Kbytes
> Is there a way to specify chunk size? and would using fewer, larger  
> chunks make any difference?
>
> I've looked at the code but I'm not getting how streaming works,  
> sorry...
>
> Or is there some obvious better way to do this (s3sync/s3cmd) ?
>
> (Thanks for updating the code to work with Ruby 1.9, by the way!)

The stream method simply delegates down to Net::HTTP's streaming code  
in read_body (written in terms of read_body_0). I'd suggest profiling  
a download to see exactly where most of the time is being spent. I've  
opened up Net::HTTP in the extensions.rb file to make the default  
chunk size larger than 2kbytes (1 MB), but I believe the streaming  
code in Net::HTTP hard codes yet again its chunk size.

I don't know the details of s3sync/s3cmd. Perhaps someone else on the  
list does (?)