From tiberiu.motoc at gmail.com Thu Apr 3 21:28:15 2008 From: tiberiu.motoc at gmail.com (Tiberiu Motoc) Date: Thu, 3 Apr 2008 18:28:15 -0700 Subject: [s3-dev] uploads >1MB time out Message-ID: <84ec458f0804031828m1ea19fd6h292ce83c4732b543@mail.gmail.com> Hi, I just started using the aws-s3 Ruby library and the store() method keeps timing out. For small files (less than 1MB) the upload always works and is almost instantaneous. For larger files than 1MB, the upload is inconsistent: 5% of the uploads are successful and the upload time is approx 4 seconds (for a 1MB file). The rest of them, I keep getting the following error: /soft/fw/tools/ruby/ruby-1.8.5-rhel3/lib/ruby/1.8/net/protocol.rb:175:in `write': Broken pipe (Errno::EPIPE) or /soft/fw/tools/ruby/ruby-1.8.5-rhel3/lib/ruby/1.8/net/protocol.rb:175:in `write': Interrupt The Windows error is a bit more descriptive: D:/ruby/lib/ruby/1.8/net/protocol.rb:175:in `write': A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Er rno::ETIMEDOUT) I guess that aws-s3 is built on top of net which probably does a POST or a PUT, so is this an Amazon problem, or is it a aws-s3 problem? Thanks, Tiberiu PS. I have a T1 connection, so the upload should be very fast. From kookster at gmail.com Fri Apr 4 12:39:07 2008 From: kookster at gmail.com (Andrew Kuklewicz) Date: Fri, 4 Apr 2008 12:39:07 -0400 Subject: [s3-dev] uploads >1MB time out In-Reply-To: <84ec458f0804031828m1ea19fd6h292ce83c4732b543@mail.gmail.com> References: <84ec458f0804031828m1ea19fd6h292ce83c4732b543@mail.gmail.com> Message-ID: <19ba6d7f0804040939l179e0e26yb92885af0f2f3c91@mail.gmail.com> those error rates are very high. I see a very occasional error of that type (e.g. weeks if not months between them). In my app we use s3 and never send files less then 7 mbs (more often over 100 mbs) and I am not seeing anything like the error rate you are describing This implies to me this is not specifically an s3 issue, but something else....what, I could not say. -Andrew On Thu, Apr 3, 2008 at 9:28 PM, Tiberiu Motoc wrote: > Hi, > > I just started using the aws-s3 Ruby library and the store() method > keeps timing out. For small files (less than 1MB) the upload always > works and is almost instantaneous. > For larger files than 1MB, the upload is inconsistent: 5% of the > uploads are successful and the upload time is approx 4 seconds (for a > 1MB file). The rest of them, I keep getting the following error: > > /soft/fw/tools/ruby/ruby-1.8.5-rhel3/lib/ruby/1.8/net/protocol.rb:175:in > `write': Broken pipe (Errno::EPIPE) > or > /soft/fw/tools/ruby/ruby-1.8.5-rhel3/lib/ruby/1.8/net/protocol.rb:175:in > `write': Interrupt > The Windows error is a bit more descriptive: > D:/ruby/lib/ruby/1.8/net/protocol.rb:175:in `write': A connection attempt > failed > > because the connected party did not properly respond after a period of > time, or > > established connection failed because connected host has failed to > respond. (Er > > rno::ETIMEDOUT) > > I guess that aws-s3 is built on top of net which probably does a POST > or a PUT, so is this an Amazon problem, or is it a aws-s3 problem? > > Thanks, > Tiberiu > > PS. I have a T1 connection, so the upload should be very fast. > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev > -- Andrew Kuklewicz -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20080404/aa1a4d8e/attachment-0001.html From ben at reframeit.com Thu Apr 17 09:32:56 2008 From: ben at reframeit.com (Ben Taitelbaum) Date: Thu, 17 Apr 2008 09:32:56 -0400 Subject: [s3-dev] Sending IO to S3Object.store Message-ID: I'm trying to send some raw byte data (neither a string nor a file) into an S3 object, and feel that just passing an IO object instead of a File should work. This works after doing the following hack IO.class_eval do alias_method :lstat, :stat end so I'm wondering if it would make sense to change the line in connection.rb to: request.content_length = body.respond_to?(:stat) ? body.stat.size : body.size I think this would be safer, since according to the docs, File.open only has to return an IO, and not necessarily a File. I'm wondering if this has other implications, however. I've already noticed that since not all IO streams can be rewound, an error occurs on retry attempts. Here's my test code by the way: >> rd, wr = IO.pipe => [#, #] >> [0x0C, 0x0A].each{ |b| wr.putc b } => [12, 10] >> wr.close => nil >> S3Object.store('data.bin', rd, 'mybucket/test') => # >> obj = mybucket['test/data.bin'] => # >> obj.value => "\f\n" Thanks, Ben