[s3-dev] I can't S3Ojbect.store large files either.

Robin Lee Powell rlpowell at digitalkingdom.org
Mon Jan 19 15:26:37 EST 2009

On Mon, Jan 19, 2009 at 02:09:11PM -0600, Michael Chaney wrote:
> On Mon, Jan 19, 2009 at 1:39 PM, Robin Lee Powell
> <rlpowell at digitalkingdom.org> wrote:
> >
> > I'm having exactly the same problem as
> > http://rubyforge.org/pipermail/amazon-s3-dev/2008-September/000157.html
> >
> > I see there wasn't any answer to that post; I really need this
> > functionality.  Can anyone help?
> A lot of us, myself included, upload plenty of huge files, so we
> know it works.  I do know that some internet connections work
> better than others.

I fully accept that this works for you, and that the problem may be
on my end, but (1) it's not just me and (2) it's not a problem with
my internet connection, or anything simple like that.

We have dual gig-E to the internet from this datacenter.  It's *not*
an internet connection problem.  Furthermore, the file is only
2.3MiB, it's failing *every* time, and dd-ing the file takes
0.007042 seconds

How, exactly, are you doing the upload?  Are you breaking the POST
into chunks at all?

I've straced the error, and it's doing some really weird stuff:

write(5, "%PDF-1.2\n%\342\343\317\323\r\n2 0 obj\n<<\n/D [1"..., 1048576)         = 11680
write(5, "\233b\251\345\21k1\7\224O\313\221\202\352#\357\326\301"..., 1036896)         = 10220
write(5, "R\n/CharProcs 133 0 R\n/Widths [23"..., 1026676)         = 16060
write(5, "\2218\t\370\210\243\216\277\253\222\21\374\tUR\247DZ\000"..., 1010616) = -1 EPIPE (Broken pipe)

It's trying to send a mebibyte of data, but only 11680 bytes gets
actually written?  Buh-waaah?  There's also a *lot* of time between
the 3rd and 4th writes.


OK, "use_ssl => true" immediately fixes the problem.  It also makes
everything much slower, of course.  Fun times.

I wonder if it's some kind of weird firewall issue?

Oh well, I guess I have a workaround now; use ssl for larger files.

If people want to keep digging into this to figure out how to
actually solve it, though, I'm available.


They say:  "The first AIs will be built by the military as weapons."
And I'm thinking:  "Does it even occur to you to try for something
other than the default outcome?" -- http://shorl.com/tydruhedufogre
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

More information about the amazon-s3-dev mailing list