From eggie5 at gmail.com Wed Sep 26 15:40:34 2007 From: eggie5 at gmail.com (Alex Egg) Date: Wed, 26 Sep 2007 12:40:34 -0700 Subject: [s3-dev] file getting cut off Message-ID: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> I have a 11676 line text file that seems to be getting cut off about 10 lines from the end. Has anybody experienced this before? AWS::S3::S3Object.store(File.basename(s_file), open(s_file), url, :access => :public_read, :content_type => 'text/javascript') -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070926/365814b9/attachment.html From francesc.esplugas at gmail.com Thu Sep 27 09:49:12 2007 From: francesc.esplugas at gmail.com (Francesc Esplugas) Date: Thu, 27 Sep 2007 15:49:12 +0200 Subject: [s3-dev] file getting cut off In-Reply-To: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> References: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> Message-ID: <9ABF860C-D227-4B9C-9846-5E01B94BF62E@gmail.com> No. I've just tested uploading a file right now an everything seems ok. I've downloaded again an a `diff file.js file-1.js` doesn't return any difference between the files. http://s3.amazonaws.com/morninglabs/file.js? AWSAccessKeyId=061A0XVVHJJN67AXPKG2&Expires=1190900908&Signature=01YiHWO 2uG4jckRf1H6BWmxYi88%3D ## file = 'file.js' S3Object.store(file, open(file), 'morninglabs', :content_type => 'text/javascript') S3Object.url_for file, 'morninglabs' ## The Ruby gem I've installed is version 0.4.0 Hope that helps, Francesc On Sep 26, 2007, at 9:40 PM, Alex Egg wrote: > I have a 11676 line text file that seems to be getting cut off > about 10 lines from the end. Has anybody experienced this before? > > AWS::S3::S3Object.store(File.basename(s_file), open(s_file), > url, :access => :public_read, :content_type => 'text/javascript') > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev -- name. francesc.esplugas email. francesc.esplugas at gmail.com voice. 00 34 678 681 603 skype. fesplugas From benles at bldigital.com Thu Sep 27 12:57:55 2007 From: benles at bldigital.com (Benjamin Allfree) Date: Thu, 27 Sep 2007 09:57:55 -0700 Subject: [s3-dev] Multi-threading In-Reply-To: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> References: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> Message-ID: Does anyone know how to fix the multithreading issue? It seems like AWS::S3 isn't used in production environments. In the current build, here are the problems I have confirmed: * Keys with + in them can not be retrieved * Objects >1GB can not be uploaded (Ruby network error) * No multithreading support Any help is appreciated. I fixed the + thing and submitted a patch. I'm at a complete loss about the 1GB limit. It just says the connection was forcibly closed and I can't see why because it's SSL. Ben 805-204-4070 x 101 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070927/546f1661/attachment.html From benles at bldigital.com Thu Sep 27 16:02:17 2007 From: benles at bldigital.com (Benjamin Allfree) Date: Thu, 27 Sep 2007 13:02:17 -0700 Subject: [s3-dev] Multi-threading In-Reply-To: <5185CFBB-313B-47C5-AADC-EE015DEBC142@web.de> References: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> <5185CFBB-313B-47C5-AADC-EE015DEBC142@web.de> Message-ID: It calls URI.encode() in connection.rb Does your key actually have a %20 in it, or is that after encoding. If you do a PUT "my key" you should have to GET "my%20key". That should work correctly with URI.encode() Likewise, if you PUT "my+key" you should have to get "my%2Bkey". That is the one that doesn't work correctly with URI.encode(). It's dumb, but with S3 you must use URI encoding when you request a GET even though the URI spec say it's not required. http://s3.amazonaws.com/my folder/my key.txt should work, but it doesn't. It has to be http://s3.amazonaws.com/my%20folder/my%20key.txt. The URI encoding is only supposed to be necessary for querystring parameters, not the main URI path. S3's implementation is wrong and nonstandard in my opinion because FireFox and other browsers will not encode the main URI path when sending the request. PS, this mailing list is kind of strange because when you hit "reply" it replies privately to the sender instead of the list. Any way to fix that? I bet that's why the traffic volume on this list is so low. Ben 805-204-4070 x 101 On Sep 27, 2007, at 12:49 PM, Niko Dittmann wrote: > It doesn't only seem to have problems with plusses in the key but > with other stuff, too. For example I observed S3 interprets as %20 > encoded blanks (so they appear as blanks and have to be retrieved > as blank). So if you PUT to /mybucket/my%20key.txt you will have > to download /mybucket/my key.txt. Other encoded UTF-8 strings seem > to stay encoded. > > No idea about the multithreading thing. Perhaps you can try to > solve it with backgroundrb. > > Niko. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070927/24cad534/attachment-0001.html From francesc.esplugas at gmail.com Thu Sep 27 16:07:19 2007 From: francesc.esplugas at gmail.com (Francesc Esplugas) Date: Thu, 27 Sep 2007 22:07:19 +0200 Subject: [s3-dev] Multi-threading In-Reply-To: References: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> <5185CFBB-313B-47C5-AADC-EE015DEBC142@web.de> Message-ID: <67D404D6-FE17-40EE-8237-F04F2BA8987D@gmail.com> First of all: Use "reply to all" to reply the sender and the list. I've never uploaded a file to s3 with AWS::S3 without having "cleaned" the filename. So I've never experienced this issue. A user has sent a patch to Marcel fixing the problems with the "+" character on the filename. Hope this fixes part of your problem. Francesc On Sep 27, 2007, at 10:02 PM, Benjamin Allfree wrote: > It calls URI.encode() in connection.rb > > Does your key actually have a %20 in it, or is that after encoding. > > If you do a PUT "my key" you should have to GET "my%20key". That > should work correctly with URI.encode() > > Likewise, if you PUT "my+key" you should have to get "my%2Bkey". > That is the one that doesn't work correctly with URI.encode(). > > It's dumb, but with S3 you must use URI encoding when you request a > GET even though the URI spec say it's not required. > > http://s3.amazonaws.com/my folder/my key.txt should work, but it > doesn't. It has to be > http://s3.amazonaws.com/my%20folder/my%20key.txt. The URI encoding > is only supposed to be necessary for querystring parameters, not > the main URI path. S3's implementation is wrong and nonstandard in > my opinion because FireFox and other browsers will not encode the > main URI path when sending the request. > > PS, this mailing list is kind of strange because when you hit > "reply" it replies privately to the sender instead of the list. Any > way to fix that? I bet that's why the traffic volume on this list > is so low. > > > > Ben > 805-204-4070 x 101 > > > > On Sep 27, 2007, at 12:49 PM, Niko Dittmann wrote: > >> It doesn't only seem to have problems with plusses in the key but >> with other stuff, too. For example I observed S3 interprets as % >> 20 encoded blanks (so they appear as blanks and have to be >> retrieved as blank). So if you PUT to /mybucket/my%20key.txt you >> will have to download /mybucket/my key.txt. Other encoded UTF-8 >> strings seem to stay encoded. >> >> No idea about the multithreading thing. Perhaps you can try to >> solve it with backgroundrb. >> >> Niko. >> > > _______________________________________________ > amazon-s3-dev mailing list > amazon-s3-dev at rubyforge.org > http://rubyforge.org/mailman/listinfo/amazon-s3-dev -- name. francesc.esplugas email. francesc.esplugas at gmail.com voice. 00 34 678 681 603 skype. fesplugas From benles at bldigital.com Thu Sep 27 18:09:24 2007 From: benles at bldigital.com (Benjamin Allfree) Date: Thu, 27 Sep 2007 15:09:24 -0700 Subject: [s3-dev] Multi-threading In-Reply-To: <67D404D6-FE17-40EE-8237-F04F2BA8987D@gmail.com> References: <6f7401650709261240r15bac2c0rf8d1f43712ae4a1e@mail.gmail.com> <5185CFBB-313B-47C5-AADC-EE015DEBC142@web.de> <67D404D6-FE17-40EE-8237-F04F2BA8987D@gmail.com> Message-ID: I was that user :) Ben 805-204-4070 x 101 On Sep 27, 2007, at 1:07 PM, Francesc Esplugas wrote: > > First of all: Use "reply to all" to reply the sender and the list. > > I've never uploaded a file to s3 with AWS::S3 without having > "cleaned" the filename. So I've never experienced this issue. > > A user has sent a patch to Marcel fixing the problems with the "+" > character on the filename. Hope this fixes part of your problem. > > Francesc > > On Sep 27, 2007, at 10:02 PM, Benjamin Allfree wrote: > >> It calls URI.encode() in connection.rb >> >> Does your key actually have a %20 in it, or is that after encoding. >> >> If you do a PUT "my key" you should have to GET "my%20key". That >> should work correctly with URI.encode() >> >> Likewise, if you PUT "my+key" you should have to get "my%2Bkey". >> That is the one that doesn't work correctly with URI.encode(). >> >> It's dumb, but with S3 you must use URI encoding when you request >> a GET even though the URI spec say it's not required. >> >> http://s3.amazonaws.com/my folder/my key.txt should work, but it >> doesn't. It has to be >> http://s3.amazonaws.com/my%20folder/my%20key.txt. The URI encoding >> is only supposed to be necessary for querystring parameters, not >> the main URI path. S3's implementation is wrong and nonstandard in >> my opinion because FireFox and other browsers will not encode the >> main URI path when sending the request. >> >> PS, this mailing list is kind of strange because when you hit >> "reply" it replies privately to the sender instead of the list. >> Any way to fix that? I bet that's why the traffic volume on this >> list is so low. >> >> >> >> Ben >> 805-204-4070 x 101 >> >> >> >> On Sep 27, 2007, at 12:49 PM, Niko Dittmann wrote: >> >>> It doesn't only seem to have problems with plusses in the key but >>> with other stuff, too. For example I observed S3 interprets as % >>> 20 encoded blanks (so they appear as blanks and have to be >>> retrieved as blank). So if you PUT to /mybucket/my%20key.txt you >>> will have to download /mybucket/my key.txt. Other encoded UTF-8 >>> strings seem to stay encoded. >>> >>> No idea about the multithreading thing. Perhaps you can try to >>> solve it with backgroundrb. >>> >>> Niko. >>> >> >> _______________________________________________ >> amazon-s3-dev mailing list >> amazon-s3-dev at rubyforge.org >> http://rubyforge.org/mailman/listinfo/amazon-s3-dev > > -- > name. francesc.esplugas > email. francesc.esplugas at gmail.com > voice. 00 34 678 681 603 > skype. fesplugas > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070927/4801d421/attachment.html