From lon at speedymac.com  Mon Apr  2 08:16:48 2007
From: lon at speedymac.com (Lon Baker)
Date: Mon, 2 Apr 2007 08:16:48 -0400
Subject: [s3-dev] Connection issues with 0.3.0?
Message-ID: <EAE9A35B-02A2-48A9-9B59-38DA6FACCF14@speedymac.com>

Hi there-

Recently deployed a version of one of my production applications on  
Amazon EC2 (Fedora Core 6, Ruby 1.8.6, Mongrel, Lighttpd) and am  
encountering what seems like some sort of connection issue.

Everything works great for a short period of time and then the  
following error comes up:

> /srv/www/mailtank/config/environment.rb:102:in `sysread'
> /srv/www/mailtank/config/environment.rb:102:in `rbuf_fill'
> /usr/local/lib/ruby/1.8/timeout.rb:56:in `timeout'
> /usr/local/lib/ruby/1.8/timeout.rb:76:in `timeout'
> /srv/www/mailtank/config/environment.rb:101:in `rbuf_fill'
> /usr/local/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
> /usr/local/lib/ruby/1.8/net/protocol.rb:126:in `readline'
> /usr/local/lib/ruby/1.8/net/http.rb:2017:in `read_status_line'
> /usr/local/lib/ruby/1.8/net/http.rb:2006:in `read_new'
> /usr/local/lib/ruby/1.8/net/http.rb:1047:in `request'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/ 
> connection.rb:41:in `request'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/ 
> connection.rb:46:in `call'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/ 
> connection.rb:46:in `request'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb: 
> 69:in `request'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb: 
> 84:in `get'
> /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/object.rb: 
> 134:in `value'
> #{RAILS_ROOT}/vendor/plugins/attachment_fu/lib/technoweenie/ 
> attachment_fu/backends/s3_backend.rb:250:in `current_data'

I am using Rick Olson's excellent attachment_fu plugin and after  
investigating attachment_fu, seemingly known issues with rbuf_fill  
having a tiny buffer and the aws-s3 module I found the only solution  
is to turn off persistent connections.

When I do that everything works great.

Has anyone else experienced this issue?

--
Lon Baker
http://speedymac.com



From marcel at vernix.org  Mon Apr  2 14:03:29 2007
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Mon, 2 Apr 2007 18:03:29 +0000
Subject: [s3-dev] Connection issues with 0.3.0?
In-Reply-To: <EAE9A35B-02A2-48A9-9B59-38DA6FACCF14@speedymac.com>
References: <EAE9A35B-02A2-48A9-9B59-38DA6FACCF14@speedymac.com>
Message-ID: <20070402180329.GR83368@comox.textdrive.com>

On Mon, Apr 02, 2007 at 08:16:48AM -0400, Lon Baker wrote:
> Recently deployed a version of one of my production applications on  
> Amazon EC2 (Fedora Core 6, Ruby 1.8.6, Mongrel, Lighttpd) and am  
> encountering what seems like some sort of connection issue.
> 
> Everything works great for a short period of time and then the  
> following error comes up:
> 
> When I do that everything works great.
> 
> Has anyone else experienced this issue?

Yup. There's a bug somewhere in the persistant connection code, either on
account of aws/s3 or somewhere down in net/http. I'd recommend just sticking
with non persistant connections for now.

marcel
-- 
Marcel Molina Jr. <marcel at vernix.org>

From lon at speedymac.com  Mon Apr  2 14:16:39 2007
From: lon at speedymac.com (Lon Baker)
Date: Mon, 2 Apr 2007 14:16:39 -0400
Subject: [s3-dev] Connection issues with 0.3.0?
In-Reply-To: <20070402180329.GR83368@comox.textdrive.com>
References: <EAE9A35B-02A2-48A9-9B59-38DA6FACCF14@speedymac.com>
	<20070402180329.GR83368@comox.textdrive.com>
Message-ID: <1285CD3F-6A81-41E7-A920-E0E0BBD12F75@speedymac.com>

Marcel,

Thanks for the response. I have been using non-persistent connections  
and everything works great.

Before posting to the list here, I tried the numerous patches from  
the Ruby mailing list for net/http. None of them helped.

I managed to verify that the aws/s3 connection claims to be connected  
when the error happens. The S3 forums have some others talking about  
similar issues with other S3 code, so I am not sure that it is in  
your code.

One interesting twist, if I run the application on OS X with the  
exact same stack and production configuration, it never has an issue  
- ever - with persistent connections.

It is only happening in production on Fedora/EC2. I am going to try  
it on an xserve at our datacenter later this week.

--
Lon Baker
http://speedymac.com


On Apr 2, 2007, at 2:03 PM, Marcel Molina Jr. wrote:

> On Mon, Apr 02, 2007 at 08:16:48AM -0400, Lon Baker wrote:
>> Recently deployed a version of one of my production applications on
>> Amazon EC2 (Fedora Core 6, Ruby 1.8.6, Mongrel, Lighttpd) and am
>> encountering what seems like some sort of connection issue.
>>
>> Everything works great for a short period of time and then the
>> following error comes up:
>>
>> When I do that everything works great.
>>
>> Has anyone else experienced this issue?
>
> Yup. There's a bug somewhere in the persistant connection code,  
> either on
> account of aws/s3 or somewhere down in net/http. I'd recommend just  
> sticking
> with non persistant connections for now.
>
> marcel
> -- 
> Marcel Molina Jr. <marcel at vernix.org>
> _______________________________________________
> amazon-s3-dev mailing list
> amazon-s3-dev at rubyforge.org
> http://rubyforge.org/mailman/listinfo/amazon-s3-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070402/44a06a87/attachment-0001.html 

From maccman at gmail.com  Wed Apr  4 12:23:26 2007
From: maccman at gmail.com (Alex MacCaw)
Date: Wed, 4 Apr 2007 17:23:26 +0100
Subject: [s3-dev] Meta data on upload
Message-ID: <14cc92570704040923g57134862y6a4d8093f594ce1a@mail.gmail.com>

Is setting metadata now supported on upload?

-- 
http://www.eribium.org | http://juggernaut.rubyforge.org |
http://www.aireofs.com | Skype: oldmanorhouse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070404/c09afaa9/attachment.html 

From kookster at gmail.com  Wed Apr  4 12:51:35 2007
From: kookster at gmail.com (Andrew Kuklewicz)
Date: Wed, 4 Apr 2007 12:51:35 -0400
Subject: [s3-dev] Meta data on upload
In-Reply-To: <14cc92570704040923g57134862y6a4d8093f594ce1a@mail.gmail.com>
References: <14cc92570704040923g57134862y6a4d8093f594ce1a@mail.gmail.com>
Message-ID: <19ba6d7f0704040951v46add327k512e91ee79c20183@mail.gmail.com>

I've been setting meta-data no problem, including some jets3t compatible
meta-data, here;s some sample of my code in production:

  def establish_s3_connection()
    AWS::S3::Base.establish_connection!(
        :access_key_id     => AppConfig.aws_access_key_id,
        :secret_access_key => AppConfig.aws_secret_access_key
      )
  end

def upload_file()
  establish_s3_connection
  AWS::S3::S3Object.store(get_file_name,
                          open(get_file_path),
                          AppConfig.s3_bucket,
                          :content_type=>'audio/mpeg',
                          'x-amz-meta-my_uploadname'=>self.name,
                          'x-amz-meta-my_upload_ownername'=>
self.owner.username,
                          'x-amz-meta-my_upload_date'=>
self.upload_date.strftime("%b %d, %Y %H:%M:%S"),
                          'x-amz-meta-my_database_mimetype'=>self.mime_type,
                          'x-amz-meta-my_myfile_id'=>self.id,
                          'x-amz-meta-jets3t-original-file-date-iso860'=>
File.mtime(get_file_path).strftime("%Y-%m-%dT%H:%M:%S.000Z"))
end



On 4/4/07, Alex MacCaw <maccman at gmail.com> wrote:
>
> Is setting metadata now supported on upload?
>
> --
> http://www.eribium.org | http://juggernaut.rubyforge.org |
> http://www.aireofs.com | Skype: oldmanorhouse
> _______________________________________________
> amazon-s3-dev mailing list
> amazon-s3-dev at rubyforge.org
> http://rubyforge.org/mailman/listinfo/amazon-s3-dev
>
>


-- 
Andrew Kuklewicz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070404/b045132c/attachment.html 

From simon at soulware.co.uk  Thu Apr  5 06:58:46 2007
From: simon at soulware.co.uk (Simon Horne)
Date: Thu, 5 Apr 2007 05:58:46 -0500 (CDT)
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
Message-ID: <1175770726.v2.fusewebmail-9566@f>

Hi,

it does not appear to be possible to access S3 with aws-s3 via a proxy.
I've had a look in connection.rb and there is a direct call to
Net::HTTP.new().
It looks like it would be simple enough to use Net::HTTP::Proxy().new()
and to use options to specify the proxy
settings.
If no :proxy_host is specified then it will safely default to no proxy.

Would this be feasible?

See below for a patch (from 0.3.0 release) -


82c82
<           @http             = Net::HTTP.new(options[:server],
options[:port])
---
>           @http             = Net::HTTP::Proxy(options[:proxy_addr],
options[:proxy_port], options[:proxy_user],
options[:proxy_pass]).new(options[:server], options[:port])
211c211
<             [:access_key_id, :secret_access_key, :server, :port,
:use_ssl, :persistent]
---
>             [:access_key_id, :secret_access_key, :server, :port,
:use_ssl, :persistent, :proxy_addr, :proxy_port,
:proxy_user, :proxy_pass]



Thanks,

Simon

From greg at supplybeyond.com  Thu Apr  5 16:06:12 2007
From: greg at supplybeyond.com (Greg Yut)
Date: Thu, 5 Apr 2007 15:06:12 -0500
Subject: [s3-dev] AWS S3 on Debian Sarge -- error in connection.rb
Message-ID: <20070405205111.9A3475240A1C@rubyforge.org>

I have a working AWS S3 with Rails on Fedora (dev env), but when I try to
run the same on Debian Sarge (prod env), I get the error below. The Fedora
setup is running Ruby 1.8.4, while the Debian is running 1.8.2.
 
Is this a known issue, or am I just doing something goofy?
 
Thanks,
 
Greg
 
 
NoMethodError (undefined method `body=' for #<Net::HTTP::Put PUT>):
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/connection.rb:38:in
`request'
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/connection.rb:27:in
`call'
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/connection.rb:46:in
`request'
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb:69:in
`request'
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/base.rb:84:in `put'
    /usr/lib/ruby/gems/1.8/gems/aws-s3-0.3.0/lib/aws/s3/object.rb:239:in
`store'
 
/vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu/backends/s3_bac
kend.rb:289:in `sav$
    /vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:358:in
`after_process_attachmen$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.
rb:333:in `send'
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.
rb:333:in `callback$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.
rb:330:in `each'
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.
rb:330:in `callback$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/callbacks.
rb:243:in `create_o$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/base.rb:15
51:in `save_without$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/validation
s.rb:762:in `save_w$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactio
ns.rb:133:in `save!$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactio
ns.rb:133:in `trans$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactio
ns.rb:95:in `transa$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactio
ns.rb:121:in `trans$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/transactio
ns.rb:133:in `save!$
 
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/validation
s.rb:727:in `create$
    /app/controllers/article_images_controller.rb:21:in `create'
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:
1095:in `send'
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:
1095:in `perform_ac$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:632:in `call_fil$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:638:in `call_fil$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:637:in `call'
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:637:in `call_fil$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:619:in `perform_$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmar
king.rb:66:in `perf$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmar
king.rb:66:in `meas$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/benchmar
king.rb:66:in `perf$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.r
b:83:in `perform_ac$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:
430:in `send'
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:
430:in `process_wit$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/filters.
rb:624:in `process_$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/session_
management.rb:114:i$
 
/usr/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/base.rb:
330:in `process'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/dispatcher.rb:41:in
`dispatch'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:168:in
`process_request'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:143:in
`process_each_request!'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:142:in
`with_signal_handler'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:142:in
`process_each_request!'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:141:in
`each_cgi'
    /usr/local/lib/site_ruby/1.8/fcgi.rb:597:in `each'
    /usr/local/lib/site_ruby/1.8/fcgi.rb:597:in `each_cgi'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:141:in
`process_each_request!'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:55:in
`process!'
    /usr/lib/ruby/gems/1.8/gems/rails-1.2.3/lib/fcgi_handler.rb:25:in
`process!'
    /home/yut/community/public/dispatch.fcgi:24
 

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://rubyforge.org/pipermail/amazon-s3-dev/attachments/20070405/01a0e06d/attachment.html 

From marcel at vernix.org  Thu Apr  5 18:42:04 2007
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Thu, 5 Apr 2007 22:42:04 +0000
Subject: [s3-dev] AWS S3 on Debian Sarge -- error in connection.rb
In-Reply-To: <20070405205111.9A3475240A1C@rubyforge.org>
References: <20070405205111.9A3475240A1C@rubyforge.org>
Message-ID: <20070405224204.GX83368@comox.textdrive.com>

On Thu, Apr 05, 2007 at 03:06:12PM -0500, Greg Yut wrote:
>    I have a working AWS S3 with Rails on Fedora (dev env), but when I try
>    to run the same on Debian Sarge (prod env), I get the error below. The
>    Fedora setup is running Ruby 1.8.4, while the Debian is running 1.8.2.
> 
>    Is this a known issue, or am I just doing something goofy?

Yup, you must have at least 1.8.4 to use the library. If you can not upgrade
Ruby on Sarge then you could get around the issue by monkey patching
Net::HTTP.

marcel
-- 
Marcel Molina Jr. <marcel at vernix.org>

From greg at supplybeyond.com  Thu Apr  5 19:40:22 2007
From: greg at supplybeyond.com (Greg Yut)
Date: Thu, 5 Apr 2007 18:40:22 -0500
Subject: [s3-dev] AWS S3 on Debian Sarge -- error in connection.rb
In-Reply-To: <20070405224204.GX83368@comox.textdrive.com>
Message-ID: <20070405234055.CE1F252409A7@rubyforge.org>

Thanks for confirming, Marcel.

I searched for guidance on upgrading Ruby to 1.8.4 on Debian and Ezra
Zygmuntowicz once again saved the day. I followed his script (manually) and
it worked perfectly.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/197077

My small app is now up-and-running with images served from S3.

Thanks to both you and Ezra!

Greg



-----Original Message-----
From: amazon-s3-dev-bounces at rubyforge.org
[mailto:amazon-s3-dev-bounces at rubyforge.org] On Behalf Of Marcel Molina Jr.
Sent: Thursday, April 05, 2007 5:42 PM
To: amazon-s3-dev at rubyforge.org
Subject: Re: [s3-dev] AWS S3 on Debian Sarge -- error in connection.rb

On Thu, Apr 05, 2007 at 03:06:12PM -0500, Greg Yut wrote:
>    I have a working AWS S3 with Rails on Fedora (dev env), but when I try
>    to run the same on Debian Sarge (prod env), I get the error below. The
>    Fedora setup is running Ruby 1.8.4, while the Debian is running 1.8.2.
> 
>    Is this a known issue, or am I just doing something goofy?

Yup, you must have at least 1.8.4 to use the library. If you can not upgrade
Ruby on Sarge then you could get around the issue by monkey patching
Net::HTTP.

marcel
--
Marcel Molina Jr. <marcel at vernix.org>
_______________________________________________
amazon-s3-dev mailing list
amazon-s3-dev at rubyforge.org
http://rubyforge.org/mailman/listinfo/amazon-s3-dev



From edeleflie at gmail.com  Thu Apr  5 21:14:29 2007
From: edeleflie at gmail.com (e deleflie)
Date: Fri, 6 Apr 2007 11:14:29 +1000
Subject: [s3-dev] bug #8171
Message-ID: <9a471d320704051814n33660de3wc6416954271b3e25@mail.gmail.com>

All,

Now that there is a bit of activity, can I point the list to bug
#8171? ... where the aws-s3 lib fails to upload to S3 on windows.

http://rubyforge.org/tracker/index.php?func=detail&aid=8171&group_id=2409&atid=9356

I know that in open source projects, developers are encouraged to
solve bugs themselves and submit a patch. I spent 2 days trying to
hunt down this bug. honestly! .... but I am a Java developer and Ruby
Newby .. whilst I leanrt a lot about Ruby trying to solve this bug, I
had to move on to other work.

If someone who knows Ruby well and has a windows machine ... I bet
this is a relatively simple issue ... it would be greatly appreciated
if someone could have a look at it ....

Etienne

From marcel at vernix.org  Thu Apr  5 11:20:27 2007
From: marcel at vernix.org (Marcel Molina Jr.)
Date: Thu, 5 Apr 2007 15:20:27 +0000
Subject: [s3-dev] using aws-s3 via a proxy (Net::HTTP::Proxy)
In-Reply-To: <1175770726.v2.fusewebmail-9566@f>
References: <1175770726.v2.fusewebmail-9566@f>
Message-ID: <20070405152027.GV83368@comox.textdrive.com>

On Thu, Apr 05, 2007 at 05:58:46AM -0500, Simon Horne wrote:
> it does not appear to be possible to access S3 with aws-s3 via a proxy.
> I've had a look in connection.rb and there is a direct call to
> Net::HTTP.new().
> It looks like it would be simple enough to use Net::HTTP::Proxy().new()
> and to use options to specify the proxy
> settings.
> If no :proxy_host is specified then it will safely default to no proxy.
> 
> Would this be feasible?
> 
> See below for a patch (from 0.3.0 release) -
> 
> 82c82
> <           @http             = Net::HTTP.new(options[:server],
> options[:port])
> ---
> >           @http             = Net::HTTP::Proxy(options[:proxy_addr],
> options[:proxy_port], options[:proxy_user],
> options[:proxy_pass]).new(options[:server], options[:port])
> 211c211
> <             [:access_key_id, :secret_access_key, :server, :port,
> :use_ssl, :persistent]
> ---
> >             [:access_key_id, :secret_access_key, :server, :port,
> :use_ssl, :persistent, :proxy_addr, :proxy_port,
> :proxy_user, :proxy_pass]

Hey Simon, that looks pretty reasonable. I'll shoe horn it in. Thanks.

marcel
-- 
Marcel Molina Jr. <marcel at vernix.org>