"upstream timed out" after upgrades

John-Paul Bader hukl at h3q.com
Thu Feb 4 03:48:50 EST 2010


Hrm its getting weirder and weirder.

I started unicorn_rails now manually in the app root which is now running on rails 2.3.3 again: 

sudo unicorn_rails -E production -d -c /usr/local/etc/unicorn.rb

Now first of all it throws lots of Exceptions it couldn't find several (important) gems and then it produces this every second or so:

Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)

When I request the javascript in question I get:

Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn/http_response.rb:67 - Invalid argument
Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)

This is the same no matter if I run it with -c or not. 

I am puzzled.

To give you the entire output of unicorn starting:

[hukl at cccms /usr/local/www/cccms]$ sudo unicorn_rails -E production -d -p 9090
{:daemonize=>false,
 :unicorn_options=>{:listeners=>["0.0.0.0:9090"]},
 :app=>
  #<Proc:0x0000000801782388@/usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/bin/unicorn_rails:124>}
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/cache
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/pids
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sessions
Exception `Errno::EEXIST' at /usr/local/lib/ruby/1.8/fileutils.rb:243 - File exists - tmp/sockets
I, [2010-02-04T09:42:08.141034 #38499]  INFO -- : listening on addr=213.73.89.122:9090 fd=3
Exception `LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- Win32API
Exception `LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- Win32API
I, [2010-02-04T09:42:08.158275 #38499]  INFO -- : worker=0 spawning...
I, [2010-02-04T09:42:08.159352 #38499]  INFO -- : master process ready
I, [2010-02-04T09:42:08.159471 #38501]  INFO -- : worker=0 spawned pid=38501
I, [2010-02-04T09:42:08.197082 #38501]  INFO -- : Refreshing Gem list
Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem builder (~> 2.1.2)

Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem memcache-client (>= 1.7.4)

Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem tzinfo (~> 0.3.12)

Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem i18n (~> 0.1.3)

Exception `TypeError' at (eval):4 - can't modify frozen object
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- fast_xs
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- json
Using c extension for JSON.
Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem tmail (~> 1.2.3)

Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- tmail/tmailscanner
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- tmail/tmailscanner
Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -- tmail/tmailscanner
Exception `Gem::LoadError' at /usr/local/lib/ruby/site_ruby/1.8/rubygems.rb:827 - Could not find RubyGem activerecord-postgresql-adapter (>= 0)

Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- pg
WARNING: Nokogiri was built against LibXML version 2.7.3, but has dynamically loaded 2.7.6
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- ruby-debug
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- classtree
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- classtree
Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -- classtree
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31 - no such file to load -- methodsig
Exception `MissingSourceFile' at /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:38 - no such file to load -- methodsig
Exception `MissingSourceFile' at /usr/local/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:162 - no such file to load -- methodsig
=> Debugger enabled
worker=0 ready
Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)
Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn/http_response.rb:67 - Invalid argument
Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)
Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn/http_response.rb:67 - Invalid argument
Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)
Exception `Errno::EINVAL' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn/http_response.rb:67 - Invalid argument
Exception `Errno::EAGAIN' at /usr/local/lib/ruby/gems/1.8/gems/unicorn-0.95.3/lib/unicorn.rb:640 - Resource temporarily unavailable - accept(2)


I seriously have no idea went wrong - It ran all so smooth couple of days ago. What am I missing - there must be something plain wrong. I left the js files out of the public site for now - it isn't needed that much and its more important to serve the content for now. In case you are trying to get it.

Kind regards, John


On 04.02.2010, at 09:16, John-Paul Bader wrote:

> Hey guys,
> 
> 
> i recently updated several components of my website setup: 
> 
> varnish   2.0.5 -> 2.0.6
> nginx      7.4.2 -> 7.4.5
> unicorn  0.95.3 -> 0.96
> ruby18  to the latest patch level 
> and rails from 2.3.2
> 
> It all seemed to work great but recently I noticed that the website hangs loading. The weird thing is that it hangs because 2 or 3 javascript files take about a minute to finish and when they do they are incomplete as in cut of in the middle of the content. The nginx error log gives the following output every few seconds:
> 
> 2010/02/04 08:58:41 [error] 29389#0: *471 upstream timed out (60: Operation timed out) while reading upstream, client: 213.73.89.122, server: www.ccc.de, request: "GET /javascripts/jquery-1.3.2.min.js?1246657683 HTTP/1.1", upstream: "http://unix:/var/run/unicorn.sock:/javascripts/jquery-1.3.2.min.js?1246657683", host: "www.ccc.de"
> 
> I don't think its varnishs fault because i get Internal Server Errors (500) if I run curl on the machine against nginx, not at first though - several requests for that jquery file run just fine and then suddenly i get a 500 on a request with the content being cut of in the middle.
> 
> Next thing I did was configuring unicorn to listen on 213.73.89.122:9090 rather than on a socket so i could run curl directly against it and even after a fresh start of unicorn gives me:
> 
> 
> curl http://213.73.89.122:9090/javascripts/jquery-1.3.2.min.js
> (lots of javascript)…,rowspan:"rowSpan",tabindex:"tabIndHTTP/1.1 500 Internal Server Error
> 
> curl: (18) transfer closed with 16256 bytes remaining to read
> 
> I also downgraded back to 0.95.3 but that didn't change anything. So now after 8h of try and error I'm out of ideas and would be happy to hear some suggestions.
> 
> Kind regards, John
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
> 



More information about the mongrel-unicorn mailing list