From musadahmed36 at gmail.com Mon Apr 2 22:02:14 2012 From: musadahmed36 at gmail.com (Musad Ahmed Khairi) Date: Tue, 03 Apr 2012 00:02:14 +0200 Subject: Compliments, Message-ID: Compliments, I am Mr. Musad Ahmed Khairi a citizen of Estonia .I have in my bank the existence of a big amount of money that belongs to a customer, Sir. Rayan and who happen to have the same surname as yours .The fund is now without any claim because Sir. Rayan died in a deadly earthquake in China in 2008. I want your cooperation so as to make the bank send you the fund as the beneficiary and next of kin to the fund. 100% Trust please! This transaction will be of a great mutual assistance to us. Send me your reply of interest so that I will give you the details. Strictly send it to my private email account:send me your email address to send you details of this transaction. At the receipt of your reply, I will give you details of the transaction. And a copy of the Deposit certificate of the fund and also the incorporation certificate of the company that generated the fund. Best regards, MR. MUSAD AHMED KHAIRI. From saipem_america447 at consultant.com Wed Apr 4 14:57:32 2012 From: saipem_america447 at consultant.com (SAIPEM AMERICA INC) Date: Wed, 4 Apr 2012 20:27:32 +0530 Subject: SAIPEM AMERICA INC JOB OFFER (OFFSHORE) Message-ID: <442027.QLNBMXTM@consultant.com> Saipem America Inc. 15950 Park Row Houston, Texas 77084 USA OUR REF: SAP/078811/JO The Saipem America offshore Inc Intends to invite experienced Individuals/Expatriates or Consultancy firm who is Capable of rendering expertise services in various fields of Engineering, Construction and Installation, NDT, Geology, Project Management, Welding and Fabrication, Plant/Start-Up, Machine operators, Service Technicians, Maintenance, Mechanical/Technical/Design Engineering, Information Technology, Administrative, Commissioning and Medical Experts. Crew Managers And Assistant. Diving vessels and Riggers. The Saipem America Inc request all applicants from various countries to send detailed CV in accordance with the format indicated. The Company further encourages all applicants to forward their respective application to this email address (saipemamerica1010 at live.com) BENEFITS, PACKAGES AND ENTITLEMENT : Saipem America value their workers by recognizing their hard work and offering a competitive salary and benefit package that includes the following. Medical Dental Vision Basic Life Insurance Accidental Death & Dismemberment (AD&D) Disability (Short & Long Term) 401K Vacation Days Sick Pay Tuition Reimbursement Program Please note that application received after the deadline will not be considered. Similarly applications not submitted in accordance with the indicated format may also be rejected. The relevant reference number should be stated on left - hand corner of any mail being sent in relation to this Job application. We shall enter into correspondence only with Short listed applicants or expatriates. Sincerely Prof. Mike Collins Human Resources Manager From boss at airbladesoftware.com Thu Apr 5 07:21:01 2012 From: boss at airbladesoftware.com (Andrew Stewart) Date: Thu, 5 Apr 2012 09:21:01 +0200 Subject: Problem restarting Unicorn Message-ID: <1D1E83EF-886F-409D-B61E-22F7309C3690@airbladesoftware.com> Hello, I seem to be confused about how to restart Unicorn when deploying updates to a Rails app via Capistrano. I just had a problem where a bundled gem wasn't being seen by the Unicorn processes, even though app code I introduced at the same time was being served by the Unicorn processes. I had restarted Unicorn numerous times using `bundle exec cap deploy:restart` (see the code below), which I thought was the right way to do zero-downtime restarts, but my problem was only solved by doing a hard restart of Unicorn with `bundle exec cap deploy:hard_restart`. With each "normal" restart I tailed the stderr log to check everything was ok; everything appeared fine, with gems refreshed etc. Please could somebody explain what I'm doing wrong with my normal restarts? config/deploy.rb: set :unicorn_config, "#{current_path}/config/unicorn.rb" set :unicorn_pid, "#{shared_path}/pids/unicorn.pid" namespace :deploy do task :start, :roles => :app, :except => { :no_release => true } do run "cd #{current_path} && #{sudo_workaround} bundle exec unicorn -c #{unicorn_config} -E #{rails_env} -D" end task :stop, :roles => :app, :except => { :no_release => true } do run "#{sudo_workaround} kill -s QUIT `cat #{unicorn_pid}`" end task :hard_stop, :roles => :app, :except => { :no_release => true } do run "#{sudo_workaround} kill `cat #{unicorn_pid}`" end task :restart, :roles => :app, :except => { :no_release => true } do run "#{sudo_workaround} kill -s USR2 `cat #{unicorn_pid}`" end task :hard_restart, :roles => :app, :except => { :no_release => true } do hard_stop start end end # Works around Ubuntu changing the PATH when you use sudo. # sudo's path can find common commands like echo but not much else. def sudo_workaround "sudo env PATH=$PATH" end config/unicorn.rb: APP_PATH = '/var/www/apps/sparkle' worker_processes 4 user 'rails', 'rails' working_directory "#{APP_PATH}/current" listen "/tmp/unicorn_sparkle.sock", :backlog => 64 timeout 30 pid "#{APP_PATH}/shared/pids/unicorn.pid" stderr_path "#{APP_PATH}/shared/log/unicorn.stderr.log" stdout_path "#{APP_PATH}/shared/log/unicorn.stdout.log" preload_app true GC.respond_to?(:copy_on_write_friendly=) and GC.copy_on_write_friendly = true before_fork do |server, worker| defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! old_pid = "#{server.config[:pid]}.oldbin" if old_pid != server.pid begin sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU Process.kill(sig, File.read(old_pid).to_i) rescue Errno::ENOENT, Errno::ESRCH end end end after_fork do |server, worker| defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection end This is all with Unicorn 4.2.0 and Rails 3.0.11. Many thanks in advance, Andy Stewart ---- http://airbladesoftware.com From mguterl at gmail.com Thu Apr 5 11:34:43 2012 From: mguterl at gmail.com (Michael Guterl) Date: Thu, 5 Apr 2012 07:34:43 -0400 Subject: Problem restarting Unicorn In-Reply-To: <1D1E83EF-886F-409D-B61E-22F7309C3690@airbladesoftware.com> References: <1D1E83EF-886F-409D-B61E-22F7309C3690@airbladesoftware.com> Message-ID: On Thu, Apr 5, 2012 at 3:21 AM, Andrew Stewart wrote: > Hello, > > I seem to be confused about how to restart Unicorn when deploying updates to a Rails app via Capistrano. > > I just had a problem where a bundled gem wasn't being seen by the Unicorn processes, even though app code I introduced at the same time was being served by the Unicorn processes. ?I had restarted Unicorn numerous times using `bundle exec cap deploy:restart` (see the code below), which I thought was the right way to do zero-downtime restarts, but my problem was only solved by doing a hard restart of Unicorn with `bundle exec cap deploy:hard_restart`. > > With each "normal" restart I tailed the stderr log to check everything was ok; everything appeared fine, with gems refreshed etc. > > Please could somebody explain what I'm doing wrong with my normal restarts? > > config/deploy.rb: > > ? ?set :unicorn_config, "#{current_path}/config/unicorn.rb" > ? ?set :unicorn_pid, ? ?"#{shared_path}/pids/unicorn.pid" > > ? ?namespace :deploy do > ? ? ?task :start, :roles => :app, :except => { :no_release => true } do > ? ? ? ?run "cd #{current_path} && #{sudo_workaround} bundle exec unicorn -c #{unicorn_config} -E #{rails_env} -D" > ? ? ?end > ? ? ?task :stop, :roles => :app, :except => { :no_release => true } do > ? ? ? ?run "#{sudo_workaround} kill -s QUIT `cat #{unicorn_pid}`" > ? ? ?end > ? ? ?task :hard_stop, :roles => :app, :except => { :no_release => true } do > ? ? ? ?run "#{sudo_workaround} kill `cat #{unicorn_pid}`" > ? ? ?end > ? ? ?task :restart, :roles => :app, :except => { :no_release => true } do > ? ? ? ?run "#{sudo_workaround} kill -s USR2 `cat #{unicorn_pid}`" > ? ? ?end > ? ? ?task :hard_restart, :roles => :app, :except => { :no_release => true } do > ? ? ? ?hard_stop > ? ? ? ?start > ? ? ?end > ? ?end > > ? ?# Works around Ubuntu changing the PATH when you use sudo. > ? ?# sudo's path can find common commands like echo but not much else. > ? ?def sudo_workaround > ? ? ?"sudo env PATH=$PATH" > ? ?end > > > config/unicorn.rb: > > ? ?APP_PATH = '/var/www/apps/sparkle' > ? ?worker_processes 4 > ? ?user 'rails', 'rails' > ? ?working_directory "#{APP_PATH}/current" > ? ?listen "/tmp/unicorn_sparkle.sock", :backlog => 64 > ? ?timeout 30 > ? ?pid "#{APP_PATH}/shared/pids/unicorn.pid" > ? ?stderr_path "#{APP_PATH}/shared/log/unicorn.stderr.log" > ? ?stdout_path "#{APP_PATH}/shared/log/unicorn.stdout.log" > ? ?preload_app true > ? ?GC.respond_to?(:copy_on_write_friendly=) and GC.copy_on_write_friendly = true > > ? ?before_fork do |server, worker| > ? ? ?defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! > > ? ? ?old_pid = "#{server.config[:pid]}.oldbin" > ? ? ?if old_pid != server.pid > ? ? ? ?begin > ? ? ? ? ?sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU > ? ? ? ? ?Process.kill(sig, File.read(old_pid).to_i) > ? ? ? ?rescue Errno::ENOENT, Errno::ESRCH > ? ? ? ?end > ? ? ?end > ? ?end > > ? ?after_fork do |server, worker| > ? ? ?defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection > ? ?end > > This is all with Unicorn 4.2.0 and Rails 3.0.11. > > Many thanks in advance, > You probably want to check out the Sandbox documentation here: http://unicorn.bogomips.org/Sandbox.html The section that I think is most relevant is related to setting the START_CTX: Unicorn::HttpServer::START_CTX[0] = "/some/path/to/bin/unicorn" Hope that helps. Best, Michael Guterl From olgai.roiyaltiy at mail.ru Mon Apr 9 17:27:38 2012 From: olgai.roiyaltiy at mail.ru (=?Windows-1251?B?SGHrb+Pu4uD/IE9t92Vt7e7x8vw=?=) Date: Mon, 09 Apr 2012 21:27:38 +0400 Subject: =?Windows-1251?B?Q+5j8uDi6+Xt6OUgb/L35W3tb/Hy6CDn4CAxLekg6uIuIDIwMTI=?= Message-ID: <874253549.20120409766797@hosting.valutayan.com> ------------------------------------------------------------------ | C?C???????? ???E?H???? ?A ? KBA???? 2012 ???A | | ?????E??? B ?P?????, H??, H???!!! | | HA??????? ?O?E??! H??O????? ?????H????! COC???????? ? ?????A! | | ?HA??? ?AP????????? ?????E??? B ?A?O????? K??????!!! | ------------------------------------------------------------------ > C?????? ? 12 a?p??? ? 2012 ?. > ?. K?e? ? ??. ??????o?????, 1/3, o???? "?a??????" > ?e?.: (044) 331-?4-14, 592-75-6? > H?? ???e?e??? ? ?op?? ??a??a ? ?op???e ?a?o??e??? ?a?o?o??? ????a???x * o???a?e????? ? ?p???? pe??????? (?. 201.1 c?. 201 HK?) * ???????e?? ???o????? ????????x (???, ??c????? ?e??, ??a????????? ???e??, ?o????????e ??e?a ? ?p.). * pe?????????? ?o ?a???e ?c?ap???, ?o ?c?? ?????????x c ??o???????, ?a?o?o??? ????a???? (?. 198.6 c?.198 H??, ??a???? ?365 ??e??, ?o???e???, ?????ep??????? ?a?? ?o??????? ?a?????o?, ??o?.). * ?co???????? o?op?????? ?c??????? ???o?o??? ???????o? (?. 15 ?op???a ??????e??? ?a?o????? ????a??o?). * ??c???? ?op?????????? ??????c??????? ? ??o???????? ?????a?????, ??a???? ??p?????????? ?a??????? ??????e????? ? ???o?????? ?p????? ?p? ?o??p??? ?o??p?? (?c???) ???o ?????e??? ?e??. * ??o??e?? ?opp????????? ?a??????? ????a??????? ?p? ?o???a?? ?o????? o? ????a????????? ??C. * ??a?c? H?C-?op?????????? ?p? c???a??? ???o??e??oc??? (?o??p???, ?e??????) * ?a????????? ?H?? (o????a???? ???c???????? 2012 ?o?a). ??C-????????????? * ?o???o? ????c?? ?ac????? ?o??????????? ?o???????????? ? ??o???????? ?o?a???????; * ?e??????????? ?op??????????: ?o????? ?o????? o? ????a????????? H??, ?????a? ????c??; * ??a??? H??-??p?????????? ?p? c??ca??? ?a?o??e???????; * o?????????? ??occ??????????? ?a?o?????? ?p????a ?o oc?o???? ?pe?????? (? ?o? ??c?? ?o ?e?????? a??o?o?????). ??a????e???? ????a??? ? ??e??? o??a??? ?H?. E????? ??e??? ???o????? ????????x (?P??) * 2012 ?o? - ???a?o ?acc???? ?e??c?p???? ?????o??? ????????x ? ?P??; * ??x????? p???c?????? ?a?o?o??? ?a??a???? ?a?????? ?oppe???????? ? ??? ? ??H?; * c?o?? ?e??c?p???? ? EPHH. O??e??????????? ?o??a?????? ? ?o???a?????; P?c??e??????? ??xo??o?o? H?C * ????p? ? ?c????, ?p???a????????? ??? o????????????? ?c?o???o????? ? o??a?a???x ? ?e o?????e??? ??C o?ep?????. * ?c?o???? c?e?c???, ???o????e??e ???o???? ? ?e???????x o?ep?????: ??o, ?o??? ? ?a? ????e? ?c??e??????? ?e?ep????? ?a?o?o???? ?pe???a; o??a?e??? ??o?o? ?ep???????? ?o ???o???? cp??????? ? ?e??apa??? ?o H?C ? ?p??o?e??? ?7. * ?o??o ?? ??e??? pac??e??????? ??x?????o? H?C ?a??????? ?a??????? ????a?e?????? (?p???a??e ?c?o??o? ?oc?a???). H?a?c? ?c?o????????? ?. 198.5 c?. 198 H??. ????a?????? ??C-?????? ?c?p??????? o???o? ?p????x ????o??? ? ????a????? ??? ? ??o??????? pac????: ??o ??????e?? * ?oc?????? ?a???c????? ?HC?. * ???????e o????? ?p? ?ac???????? ?o???a?????? ? ?a? ?x ???pa???? ( ?e??a??????? pac???????? ?????a?????? ? pe?????? ? ?p????e??? 5). * ??pa?? ? ?e?? ?p? ca?o??????????? o???p?????? o???o?: ??e??? ?? ???o ?x ???a??????? > ???O? H? ?P????? 1. ?epe??? ?????a????? ?? ?c?????? ?o??? ????ap???? ? ??o????. ??o ?????e???? ?p? ?e??????? 2. ??c??????? a?a??? ???o?????? ?p??o????? o ??o??? ?o????? ? ?po??x p?c?????, ? ?o? ??c?e: * c??c???? ?????o????? ? ?p?????????? ?a?o??e???????; * ????p?? ???ap??, ???e?e??? c?o?????? ? ??o??e ?o?????? ??pp?????????; * ??e????? ??e????? ? ?pe?????????: ? ?a?o? c?p??? ?e???????? ??????ee ??pa???? ?o???. 3. ?p???????? o? ??o???????? ? ?e??????: c???????? ?? ?????e ????o????? c ?e?????????? 4. Pac???????? ??a??oo???????? c ????e?????-???????????: ?a? ?a??????? ? ?a ?c?o????? ????x ????????o? 5. ?ec??????? ?a????? ?o ?c?? pa??p???: o? ???a???? ?p??????? ?o ???a???? ? ????c?????. ?oxo??, pac????, ?a?o????? o?????e?????? ? ?a???o??? ??e???: ????o???? a????? ?o?o, ??o ? ?o??? ?o????ae?. 6. ???p???? c o??o????? ????a??: * ?a???? 10-?? ?po?e?????? ?????a ?e?o????? ?ac????? ? 2012 ?o??. ??o ?p??????? ?a ?a??, ?a? ???c?? ????e???? ?a??????? p??????, ?o??????? ?o ??c?????? ?a 01.04.2011 ?.; * ?ex???e???? o???????a??? ? c?c???? p?c?????. ?o???e??a????? o??c??????? ?ac????? ?a ??x????????????; * ??p???? ?a?p?? ?a ???o?? ape????????? oc?????? ?pe???? ?p? ??c??c???? 10%-?o ?????a; * o?????????? ???o??????????? ??ep???? ?o ??????a??? ???o???? cpe????, ?o ????p?? ?c???o????? ??????a??o??a? ?????o???. 7. ?p?????? o??p????: * ?p???????? ?K?, ?o????? ?o??? ?o?????? ?a ???o?o??? ??e? a?e????? o??p????; * a??????? o?ep???? c ??????a??, ??ep?a??? H??? ?p? ?????e??? ap??????????? o??e???; * ?pe??? a????o????: ???e? ?? ???e??? ??c? ? ?o ?a??? ????e. 8. ??e????? ?o ??e??e??o?????????? ????e???????: * ??pe???? ???c???? p????? ?o ?c??p??? ?????a???? ???o??????????; * ??pc???? pa????? o? o?e?????, ?e c???a???? c ?o???c??????? ????????o????: ????o ?? o??a???? ? ?a??????? ??e?e; * ?o????? ? ?po?a?? ?a????: ?a? ???????e????? ???e?e??? ?o??? ???o?? ? ?a?o????? ????e. 9. Ko?a??????????? p?c????: * ?????e???????? ??op??????, ? ?o? ??c?? ?p? ??a???o????????? c ?p???????????????; * ???a???????, ?o?o??? ?e? ? ?K? ? ???op?? ??p???????e?? ?H??; * ?o?????p??????? pac???? ? ???c??????? ?a???e ? ????c? ???????e?????? ? ?????o???? ??e?a; * c???????? ??p????e ?a????? ?o ??????p??????? ?a???e; * ???o???o????? ??e???????? ???a???e???, ????e???? ????o???. > H??? 1. ?ap??a???? o??e????? ? 2012 ?o?.?? ??????e ?a ?o?a?a???? ? ???o?o????????? ?a???????? ??a??. K???? ?ox?? ???a?a?? ?e??p? ?o? c????? H??? 15%? B??????a ???c???????? ??pa?????? ??a??, ???a?????? E??. ?o?e? ?? ?ap??a?? ?a ?pe??p????? ???? ?????e ?????a?????? 2. B????? ??a??? ? ?a????????????? H???: * c????a 15%-17% ?o??? K????? ?c????????? ???o?o????????? ?o????? ?e ?o o??o????? ?e??? ?a????. K?o ????e? ?o?a?? ????a????? o ?oxo??? ?a 2011 ?.? * ?ce o ?e??a????????? ?oxo??? ??????a??. K?o o????? ?o?a?? ?e??a?????, a ??o ???e? ??a?o? O?????????????? ?a ?e??e???????????. K?? ?a??????? ?????p????. * cp??? ???a?? H???: ?e???? ??a???, ?a?? ?e ????a??? ??x??!!! A ????e ???a??? * a????? ????a? ?e o????a???? H???: ???ap??, ?e???e??? ????o???????????? ?o????, ???o?? ?a ?e????e ?o?????????, p?c???? ?a o???????. ?o??????a????? ???p?????? ?a??x ????a?. * ?ce o ?????a???? ???a??o??? ?o?o?? ?o ? ?oc?? ???e?e??? ? ?o???c. K?? ???a???? c???? ?o???? ? ?op?? 1-??; * ???e?????? ?o ??p?? 1-?? ?a: oco???????? ??p?????? ???o????? ??????o??????? ??x????: ???e??????????? ?a o?????, ?a? o????? ?c?p?????, a ?a??? o????? ?c???????? ?e c????? 3. HO???!!! ??a???a pa?o?? c ??:c 01.01.2012 ?. ???c????? ?a??e? XIV?a?o?????? ???e???. ?a? ??op???? ?????o????????? c ?? ?e? p?c?o? ??????c????? H??? ? E?B. C ??-e????????? ????x ?p??? ?o??? pa??????? ?co???????? ?pe??? ?????c??a ? ?????? ? ??-e??????o?. C ?a???? ?????a?? ??a???????o? ????o?o ?a???? ????o p???????? H??o????? ??c??. 4. Bce o ???a????????? c ??e?o? ?oc?e???? ???e????? ? ??c?e???? pa?????????: * B??????? !!! C??????? ????????c? c 01.01.12 ?. * K?? o??p???? ?a????? ? ?o?????????? c ??-????e??????? e?????o ?a?o?a? K????? ?o??????a?? ?o???ep???? c??o???? ? ?o????????? o??e?? ???a???p???????? ??oc?o???????? C??o???? ?o???? ??? ?e???? c???, ????a???? ? c?.140.1.7. ?????e??a????? o????????? ?o?a???p??????? pac?????: ?a???? p??? ?p?????? ? ?o???e???, c???? ???a???p???? c ?e??e???o???? ??e??p?????, ??e??p????? ???e??, ???a?? ??o???a??? ? ?? ? ????o???? ??c??. 5. ????a?a ??co??? ?o ?e????o???????????, ??co??? ?o ?e?e???????? ? ?o?a?: * ?o???e???????? o????????? ? pa????, ?????o? ???a?c?p?????? ?o??o? ? ??op?????? ?a????, * ?a??? o????? ??????e? ??c?? ??????a??, ? ?a? ?x ?c??a????? BH??????!!! C??? o?pe???????? ?o ?o??? ??a???a?. 6. B?????!!! ???e????? ? ?o????? ?a????e??? ?C?: ???e????? ? o??e??o??? ?o EC?: Ha???????? E?? ?a ?o?o??? ?o ?e????o??????????? p??o?????? ? ???a?????: ?a? ??pe??????? c 33,2% ?a 8,41%: ?o??? ????p????? ?21-5 ? ?????e ?e?c?o????? ????a. 7. ?co???????? ????c?e??? ? ???a?? ??????o c???a?????? ???oc?: ?a?a ???o?o????????? ?CB, 8. ???e???? ????a?, ?a ?o??p?? ????c ?e ?a??c??????; ??o???????? ?a?o??????????? 9. ?o??a????????? ?o ???o????? ???. Ka? pa???????? ???oc, ?c?? ???o?? ???o??????? ?a 10. ??o???e??? ?ec?o????? ?ec????; cpo?? ???a?? ?CB, ??????a ??a????, ?p???????????? ????a? ? ?a???a????? ??p?e 11. O?o???????? ?o????????? ?????a????? ??????o??? ??a???????a?? ????o?o ???o?a. B??????!!! ?co???????? c?c???????? ????ap???? ?a 1 ??ap?a? 2012 ?. O?o??a???? ?a?o?o??? ?o?c????a??? ??? ??????????o? e???o?o ?a?o?a: ?a? ?op??p??? ?ac???? ??a?e?????? ?a???a ?a ?p????? ?p? ???o?? c e??????o?? ?a? e??????? ??a???? ? a?e??? ???e?e???? ? ?p. ?o?????. > ?OK??????? * KO????? ?????a ???e?????? ? ?????p?-??????????? ?o ?o?p???? ?a???o?????????, ??e?? ??a????????? o??? ????e????? a????op???? ?po?e???. ???a?? ?????a?? ?o ???p???? ?????o????????? c 2000 ?o?a. Cep???????? ?C?? ???, ?C??. * ??o????? Ce???? H??o?????? - ??c?e??-???????????, a??o? ??o?o????????? ????e? ? ??????a??? ?o ?????o????????? (H??) ? ???, ??ee? ????o?????? o??? ???o?? ? ?HA?, p??a???? ?a?e?? "??x?a??e???". * ?A???B? O????? ??????a - ?pa????????? ?????o?, ??c????-??????????? ?o ?o?????? ???o??????????? ? ?p???. ??pe??o? A?????p???? ??p?? ????C??????? KA?????". O??? p??o?? ? ?p???????? ?o??????? ????????a??? ??x?a??????. ?e?a???? ??????e??? ?a?o?a?? ? ???ae? ce???a?? c 2000 ?o?a. > ??C??????? * ?e??c?????? c 9:30 ?o 9:30 * H??a?? ? 9:30 > ?????O??? * 835.00 ?p?. ? ?a o??o?o ??ac????a. * B c???????? ?x????: o??????e, ???c????a??? ??????????o?????? c?e?????????, o????? ?a ???ep??????? Ba? ???po??, ????o??? ?o????????? ?o???e??o? ? c??????? c ?o?e??e? ???op??????, CD-??c? c ?a?ep??????, ?o?????o??? ???e??? c ????e????-????????????, ??p????: ??e? ? p????????, ?o?e-?p????, ?????o?a??. ? ????? ????? ????????? ??????? ??????? ??????????? ????? ??????? ???????? ????? ??????? ????? ??????????? ???????? ????????? ?????? ????????? ?????? ??????? ?????? ????????? ?????? ???????? ??????????? > ??pe?????????????? ?o??o ?o: (044) ?31-64-14, 592-75-6? From boss at airbladesoftware.com Tue Apr 10 08:52:57 2012 From: boss at airbladesoftware.com (Andrew Stewart) Date: Tue, 10 Apr 2012 10:52:57 +0200 Subject: Problem restarting Unicorn In-Reply-To: References: <1D1E83EF-886F-409D-B61E-22F7309C3690@airbladesoftware.com> Message-ID: <6C172DB9-D159-48A5-AB7C-4F41C2190AF6@airbladesoftware.com> On 5 Apr 2012, at 13:34, Michael Guterl wrote: > On Thu, Apr 5, 2012 at 3:21 AM, Andrew Stewart wrote: >> I just had a problem where a bundled gem wasn't being seen by the Unicorn processes, even though app code I introduced at the same time was being served by the Unicorn processes. I had restarted Unicorn numerous times using `bundle exec cap deploy:restart` (see the code below), which I thought was the right way to do zero-downtime restarts, but my problem was only solved by doing a hard restart of Unicorn with `bundle exec cap deploy:hard_restart`. > > You probably want to check out the Sandbox documentation here: > http://unicorn.bogomips.org/Sandbox.html Thanks for the pointer to the Sandbox documentation, the one page on the Unicorn website I hadn't read. The section on BUNDLE_GEMFILE for Capistrano users was what I needed. Yours, Andy Stewart From yan.nuo at gmail.com Wed Apr 11 08:07:59 2012 From: yan.nuo at gmail.com (Nuo Yan) Date: Wed, 11 Apr 2012 01:07:59 -0700 Subject: Request-URI Too Long from mongrel-unicorn Message-ID: In my understanding, the mongrel request uri limit is 12k. My request uri is certainly not that long (about 1445 bytes), but for some reason I'm getting 414 Request-URI Too Long from the server. In my experiment, if I shrink the request URI to 1016 bytes, it works. Getting this error on all GET, POST, PUT, and DELETE requests. There is no cookie being sent and the number of headers and their length are all very small. Any idea why? From normalperson at yhbt.net Wed Apr 11 20:30:54 2012 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 11 Apr 2012 20:30:54 +0000 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: References: Message-ID: <20120411203054.GB22267@dcvr.yhbt.net> Nuo Yan wrote: > In my understanding, the mongrel request uri limit is 12k. Correct, that's the entire REQUEST_URI, though: REQUEST_PATH?QUERY_STRING#FRAGMENT > My request uri is certainly not that long (about 1445 bytes), but for > some reason I'm getting 414 Request-URI Too Long from the server. Perhaps one of your URI components (perhaps REQUEST_PATH) is too long? The current limits are as follows (in ext/unicorn_http/global_variables.h) DEF_MAX_LENGTH(REQUEST_URI, 1024 * 12); DEF_MAX_LENGTH(FRAGMENT, 1024); /* Don't know if this length is specified somewhere or not */ DEF_MAX_LENGTH(REQUEST_PATH, 1024); DEF_MAX_LENGTH(QUERY_STRING, (1024 * 10)); I will consider upping REQUEST_PATH to 4096 (and REQUEST_URI to 15K) since 4096 is a common filesystem PATH_MAX on modern systems. Would anybody object to this? Given we already allow huge headers and Ruby uses quite a bit of memory, I don't think a potential extra 3K will negatively impact anybody. From yan.nuo at gmail.com Wed Apr 11 21:44:09 2012 From: yan.nuo at gmail.com (Nuo Yan) Date: Wed, 11 Apr 2012 14:44:09 -0700 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: <20120411203054.GB22267@dcvr.yhbt.net> References: <20120411203054.GB22267@dcvr.yhbt.net> Message-ID: Hi Eric, On Apr 11, 2012, at 1:30 PM, Eric Wong wrote: > > I will consider upping REQUEST_PATH to 4096 (and REQUEST_URI to 15K) > since 4096 is a common filesystem PATH_MAX on modern systems. > > Would anybody object to this? Given we already allow huge headers and > Ruby uses quite a bit of memory, I don't think a potential extra 3K > will negatively impact anybody. Lately I figured REQUEST_PATH should probably be the reason as well. It would be great if you bump that up to 4096 - so that we could continue to use released versions instead of having to patch on our own. 4k makes sense to me. When do you think you can cut a gem on this? Also, what do you think about making these values configurable in the conf file? I understand these were hard coded by design to protect the server. However, I think it would be nice if it defaults to these small values and can be configured flexibly. Thanks a lot, Nuo From lawrence.pit at gmail.com Wed Apr 11 23:43:03 2012 From: lawrence.pit at gmail.com (Lawrence Pit) Date: Thu, 12 Apr 2012 09:43:03 +1000 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: <20120411203054.GB22267@dcvr.yhbt.net> References: <20120411203054.GB22267@dcvr.yhbt.net> Message-ID: <1437AF4F-9109-4B9B-B30C-C364D16C61B0@gmail.com> Hi Eric & All, > DEF_MAX_LENGTH(REQUEST_URI, 1024 * 12); > DEF_MAX_LENGTH(FRAGMENT, 1024); /* Don't know if this length is specified somewhere or not */ > DEF_MAX_LENGTH(REQUEST_PATH, 1024); > DEF_MAX_LENGTH(QUERY_STRING, (1024 * 10)); > > I will consider upping REQUEST_PATH to 4096 (and REQUEST_URI to 15K) > since 4096 is a common filesystem PATH_MAX on modern systems. IE browsers up to and including IEv8 have a max URL length of 2,083 characters : http://support.microsoft.com/kb/208427. IE9 seems to allow up to 5K last time I checked. There may be other clients (ruby http clients, bots, traffic analysers, database clients with limited column spaces to hold URLs, etc.) that can't handle long URLs if you take it too far. So practically it's advisable to keep your URLs within limits. Having said that, the HTTP/1.1 spec state that servers SHOULD be able to handle URIs of unbounded length. So yes, unicorn should allow this if one chooses. > Given we already allow huge headers and > Ruby uses quite a bit of memory, I don't think a potential extra 3K > will negatively impact anybody. Should there be a limit at all in unicorn? Should it not be assumed this is configured at the webserver level, like: http://wiki.nginx.org/NginxHttpCoreModule#large_client_header_buffers nginx currently uses a default of 4 buffers of 8K. (i.e. a path_max of 8K) The webserver takes care of bombing out with a 414 before it even reaches unicorn. If traffic does reach unicorn then unicorn should be able to handle whatever length it's given. PS. I'm fine with 4096 as well of course, I keep to the 2K limit anyways ;-) Cheers, Lawrence From normalperson at yhbt.net Thu Apr 12 03:45:46 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 03:45:46 +0000 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: <1437AF4F-9109-4B9B-B30C-C364D16C61B0@gmail.com> References: <20120411203054.GB22267@dcvr.yhbt.net> <1437AF4F-9109-4B9B-B30C-C364D16C61B0@gmail.com> Message-ID: <20120412034546.GA19006@dcvr.yhbt.net> Lawrence Pit wrote: > IE browsers up to and including IEv8 have a max URL length of 2,083 > characters : http://support.microsoft.com/kb/208427. Ah, that probably explains why MAX_URI_LENGTH=2083 in WEBrick (lib/webrick/httprequest.rb). > There may be other clients (ruby http clients, bots, traffic > analysers, database clients with limited column spaces to hold URLs, > etc.) that can't handle long URLs if you take it too far. So > practically it's advisable to keep your URLs within limits. Agreed. I'm really hoping no apps out there depend on the Mongrel/unicorn internal limits to enforce database column length :) > Should there be a limit at all in unicorn? Should it not be assumed > this is configured at the webserver level, like: > > http://wiki.nginx.org/NginxHttpCoreModule#large_client_header_buffers There should be a limit in unicorn, it's cheap to enforce and there could be corner cases (nginx bugs, internal security probes) where it's helpful. The unicorn parser is also used by servers (Rainbows!) that expect untrusted/malicious clients without nginx to protect it. From normalperson at yhbt.net Thu Apr 12 03:52:05 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 03:52:05 +0000 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: References: <20120411203054.GB22267@dcvr.yhbt.net> Message-ID: <20120412035205.GB19006@dcvr.yhbt.net> Nuo Yan wrote: > On Apr 11, 2012, at 1:30 PM, Eric Wong wrote: > > I will consider upping REQUEST_PATH to 4096 (and REQUEST_URI to 15K) > > since 4096 is a common filesystem PATH_MAX on modern systems. > > 4k makes sense to me. When do you think you can cut a gem on this? I might wait a week for others to chime in before making an official release. However, packaging your own gem should be relatively easy and it is documented in the HACKING file. > Also, what do you think about making these values configurable in the > conf file? I understand these were hard coded by design to protect the > server. However, I think it would be nice if it defaults to these > small values and can be configured flexibly. I'm against adding too many configuration variables as it's difficult to support (needs documentation, testing, including testing for corner cases, error handling, etc...). Too many exposed configuration values scares off new users. Given how few people have an issue with these values, I don't think it's worth the effort. From normalperson at yhbt.net Thu Apr 12 08:18:31 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 08:18:31 +0000 Subject: Request-URI Too Long from mongrel-unicorn In-Reply-To: <20120412034546.GA19006@dcvr.yhbt.net> References: <20120411203054.GB22267@dcvr.yhbt.net> <1437AF4F-9109-4B9B-B30C-C364D16C61B0@gmail.com> <20120412034546.GA19006@dcvr.yhbt.net> Message-ID: <20120412081831.GA968@dcvr.yhbt.net> Eric Wong wrote: > Lawrence Pit wrote: > > Should there be a limit at all in unicorn? Should it not be assumed > > this is configured at the webserver level, like: > > > > http://wiki.nginx.org/NginxHttpCoreModule#large_client_header_buffers > > There should be a limit in unicorn, it's cheap to enforce and there > could be corner cases (nginx bugs, internal security probes) where it's > helpful. The unicorn parser is also used by servers (Rainbows!) that > expect untrusted/malicious clients without nginx to protect it. On the other hand, the _granularity_ of limits may unnecessary. There is already a 112K limit on the overall header size (which IMHO is really huge). However, this 112K overall limit is tunable on Rainbows! because Rainbows! is designed to handle hundreds/thousands of clients in one process. From paddor at gmail.com Thu Apr 12 17:36:25 2012 From: paddor at gmail.com (paddor) Date: Thu, 12 Apr 2012 19:36:25 +0200 Subject: Background jobs with #fork Message-ID: Hi I've migrated from Passenger to Unicorn about a week ago. It's great. Great transparency and management, thanks for this great software! A few of my Rails applications start background jobs using Kernel#fork. Of course, the ActiveRecord connections are closed and reopened again in the parent and child processes. The child process also does its job. Unfortunately, it seems that the parent (a Unicorn worker) waits for the child (background job) to finish before serving any new requests. Process.detach is done in the child. Process.setsid is not done. The child's STDOUT, STDERR and the Rails logger are redirected to their own files right after forking. Software used: ruby 1.9.1p376 Rubygems 1.8.17 Linux 2.6.16.60-0.21-smp (SUSE 10.2) unicorn 4.2.1 nginx 0.8.53 The problem persists even when multiple workers are started. And the problem was not present in the old setup with Passenger. My question: Does Unicorn somehow check/wait for child processes forked by the worker processes? Thanks in advance for your help. -- paddor From normalperson at yhbt.net Thu Apr 12 20:39:56 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 20:39:56 +0000 Subject: Background jobs with #fork In-Reply-To: References: Message-ID: <20120412203956.GA8980@dcvr.yhbt.net> paddor wrote: > Hi > > I've migrated from Passenger to Unicorn about a week ago. It's great. > Great transparency and management, thanks for this great software! :> > A few of my Rails applications start background jobs using > Kernel#fork. Of course, the ActiveRecord connections are closed and > reopened again in the parent and child processes. The child process > also does its job. OK, that's good. > Unfortunately, it seems that the parent (a Unicorn worker) waits for > the child (background job) to finish before serving any new requests. > Process.detach is done in the child. Process.setsid is not done. The > child's STDOUT, STDERR and the Rails logger are redirected to their > own files right after forking. So you're only calling fork and not exec (or system/popen, right?) It may be the case that the client socket is kept alive in the background process. The client socket has the close-on-exec flag (FD_CLOEXEC) set, but there's no close-on-fork flag, so you might have to find + close it yourself. Here's is a nasty workaround for the child process: ObjectSpace.each_object(Kgio::Socket) do |io| io.close unless io.closed? end > The problem persists even when multiple workers are started. And the > problem was not present in the old setup with Passenger. > > My question: Does Unicorn somehow check/wait for child processes > forked by the worker processes? Unicorn workers do not explicitly wait on child processes themselves, unicorn workers set: trap(:CHLD,"DEFAULT") after forking, even (the unicorn master must handle SIGCHLD, of course) The difference between nginx+unicorn and Passenger is probabably: nginx relies on unicorn generating an EOF to signal the end-of-response (nginx <-> unicorn uses HTTP/1.0), this I'm sure about. I think Passenger uses a protocol which can signal the end-of-request inline without relying on an EOF on the socket (Hongli can correct me on this if I'm wrong). However, nginx can still forward subsequent requests to the same unicorn (even the same unicorn worker), because as far as the unicorn worker is concerned (but not the OS), it's done with the original request. It's just the original request (perhaps the original client) is stuck waiting for the background process to finish. I can probably writeup a better explanation (perhaps on the usp.ruby (Unix Systems Programming for Ruby mailing list) if this doesn't make sense. From hongli at phusion.nl Thu Apr 12 21:15:54 2012 From: hongli at phusion.nl (Hongli Lai) Date: Thu, 12 Apr 2012 23:15:54 +0200 Subject: Background jobs with #fork In-Reply-To: <20120412203956.GA8980@dcvr.yhbt.net> References: <20120412203956.GA8980@dcvr.yhbt.net> Message-ID: On Thu, Apr 12, 2012 at 10:39 PM, Eric Wong wrote: >> The problem persists even when multiple workers are started. And the >> problem was not present in the old setup with Passenger. >> >> My question: Does Unicorn somehow check/wait for child processes >> forked by the worker processes? > > Unicorn workers do not explicitly wait on child processes themselves, > unicorn workers set: trap(:CHLD,"DEFAULT") after forking, even (the > unicorn master must handle SIGCHLD, of course) > > The difference between nginx+unicorn and Passenger is probabably: nginx > relies on unicorn generating an EOF to signal the end-of-response (nginx > <-> unicorn uses HTTP/1.0), this I'm sure about. ?I think Passenger uses a > protocol which can signal the end-of-request inline without relying on > an EOF on the socket (Hongli can correct me on this if I'm wrong). We don't. We call #close_write on the socket to prevent this problem. #close_write calls shutdown(fd, SHUT_WR) which isn't affected by the number of processes that have inherited the socket handle. -- Phusion | Ruby & Rails deployment, scaling and tuning solutions Web: http://www.phusion.nl/ E-mail: info at phusion.nl Chamber of commerce no: 08173483 (The Netherlands) From normalperson at yhbt.net Thu Apr 12 22:10:22 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 22:10:22 +0000 Subject: Background jobs with #fork In-Reply-To: References: <20120412203956.GA8980@dcvr.yhbt.net> Message-ID: <20120412221022.GA20640@dcvr.yhbt.net> Hongli Lai wrote: > On Thu, Apr 12, 2012 at 10:39 PM, Eric Wong wrote: > > paddor wrote: > >> The problem persists even when multiple workers are started. And the > >> problem was not present in the old setup with Passenger. > > > > The difference between nginx+unicorn and Passenger is probabably: nginx > > relies on unicorn generating an EOF to signal the end-of-response (nginx > > <-> unicorn uses HTTP/1.0), this I'm sure about. ?I think Passenger uses a > > protocol which can signal the end-of-request inline without relying on > > an EOF on the socket (Hongli can correct me on this if I'm wrong). > > We don't. We call #close_write on the socket to prevent this problem. > #close_write calls shutdown(fd, SHUT_WR) which isn't affected by the > number of processes that have inherited the socket handle. Ah, thanks for that clarification. It's an extra syscall for an uncommon case, but I doubt most apps will notice the hit... Might as well shutdown(SHUT_RDWR), too: diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index ede6264..f942e2f 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -536,6 +536,7 @@ class Unicorn::HttpServer end @request.headers? or headers = nil http_response_write(client, status, headers, body) + client.shutdown # in case of fork() in Rack app client.close # flush and uncork socket immediately, no keepalive rescue => e handle_error(client, e) From paddor at gmail.com Thu Apr 12 22:41:03 2012 From: paddor at gmail.com (Patrik Wenger) Date: Fri, 13 Apr 2012 00:41:03 +0200 Subject: Background jobs with #fork In-Reply-To: <20120412203956.GA8980@dcvr.yhbt.net> References: <20120412203956.GA8980@dcvr.yhbt.net> Message-ID: Thanks for the answers. > So you're only calling fork and not exec (or system/popen, right?) It > may be the case that the client socket is kept alive in the background > process. Yes, I'm only calling Kernel#fork. @Eric, your guess makes sense to me. > The client socket has the close-on-exec flag (FD_CLOEXEC) set, but > there's no close-on-fork flag, so you might have to find + close it > yourself. Here's is a nasty workaround for the child process: > > ObjectSpace.each_object(Kgio::Socket) do |io| > io.close unless io.closed? > end Isn't there another way to retrieve the right socket? Here some additional info that might bring some clarification. Another action in the same controller which does about the same regarding the background job works (check.run!). The only differences I see are: 1) it's called via AJAX 2) the response is nothing (render :nothing => true) instead of a redirect (redirect_to checks_path) I reckon the second difference kind of confirms Eric's guess, as the client socket probably isn't considered anymore with render :nothing => true. > However, nginx can still forward subsequent requests to the same unicorn > (even the same unicorn worker), because as far as the unicorn worker is > concerned (but not the OS), it's done with the original request. It's > just the original request (perhaps the original client) is stuck > waiting for the background process to finish. > > I can probably writeup a better explanation (perhaps on the usp.ruby > (Unix Systems Programming for Ruby mailing list) if this doesn't make > sense. Yeah I don't really understand this part. The "hanging" Unicorn worker can read another request because the client socket wasn't closed because it's still open in the child process? I would appreciate a better explanation, thank you. From normalperson at yhbt.net Thu Apr 12 23:04:57 2012 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 12 Apr 2012 23:04:57 +0000 Subject: Background jobs with #fork In-Reply-To: References: <20120412203956.GA8980@dcvr.yhbt.net> Message-ID: <20120412230457.GC22276@dcvr.yhbt.net> Patrik Wenger wrote: > Thanks for the answers. No problem. > Isn't there another way to retrieve the right socket? Actually, I think my proposed patch (in reply to Hongli) at http://mid.gmane.org/20120412221022.GA20640 at dcvr.yhbt.net should fix your issue. > > I can probably writeup a better explanation (perhaps on the usp.ruby > > (Unix Systems Programming for Ruby mailing list) if this doesn't make > > sense. > Yeah I don't really understand this part. The "hanging" Unicorn worker > can read another request because the client socket wasn't closed > because it's still open in the child process? I would appreciate a > better explanation, thank you. Basically, fork() has a similar effect as dup() in that it creates multiple references to the same kernel object (the client socket). close() basically lowers the refcount of a kernel object, when the refcount is zero, resources inside the kernel are freed. When the refcount of a kernel object reaches zero, a shutdown(SHUT_RDWR) is implied. This works for 99% of Rack apps since they don't fork() nor call dup() on the client socket, so refcount==1 when unicorn calls close(), leading to unicorn setting refcount=0 upon close() => everything is freed. However, since the client socket increments refcount via fork(), close() in the parent (unicorn worker) no longer implies shutdown(SHUT_RDWR). parent timeline | child timeline ------------------------------------------------------------------ | accept() -> sockfd created | (child doesn't exist, yet) sockfd.refcount == 1 | | fork() | child exists, now | sockfd is shared by both processes now: sockfd.refcount == 2 if either the child or parent forks again: sockfd.recount += 1 | close() => sockfd.recount -= 1 | child continues running since sockfd.refcount == 1 at this point, the socket is still considerd "alive" by the kernel. If the child calls close() (or exits), sockfd.refcount is decremented again (and now reaches zero). Now, to write this as a commit message :> From paddor at gmail.com Fri Apr 13 01:04:24 2012 From: paddor at gmail.com (Patrik Wenger) Date: Fri, 13 Apr 2012 03:04:24 +0200 Subject: Background jobs with #fork In-Reply-To: <20120412230457.GC22276@dcvr.yhbt.net> References: <20120412203956.GA8980@dcvr.yhbt.net> <20120412230457.GC22276@dcvr.yhbt.net> Message-ID: > Actually, I think my proposed patch (in reply to Hongli) at > http://mid.gmane.org/20120412221022.GA20640 at dcvr.yhbt.net > should fix your issue. Oh, this sounds great! I see it's just one line. I'll try adding this line and see if it fixes my problem. Thanks already! > Basically, fork() has a similar effect as dup() in that it creates > multiple references to the same kernel object (the client socket). > > close() basically lowers the refcount of a kernel object, when the > refcount is zero, resources inside the kernel are freed. When > the refcount of a kernel object reaches zero, a shutdown(SHUT_RDWR) > is implied. > > This works for 99% of Rack apps since they don't fork() nor call dup() > on the client socket, so refcount==1 when unicorn calls close(), leading > to unicorn setting refcount=0 upon close() => everything is freed. > > However, since the client socket increments refcount via fork(), > close() in the parent (unicorn worker) no longer implies > shutdown(SHUT_RDWR). I don't know much about system calls (honestly, never heard of shutdown() before) but I think I understand. The kernel then doesn't fee the allocated client socket and thus Unicorn (or Nginx? That's the other end of the client socket, I think...) thinks there's still more to come. From normalperson at yhbt.net Fri Apr 13 02:10:54 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 13 Apr 2012 02:10:54 +0000 Subject: Background jobs with #fork In-Reply-To: References: <20120412203956.GA8980@dcvr.yhbt.net> <20120412230457.GC22276@dcvr.yhbt.net> Message-ID: <20120413021054.GA18934@dcvr.yhbt.net> Patrik Wenger wrote: > > Actually, I think my proposed patch (in reply to Hongli) at > > http://mid.gmane.org/20120412221022.GA20640 at dcvr.yhbt.net > > should fix your issue. > > Oh, this sounds great! I see it's just one line. I'll try adding this line and see if it fixes my problem. > Thanks already! No problem. I've also pushed it out as a commit + test case: http://bogomips.org/unicorn.git/patch?id=b26d3e2c4387707c I'm looking at releasing unicorn 4.3.0 early next week, probably Tuesday (including REQUEST_PATH/REQUEST_URI length limit tweaks). > I don't know much about system calls (honestly, never heard of > shutdown() before) but I think I understand. > The kernel then doesn't fee the allocated client socket and thus > Unicorn (or Nginx? That's the other end of the client socket, I > think...) thinks there's still more to come. Sorta, I think I conflated the shutdown vs releasing resources a bit. Releasing kernel resources happens for all file descriptors. However, connected sockets may negotiate a graceful termination of the connection, and that's what shutdown() can do. However, close() will do it implicitly. If a kernel were implemented in Ruby, close() would be something like this, showing how shutdown() can get called automatically: def SYS_close(fd) # assume @fd_map is an array mapping fd to file/socket objects io_object = @fd_map[fd] return Errno::EBADF if io_object.nil? # allow +fd+ to be reused immediately @fd_map[fd] = nil io_object.refcount -= 1 # if there are no more references, do other work: if io_object.refcount == 0 if io_object.kind_of?(Socket) # assume this is idempotent SYS_shutdown(fd, SHUT_RDWR) end # free memory and any other resources allocated io_object.destroy! end end From paddor at gmail.com Fri Apr 13 12:11:02 2012 From: paddor at gmail.com (paddor) Date: Fri, 13 Apr 2012 14:11:02 +0200 Subject: Background jobs with #fork In-Reply-To: <20120413021054.GA18934@dcvr.yhbt.net> References: <20120412203956.GA8980@dcvr.yhbt.net> <20120412230457.GC22276@dcvr.yhbt.net> <20120413021054.GA18934@dcvr.yhbt.net> Message-ID: > No problem. ?I've also pushed it out as a commit + test case: > > ?http://bogomips.org/unicorn.git/patch?id=b26d3e2c4387707c Thank you (and Hongli Lai) so much. It works like a charm :-) > If a kernel were implemented in Ruby, close() would be something like > this, showing how shutdown() can get called automatically: > > ?def SYS_close(fd) > ? ?# assume @fd_map is an array mapping fd to file/socket objects > ? ?io_object = @fd_map[fd] > > ? ?return Errno::EBADF if io_object.nil? > > ? ?# allow +fd+ to be reused immediately > ? ?@fd_map[fd] = nil > > ? ?io_object.refcount -= 1 > > ? ?# if there are no more references, do other work: > ? ?if io_object.refcount == 0 > ? ? ?if io_object.kind_of?(Socket) > ? ? ? ?# assume this is idempotent > ? ? ? ?SYS_shutdown(fd, SHUT_RDWR) > ? ? ?end > > ? ? ?# free memory and any other resources allocated > ? ? ?io_object.destroy! > ? ?end > ?end Thanks for this explanation. I understand now. :-) -- paddor From normalperson at yhbt.net Tue Apr 17 22:14:27 2012 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 17 Apr 2012 22:14:27 +0000 Subject: [ANN] unicorn 4.3.0 - minor fixes and updates Message-ID: <20120417221427.GB4123@dcvr.yhbt.net> Changes: * PATH_INFO (aka REQUEST_PATH) increased to 4096 (from 1024). This allows requests with longer path components and matches the system PATH_MAX value common to GNU/Linux systems for serving filesystem components with long names. * Apps that fork() (but do not exec()) internally for background tasks now indicate the end-of-request immediately after writing the Rack response. Thanks to Hongli Lai, Lawrence Pit, Patrick Wenger and Nuo Yan for their valuable feedback for this release. * http://unicorn.bogomips.org/ * mongrel-unicorn at rubyforge.org * git://bogomips.org/unicorn.git * http://unicorn.bogomips.org/NEWS.atom.xml From jamesandham at hotmail.com Thu Apr 19 12:22:20 2012 From: jamesandham at hotmail.com (Jame Sandham) Date: 19 Apr 2012 08:22:20 -0400 Subject: Please reply Message-ID: <20120419122220.9764.qmail@mattrichling.com> Dear friend, I have emailed you earlier without a response.In my first email I mentioned about my late client whose relatives I cannot get in touch with. I am compelled to do this because I would not want the bank to push my clients funds into the bank treasury as unclaimed inheritance. This mail is written and intended to solicit your assistance to be presented as the HEIR to my Late Client since you are a foreigner and only a foreigner can lay claims on this inheritance. The Governing Body of the Bank has contacted me on this matter and I am yet to provide the HEIR to lay claims to the Fund. Under a clear and legitimate agreement with you,I shall seek your consent to be presented as the HEIR so that my late Client's fund will not be Confiscated by the Bank and pushed into the bank treasury as unclaimed bills. For the sake of Transparency on this matter,you are free to make immediate contact for further clarification and explanation on this matter. I will need you also to reconfirm your:Full Name:Marital Status:Address:Occupation:Age:Contact telephone number to ascertain the fact that I am dealing with the right person.Thank you very much for your anticipated acceptance while we expect your prompt response to this matter as the urgency demands. Yours faithfully, James Sandham(ESQ.) LONDON UNITED KINGDOM From takemawmsg90yaon at yahoo.com Sat Apr 21 12:47:08 2012 From: takemawmsg90yaon at yahoo.com (Rick) Date: Sat, 21 Apr 2012 20:47:08 +0800 Subject: =?GB2312?B?RGlnaXRhbCBQaG90byBFZGl0aW5nIFNlcg==?= =?GB2312?B?dmljZXMgLSBQaG90byBDdXRvdXQgU2Vydg==?= =?GB2312?B?aWNl?= Message-ID: You are receiving this email because we wish you to use our digital photo editing services. We are a China based Imaging Professionals. We offer basic and advanced digital photo Editing services and solutions like photo Cutout, morphological photo Editing, photoshop photo Editing, satellite photo Editing, color photo Editing and vector photo Editing using latest techniques. Our mainly services are: 1. Photo Cutout 2. Photo Enhancement 3. Photo Retouching 4. Vector Conversion 5. Pop Art 6. Images Masking 7. Clipping Path 8. Photo Restoration 9. Web Design Looking forward to work for you! Best regards, Rick Rickkanuica Imaging Professionals Contact: contactrick at yeah.net Send address to imremove at yeah.net for remove From amhod707 at gmail.com Thu Apr 26 03:09:23 2012 From: amhod707 at gmail.com (=?ISO-2022-JP?B?GyRCIVobKEJKdXN0R2l2aW5nGyRCIVsbKEI=?=) Date: Thu, 26 Apr 2012 12:09:23 +0900 Subject: =?ISO-2022-JP?B?GyRCIVobKEJKdXN0R2l2aW5nGyRCIVsbKEJGdW5kcyBUcmFuc2Zlci4=?= Message-ID: <20120426030923.879DAAC98A@www.justgiving.jp> Hello Sir, I write to request your partnership in a legitimate high yield business transaction on funds transfer to enable us contribute part of the proceed to this charity. There are legitimate documents available here now to effect the payment and transfer of a large sum of money in US Dollars to your safe account abroad. I need your full co-operation for the works and logistics. As I work for government http://www.dprnigeria.com/ at a responsible position, I have planned a safe way to get things done and use my position to help you achieve a successful operation. Once you receive the money, we will donate 10% of it to this charity on whose platform am able to reach you. If this is possible with you, then treat as urgent and confidential as you reply so we can agree on terms of co-operation and sharing. This is a safe deal. Thanks, A. M. From jnimety at continuity.net Fri Apr 27 14:36:04 2012 From: jnimety at continuity.net (Joel Nimety) Date: Fri, 27 Apr 2012 10:36:04 -0400 Subject: app error: Socket is not connected (Errno::ENOTCONN) Message-ID: I'm getting the following errors multiple times per request when using 4.3.0. I do not receive any errors when using 4.2.1. Please CC me on replies, I'm not subscribed to the mailing list. 16:48:42 web.1 | E, [2012-04-26T16:48:42.733954 #87940] ERROR -- : app error: Socket is not connected (Errno::ENOTCONN) 16:48:42 web.1 | E, [2012-04-26T16:48:42.734193 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `shutdown' 16:48:42 web.1 | E, [2012-04-26T16:48:42.734343 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `process_client' 16:48:42 web.1 | E, [2012-04-26T16:48:42.734517 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:605:in `worker_loop' 16:48:42 web.1 | E, [2012-04-26T16:48:42.734635 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:487:in `spawn_missing_workers' 16:48:42 web.1 | E, [2012-04-26T16:48:42.734775 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:137:in `start' 16:48:42 web.1 | E, [2012-04-26T16:48:42.734946 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/gems/unicorn-4.3.0/bin/unicorn:121:in `' 16:48:42 web.1 | E, [2012-04-26T16:48:42.735072 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/bin/unicorn:19:in `load' 16:48:42 web.1 | E, [2012-04-26T16:48:42.735208 #87940] ERROR -- : /Users/jnimety/.rvm/gems/ruby-1.9.3-p125 at control/bin/unicorn:19:in `
' -- Joel Nimety From matt at nearapogee.com Fri Apr 27 15:08:26 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 08:08:26 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: Message-ID: On Fri, Apr 27, 2012 at 7:36 AM, Joel Nimety wrote: > I'm getting the following errors multiple times per request when using > 4.3.0. I too started noticing these errors. I was trying to do some high level debugging before posting. Here are two examples of what I am getting: cache: [GET /assets/marketing/splash.jpg] stale, valid, store 127.0.0.1 - - [27/Apr/2012 07:50:01] "GET /assets/marketing/splash.jpg HTTP/1.1" 200 30431 0.0411 E, [2012-04-27T07:50:01.636410 #24786] ERROR -- : app error: Socket is not connected (Errno::ENOTCONN) E, [2012-04-27T07:50:01.636556 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `shutdown' E, [2012-04-27T07:50:01.636664 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `process_client' E, [2012-04-27T07:50:01.636781 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:605:in `worker_loop' E, [2012-04-27T07:50:01.636892 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:487:in `spawn_missing_workers' E, [2012-04-27T07:50:01.636996 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:137:in `start' E, [2012-04-27T07:50:01.637112 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/bin/unicorn:121:in `' E, [2012-04-27T07:50:01.637222 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/bin/unicorn:23:in `load' E, [2012-04-27T07:50:01.637333 #24786] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/bin/unicorn:23:in `
' cache: [GET /assets/marketing/splash-button.gif] stale, valid, store 127.0.0.1 - - [27/Apr/2012 07:50:01] "GET /assets/marketing/splash-button.gif HTTP/1.1" 200 5735 0.0188 cache: [GET /] miss 127.0.0.1 - - [27/Apr/2012 08:03:27] "GET / HTTP/1.1" 200 - 0.7057 E, [2012-04-27T08:03:27.811826 #25138] ERROR -- : app error: Socket is not connected (Errno::ENOTCONN) E, [2012-04-27T08:03:27.811968 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `shutdown' E, [2012-04-27T08:03:27.812080 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `process_client' E, [2012-04-27T08:03:27.812189 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:605:in `worker_loop' E, [2012-04-27T08:03:27.812301 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:487:in `spawn_missing_workers' E, [2012-04-27T08:03:27.812411 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:137:in `start' E, [2012-04-27T08:03:27.812523 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/unicorn-4.3.0/bin/unicorn:121:in `' E, [2012-04-27T08:03:27.812635 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/bin/unicorn:23:in `load' E, [2012-04-27T08:03:27.812747 #25138] ERROR -- : /Users/matt/.rbenv/versions/1.9.3-p194/bin/unicorn:23:in `
' cache: [GET /assets/reset.css?body=1] stale, valid, store 127.0.0.1 - - [27/Apr/2012 08:03:28] "GET /assets/reset.css?body=1 HTTP/1.1" 304 - 0.0187 The cache statements are from rack-cache in development. This seems to only be related to static files: images, css, and js assets. The errors change on every request. My application is a 3.2.3 rails app. I can look into this further later in the day. Please let me know if I can provide further debugging information. Thanks! Matt From lists at SouthernOhio.net Fri Apr 27 17:54:35 2012 From: lists at SouthernOhio.net (George) Date: Fri, 27 Apr 2012 13:54:35 -0400 Subject: Errors after 4.3.0 update Message-ID: Having just updated to 4.3.0 just a moment ago, I thought I'd test things out before deploying the updated app. The app for which I'm using Unicorn seems to work fine, however I am seeing some odd errors in the logs that I've not hitherto seen and I'm not entirely sure of how to interpret them or to find out more of what may actually be going on. This is running with rails 3.1.4. There are many variations for different files of the following error: 13:38:27 web.1 | Served asset /lofslidernews/lofslidernews_index.js - 304 Not Modified (5ms) 13:38:27 web.1 | 124.253.77.245 - - [27/Apr/2012 13:38:27] "GET /assets/lofslidernews/lofslidernews_index.js?body=1 HTTP/1.1" 304 - 0.0788 13:38:27 web.1 | E, [2012-04-27T13:38:27.179177 #58563] ERROR -- : app error: Socket is not connected (Errno::ENOTCONN) 13:38:27 web.1 | E, [2012-04-27T13:38:27.179373 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `shutdown' 13:38:27 web.1 | E, [2012-04-27T13:38:27.179553 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:539:in `process_client' 13:38:27 web.1 | E, [2012-04-27T13:38:27.179719 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:605:in `worker_loop' 13:38:27 web.1 | E, [2012-04-27T13:38:27.179881 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:487:in `spawn_missing_workers' 13:38:27 web.1 | E, [2012-04-27T13:38:27.180048 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/lib/unicorn/http_server.rb:137:in `start' 13:38:27 web.1 | E, [2012-04-27T13:38:27.180161 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/gems/unicorn-4.3.0/bin/unicorn:121:in `' 13:38:27 web.1 | E, [2012-04-27T13:38:27.180290 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/bin/unicorn:19:in `load' 13:38:27 web.1 | E, [2012-04-27T13:38:27.180425 #58563] ERROR -- : /Users/username/.rvm/gems/ruby-1.9.3-p125 at all/bin/unicorn:19:in `
' Would appreciate any help in isolating what may be the issue here. -George From normalperson at yhbt.net Fri Apr 27 18:59:40 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 18:59:40 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: Message-ID: <20120427185940.GA27312@dcvr.yhbt.net> Joel Nimety wrote: > I'm getting the following errors multiple times per request when using > 4.3.0. I do not receive any errors when using 4.2.1. Please CC me on > replies, I'm not subscribed to the mailing list. Multiple times per request? Yikes. This can be expected occasionally. The below patch will just silently ignore them since they're not errors we can avoid/deal with any other way. I'd still prefer exceptions to not get raised at all (they're expensive). Are the rest of you guys getting these errors intermittently, or only once in a while. Rainbows! has long had a similar patch as below, but I'm a bit concerned as to why Joel is getting this multiple times per request... >From 04901da5ae0b4655c83be05d24ae737f1b572002 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Fri, 27 Apr 2012 11:48:16 -0700 Subject: [PATCH] http_server: ignore ENOTCONN (mostly from shutdown(2)) Since there's nothing unicorn can do to avoid this error on unconnected/halfway-connected clients, ignoring ENOTCONN is a safe bet. Rainbows! has long had this rescue as it called getpeername(2) on untrusted sockets --- lib/unicorn/http_server.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index f942e2f..14a6f9a 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -507,7 +507,8 @@ class Unicorn::HttpServer # the socket is closed at the end of this function def handle_error(client, e) msg = case e - when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF + when EOFError,Errno::ECONNRESET,Errno::EPIPE,Errno::EINVAL,Errno::EBADF, + Errno::ENOTCONN Unicorn::Const::ERROR_500_RESPONSE when Unicorn::RequestURITooLongError Unicorn::Const::ERROR_414_RESPONSE -- Eric Wong From jnimety at continuity.net Fri Apr 27 19:12:01 2012 From: jnimety at continuity.net (Joel Nimety) Date: Fri, 27 Apr 2012 15:12:01 -0400 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427185940.GA27312@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 2:59 PM, Eric Wong wrote: > Joel Nimety wrote: >> I'm getting the following errors multiple times per request when using >> 4.3.0. ?I do not receive any errors when using 4.2.1. ?Please CC me on >> replies, I'm not subscribed to the mailing list. > > Multiple times per request? ?Yikes. ?This can be expected occasionally. > 46 times in development on the last request I tested. I can provide my Gemfile, logs and any other debug info if you like. Just to reiterate, if I downgrade to 4.2.1 I don't see the error at all. -- Joel Nimety From normalperson at yhbt.net Fri Apr 27 19:32:56 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 19:32:56 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: <20120427185940.GA27312@dcvr.yhbt.net> Message-ID: <20120427193256.GA31090@dcvr.yhbt.net> Joel Nimety wrote: > On Fri, Apr 27, 2012 at 2:59 PM, Eric Wong wrote: > > Joel Nimety wrote: > >> I'm getting the following errors multiple times per request when using > >> 4.3.0. ?I do not receive any errors when using 4.2.1. ?Please CC me on > >> replies, I'm not subscribed to the mailing list. > > > > Multiple times per request? ?Yikes. ?This can be expected occasionally. > > 46 times in development on the last request I tested. I can provide > my Gemfile, logs and any other debug info if you like. Just to > reiterate, if I downgrade to 4.2.1 I don't see the error at all. Yes, the addition of the shutdown() call[1] in 4.3.0 can cause ENOTCONN occasionally. Ignoring the exception works, but given the cost of generating exceptions in Ruby in the first place (especially 1.9), I don't think it's a good idea... I can't reproduce this error. Are you on a UNIX listener or TCP listene? What environment (OS/kernel version) is this? [1] see thread starting here about adding shutdown(): http://mid.gmane.org/CAOG6bOTseAPbjU5LYchODqjdF3-Ez4+M8jo-D_D2Wq0jkdc4Rw at mail.gmail.com From matt at nearapogee.com Fri Apr 27 19:33:47 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 12:33:47 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427185940.GA27312@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 11:59 AM, Eric Wong wrote: > Are the rest of you guys getting these errors intermittently, or only > once in a while. > > Rainbows! has long had a similar patch as below, but I'm a bit concerned > as to why Joel is getting this multiple times per request... I usually get several every request as well (in dev). It can be one or a handful at a time. They seem to be somewhat random as to when they are thrown. Thanks for the patch (and the fantastic support, Eric!). If I can help narrow down the problem please let me know! It seems the issue is when unicorn is serving static files, at least for me. In production, Nginx serves my static files so I doubt there will be any issue. I do have 4.3.0 deployed on Ubuntu 10.04.4 LTS and I just checked the logs and I see no errors when I make the same request. From matt at nearapogee.com Fri Apr 27 19:45:11 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 12:45:11 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427193256.GA31090@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427193256.GA31090@dcvr.yhbt.net> Message-ID: > Yes, the addition of the shutdown() call[1] in 4.3.0 can cause ENOTCONN > occasionally. > > Ignoring the exception works, but given the cost of generating > exceptions in Ruby in the first place (especially 1.9), I don't > think it's a good idea... > > I can't reproduce this error. ?Are you on a UNIX listener or TCP > listene? ?What environment (OS/kernel version) is this? I am running OS X 10.7.3, ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.3.0]. I was going to try it on my linux dev machine in a bit. I can get you kernel version then, if it occurs. I am just running unicorn in local dev and hitting it with a browser (Safari Version 5.1.5 (7534.55.3)) on localhost:8080. In production, I use a UNIX listener. (I am seeing no errors in production.) From matt at nearapogee.com Fri Apr 27 19:47:16 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 12:47:16 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427193256.GA31090@dcvr.yhbt.net> Message-ID: > In production, I use a UNIX listener. (I am seeing no errors in production.) (Again unicorn is not serving static files in production.) From jnimety at continuity.net Fri Apr 27 19:47:20 2012 From: jnimety at continuity.net (Joel Nimety) Date: Fri, 27 Apr 2012 15:47:20 -0400 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427193256.GA31090@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427193256.GA31090@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 3:32 PM, Eric Wong wrote: > Joel Nimety wrote: >> On Fri, Apr 27, 2012 at 2:59 PM, Eric Wong wrote: >> > Joel Nimety wrote: >> >> I'm getting the following errors multiple times per request when using >> >> 4.3.0. ?I do not receive any errors when using 4.2.1. ?Please CC me on >> >> replies, I'm not subscribed to the mailing list. >> > >> > Multiple times per request? ?Yikes. ?This can be expected occasionally. >> >> 46 times in development on the last request I tested. ?I can provide >> my Gemfile, logs and any other debug info if you like. ?Just to >> reiterate, if I downgrade to 4.2.1 I don't see the error at all. > > Yes, the addition of the shutdown() call[1] in 4.3.0 can cause ENOTCONN > occasionally. > > Ignoring the exception works, but given the cost of generating > exceptions in Ruby in the first place (especially 1.9), I don't > think it's a good idea... > > I can't reproduce this error. ?Are you on a UNIX listener or TCP > listene? ?What environment (OS/kernel version) is this? OS X 10.7.3, ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.3.0] with the latest command line x code. tcp, using the the following config. I've tried removing preload_app but it didn't help. listen 3000 worker_processes 4 preload_app true timeout 30 > > > [1] see thread starting here about adding shutdown(): > ?http://mid.gmane.org/CAOG6bOTseAPbjU5LYchODqjdF3-Ez4+M8jo-D_D2Wq0jkdc4Rw at mail.gmail.com -- Joel Nimety SVP Technology Continuity Control 203.285.8133 (office) 203.763.9263 (cell) http://www.continuity.net http://www.linkedin.com/in/jnimety From normalperson at yhbt.net Fri Apr 27 19:51:00 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 19:51:00 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: <20120427185940.GA27312@dcvr.yhbt.net> Message-ID: <20120427195100.GA322@dcvr.yhbt.net> George wrote: > Just fyi, my dev logs are now rife with these: http://j.mp/IezMAu > > Will plug your patch in, but will have to figure out another option for > heroku deployment. Is this affecting your heroku deployment? What OS/kernel version are you running? From what Joel and Matt say, it could be more likely to trigger on *BSD-based systems. From normalperson at yhbt.net Fri Apr 27 19:53:58 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 19:53:58 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427195100.GA322@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427195100.GA322@dcvr.yhbt.net> Message-ID: <20120427195358.GA985@dcvr.yhbt.net> Eric Wong wrote: > George wrote: > > Just fyi, my dev logs are now rife with these: http://j.mp/IezMAu > > > > Will plug your patch in, but will have to figure out another option for > > heroku deployment. > > Is this affecting your heroku deployment? What OS/kernel version > are you running? From what Joel and Matt say, it could be more likely > to trigger on *BSD-based systems. Also, if you revert my previous patch, does this also prevent the error from manifesting? (I expect ignoring ENOTCONN _once_in_a_while_ will be required, but definitely not every request). diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index 14a6f9a..bb91e7f 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -537,7 +537,7 @@ class Unicorn::HttpServer end @request.headers? or headers = nil http_response_write(client, status, headers, body) - client.shutdown # in case of fork() in Rack app + client.close_write # in case of fork() in Rack app client.close # flush and uncork socket immediately, no keepalive rescue => e handle_error(client, e) From matt at nearapogee.com Fri Apr 27 20:30:10 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 13:30:10 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427195100.GA322@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427195100.GA322@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 12:51 PM, Eric Wong wrote: > George wrote: >> Just fyi, my dev logs are now rife with these: http://j.mp/IezMAu >> >> Will plug your patch in, but will have to figure out another option for >> heroku deployment. > > Is this affecting your heroku deployment? ?What OS/kernel version > are you running? ?From what Joel and Matt say, it could be more likely > to trigger on *BSD-based systems. I just tried on a 64-bit Ubuntu 11.10 (kernel 3.0.0-17-generic), ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]. No issue. Reloaded a several pages, many times. No ENOTCONN. Not one. Same app. Neither on ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]. BSD issue? From normalperson at yhbt.net Fri Apr 27 21:15:36 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 21:15:36 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427195100.GA322@dcvr.yhbt.net> Message-ID: <20120427211536.GA4903@dcvr.yhbt.net> Matt Smith wrote: > I just tried on a 64-bit Ubuntu 11.10 (kernel 3.0.0-17-generic), ruby > 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]. No issue. Reloaded > a several pages, many times. No ENOTCONN. Not one. Same app. OK, thank you for the confirmation. I wouldn't rule out an occasional ENOTCONN, though, even if nginx <-> unicorn are on the same LAN. So I'll still push out my change to ignore Errno::ENOTCONN. Since it's highly uncommon in production environments, I'm not concerned about the performance overhead. > BSD issue? It doesn't affect my FreeBSD 9.0 nor Debian/kFreeBSD 6.0 VMs, either. OSX is non-Free Software, so unicorn cannot officially support it and performance is irrelevant on it. From normalperson at yhbt.net Fri Apr 27 21:16:57 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 21:16:57 +0000 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427195100.GA322@dcvr.yhbt.net> Message-ID: <20120427211657.GA2869@dcvr.yhbt.net> George wrote: > No, you know I think you're right Eric. I am testing on OS X 10.7.4 prior > to deployment at the moment and so it appears that the bsd core has some > bearing here. It might just be OSX, I can't reproduce the issue on FreeBSD 9.0 nor Debian/kFreeBSD 6.0. Any OpenBSD/NetBSD users care to chime in? > I'm also curious, have you considered switching over to GitHub for issue > management? I'm spoiled nowadays with issue reporting vs the good old > mailing lists. Even Google's mailing lists are a step above. ;) No. I can't stand web browsers and logging into websites. I'll never endorse commercial interests when working on Free Software, either. Usenet would be ideal, but it's mostly dead nowadays but everybody still has email (more or less, HTML mail isn't appropriate for *nix-based development) For mailing lists, consider a proper Unix MUA with threading support[1] an essential part of your development environment (as important as your $EDITOR). With Unix pipes, messages flow between inbox/working directory easily (via tools like patch/git-am). This also lets me use the same $EDITOR for both writing code and emails. [1] - mutt in my case, but I've heard good things about alpine and GNUS From normalperson at yhbt.net Fri Apr 27 22:00:32 2012 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 27 Apr 2012 22:00:32 +0000 Subject: [ANN] unicorn 4.3.0.2.g4551 gem prerelease Message-ID: <20120427220032.GA14798@dcvr.yhbt.net> Can you guys test this out? Thanks. I'll make a real 4.3.1 release tomorrow. >From RubyGems.org: gem install --pre unicorn ChangeLog since v4.3.0: commit 4551c8ad4d63d4031c618f76d39532b39e88f9be Author: Eric Wong Date: Fri Apr 27 14:42:38 2012 -0700 stream_input: call shutdown(2) if a client EOFs on us In case the Rack app forks before a client upload is complete, shutdown(2) the socket to ensure the client isn't attempting to read from us (even if it explicitly stopped writes). commit 04901da5ae0b4655c83be05d24ae737f1b572002 Author: Eric Wong Date: Fri Apr 27 11:48:16 2012 -0700 http_server: ignore ENOTCONN (mostly from shutdown(2)) Since there's nothing unicorn can do to avoid this error on unconnected/halfway-connected clients, ignoring ENOTCONN is a safe bet. Rainbows! has long had this rescue as it called getpeername(2) on untrusted sockets -- From matt at nearapogee.com Fri Apr 27 22:07:54 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 15:07:54 -0700 Subject: [ANN] unicorn 4.3.0.2.g4551 gem prerelease In-Reply-To: <20120427220032.GA14798@dcvr.yhbt.net> References: <20120427220032.GA14798@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 3:00 PM, Eric Wong wrote: > Can you guys test this out? ?Thanks. ?I'll make a real 4.3.1 release > tomorrow. Stopped the errors. Everything seems to be working fine here. From matt at nearapogee.com Fri Apr 27 22:14:18 2012 From: matt at nearapogee.com (Matt Smith) Date: Fri, 27 Apr 2012 15:14:18 -0700 Subject: app error: Socket is not connected (Errno::ENOTCONN) In-Reply-To: <20120427211657.GA2869@dcvr.yhbt.net> References: <20120427185940.GA27312@dcvr.yhbt.net> <20120427195100.GA322@dcvr.yhbt.net> <20120427211657.GA2869@dcvr.yhbt.net> Message-ID: > For mailing lists, consider a proper Unix MUA with threading support[1] > an essential part of your development environment (as important as your > $EDITOR). ?With Unix pipes, messages flow between inbox/working > directory easily (via tools like patch/git-am). ?This also lets me use > the same $EDITOR for both writing code and emails. I keep meaning to make the transition to mutt... Thanks for the reminder. ;-) From adam.ffa at gmail.com Sat Apr 28 02:46:31 2012 From: adam.ffa at gmail.com (adam k) Date: Fri, 27 Apr 2012 19:46:31 -0700 Subject: unicorn + sleep = long wake up loading time Message-ID: Hey, I'm running unicorn behind nginx. If the site isn't accessed for a long period of time, the next access of the site is loads very slowly, sometimes even a 404 error shows. Subsequent requests have a normal loading time until the site isn't accessed for a long period of time again. How do I stop this behavior? I looked at the code for unicorn and it seems to happen when unicorn is looking to kill "lazy" workers and then if it chooses not to, it creates a sleep period based on a timeout + 1. I also read the documentation and searched the mailing list starting from 2011. From normalperson at yhbt.net Sat Apr 28 04:40:03 2012 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Apr 2012 04:40:03 +0000 Subject: unicorn + sleep = long wake up loading time In-Reply-To: References: Message-ID: <20120428044003.GA10474@dcvr.yhbt.net> adam k wrote: > I'm running unicorn behind nginx. If the site isn't accessed for a > long period of time, the next access of the site is loads very slowly, > sometimes even a 404 error shows. Subsequent requests have a normal > loading time until the site isn't accessed for a long period of time > again. How do I stop this behavior? unicorn is likely getting swapped out or a backend connection (e.g. database connection) is getting timed out due to inactivity. * How much memory (RSS) is each worker using when the site is active? * Is RSS noticeably lower when the site's been idle for a long time? * How many unicorn worker processes? (the above 3 questions also apply to nginx, but nginx usually uses much less memory than unicorn) * How much physical RAM do you have? * What is the timeout value of unicorn? (default is 60s) * What else is running on the machine? * Anything mentioned in the unicorn stderr logs, or your application logs? The updatedb cronjob is one major culprit of causing apps to swap out, but it could be any number of things (backup jobs, rootkit scanners, prelink, ...). Try setting up a cronjob run curl to hit an endpoint of your app every few minutes. Make sure that endpoint exercises whatever non-local dependencies (hits the DB, etc...) your app has. > I looked at the code for unicorn and it seems to happen when unicorn > is looking to kill "lazy" workers and then if it chooses not to, it > creates a sleep period based on a timeout + 1. I also read the > documentation and searched the mailing list starting from 2011. I think you're looking at the master process, which shouldn't affect performance of the workers. The workers wake up whenever there's socket activity, but it can take longer if they're swapped out. From jnimety at continuity.net Sat Apr 28 17:07:23 2012 From: jnimety at continuity.net (Joel Nimety) Date: Sat, 28 Apr 2012 13:07:23 -0400 Subject: [ANN] unicorn 4.3.0.2.g4551 gem prerelease In-Reply-To: References: <20120427220032.GA14798@dcvr.yhbt.net> Message-ID: On Fri, Apr 27, 2012 at 6:07 PM, Matt Smith wrote: > On Fri, Apr 27, 2012 at 3:00 PM, Eric Wong wrote: >> Can you guys test this out? ?Thanks. ?I'll make a real 4.3.1 release >> tomorrow. > > Stopped the errors. Everything seems to be working fine here. Stopped the errors for me as well. -- Joel Nimety From normalperson at yhbt.net Sat Apr 28 20:49:53 2012 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Apr 2012 20:49:53 +0000 Subject: Fwd: Rack::Lint::LintError In-Reply-To: References: Message-ID: <20120428204953.GA723@dcvr.yhbt.net> George wrote: > Hi Eric, why would I get this message? > > ---------- Forwarded message ---------- > From: > Date: Sat, Apr 28, 2012 at 3:32 PM > Subject: Rack::Lint::LintError > To: lists at southernohio.net > > The message's content type was not explicitly allowed Don't send HTML mail, it's a waste of network/disk bandwidth and spam filters hate it. > ---------- Forwarded message ---------- > From: George > To: mongrel-unicorn > Cc: > Date: Sat, 28 Apr 2012 15:32:22 -0400 > Subject: Rack::Lint::LintError > Eric, I have noticed something else here on Mac OS X when a monitoring > service pulls the HEAD every 5 minutes. We've not been able to hunt down > why this is happening yet and thought I should pass it your way for your > thoughts even though it may not be wholly dependent on Unicorn. The odd > thing is though, that like this last issue, it does not appear to occur on > Heroku / linux servers: > > 15:21:12 web.1 | 74.86.158.106 - - [28/Apr/2012 15:21:12] "GET / You said HEAD above, but it says GET here --------^^^ > HTTP/1.1" 200 33686 0.3897 > 15:21:12 web.1 | E, [2012-04-28T15:21:12.423657 #51950] ERROR -- : app > error: Content-Length header was 33686, but should be 0 > (Rack::Lint::LintError) This might be a Rails(?) bug, Joe encountered this last year. http://mid.gmane.org/CACfv+pJww_nSAuBrSLSaTDO5tdbhYAcZofcB2BBvccSKmF99Wg at mail.gmail.com (the whole thread is probably worth a read: http://news.gmane.org/find-root.php?message_id=%3cCACfv%2bpJww%5fnSAuBrSLSaTDO5tdbhYAcZofcB2BBvccSKmF99Wg%40mail.gmail.com%3e ) From flippedootninja at gmail.com Sat Apr 28 21:33:49 2012 From: flippedootninja at gmail.com (Pico Aeterna) Date: Sat, 28 Apr 2012 14:33:49 -0700 Subject: Socket errors and nginx 502 gateway errors Message-ID: Hi, We've been getting reports that our users have been getting random 502 Errors. After investigating I noticed 'upstream prematurely closed connection while reading response header from upstream' in my nginx error logs. From using strace it appears that there's numerous socket SO_ERROR.. I temporarily switched out the domain sockets with TCP, however am seeing the same result/issue. Strace and Nginx error log below: 7366 epoll_ctl(16, EPOLL_CTL_ADD, 89, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=2473356496, u64=46963645770960}}) = 0 7366 connect(89, {sa_family=AF_FILE, path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0 7366 getsockopt(89, SOL_SOCKET, SO_ERROR, [1422340301959200768], [4]) = 0 7366 writev(89, [{"GET /natalieliao HTTP/1.0\r\nX-Forwarded-For: 94.224.228.239, 94.224.228.239\r\nHost"..., 1084}], 1) = 1084 7366 epoll_wait(16, {{EPOLLOUT, {u32=2473356496, u64=46963645770960}}, {EPOLLOUT, {u32=2473355152, u64=46963645769616}}}, 512, 4316) = 2 7366 recvfrom(88, 0x7fff856374b7, 1, 2, 0, 0) = -1 EAGAIN (Resource temporarily unavailable) 7366 epoll_wait(16, {{EPOLLIN|EPOLLOUT|EPOLLHUP, {u32=2473356496, u64=46963645770960}}}, 512, 4315) = 1 7366 recvfrom(89, "", 4096, 0, NULL, NULL) = 0 7366 write(3, "2012/04/28 17:14:45 [error] 7366#0: *21826315 upstream prematurely closed connec"..., 319) = 319 7366 close(89) = 0 7366 stat("/var/www/domain/current/public/system/maintenance.html", 0x7fff85637210) = -1 ENOENT (No such file or directory) 7366 stat("/var/www/domain/current/public/500.html/index.html", 0x7fff85637190) = -1 ENOENT (No such file or directory) 7366 stat("/var/www/domain/current/public/500.html.html", 0x7fff85637190) = -1 ENOENT (No such file or directory) 7366 stat("/var/www/domain/current/public/500.html", 0x7fff85637190) = -1 ENOENT (No such file or directory) 7366 socket(PF_FILE, SOCK_STREAM, 0) = 89 7366 ioctl(89, FIONBIO, [1]) = 0 7366 epoll_ctl(16, EPOLL_CTL_ADD, 89, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=2473356497, u64=46963645770961}}) = 0 7366 connect(89, {sa_family=AF_FILE, path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0 7366 getsockopt(89, SOL_SOCKET, SO_ERROR, [1422495985933746176], [4]) = 0 7366 writev(89, [{"GET /natalieliao HTTP/1.0\r\nX-Forwarded-For: 94.224.228.239, 94.224.228.239\r\nHost"..., 1084}], 1) = 1084 7366 epoll_wait(16, {{EPOLLOUT, {u32=2473356497, u64=46963645770961}}}, 512, 4285) = 1 7366 epoll_wait(16, {{EPOLLIN|EPOLLOUT|EPOLLHUP, {u32=2473356497, u64=46963645770961}}}, 512, 4284) = 1 7366 recvfrom(89, "", 4096, 0, NULL, NULL) = 0 7366 write(3, "2012/04/28 17:14:45 [error] 7366#0: *21826315 upstream prematurely closed connec"..., 319) = 319 7366 close(89) = 0 7366 writev(88, [{"HTTP/1.1 502 Bad Gateway\r\nServer: nginx\r\nDate: Sat, 28 Apr 2012 21:14:45 GMT\r\nCo"..., 145}, {"\r\n502 Bad Gateway\r\n\r\n
nginx
\r\n\r\n\r\n", 46}], 3) = 311 7366 shutdown(88, 1 /* send */) = 0 2012/04/28 17:14:45 [error] 7366#0: *21826315 upstream prematurely closed connection while reading response header from upstream, client:94.224.228.239, server: *.domain.nu, request: "GET /natalieliao HTTP/1.1", upstream: "http://unix:/var/www/domain/shared/sockets/unicorn.sock:/natalieliao", host: "domain.nu" 2012/04/28 17:14:45 [error] 7366#0: *21826315 upstream prematurely closed connection while reading response header from upstream, client: 94.224.228.239, server: *.domain.nu, request: "GET /natalieliao HTTP/1.1", upstream: "http://unix:/var/www/domain/shared/sockets/unicorn.sock:/natalieliao", host: "domain.nu" Nginx information: nginx version: nginx/1.0.15 built by gcc 4.1.2 20080704 (Red Hat 4.1.2-52) TLS SNI support disabled configure arguments: --prefix=/etc/nginx --with-http_ssl_module --sbin-path=/usr/sbin --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --with-http_realip_module --with-http_stub_status_module --pid-path=/var/run/nginx.pid Unicorn information: $ bundle show unicorn /var/www/lookbook/shared/bundle/ruby/1.8/gems/unicorn-4.2.1 Ruby: ruby: interpreter: "ruby" version: "1.8.7" date: "2011-02-18" platform: "x86_64-linux" patchlevel: "2011-02-18 patchlevel 334" full_version: "ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.03" Setup: Load balancer (HaProxy) => nginx => unicorn Relevant sysctl.conf changes net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.tcp_fin_timeout = 15 #timewait sockets net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.core.netdev_max_backlog = 4096 net.core.somaxconn = 4096 Just curious to know if there's a socket bug that I missed with the current version of unicorn that's installed? From normalperson at yhbt.net Sat Apr 28 21:59:04 2012 From: normalperson at yhbt.net (Eric Wong) Date: Sat, 28 Apr 2012 21:59:04 +0000 Subject: Socket errors and nginx 502 gateway errors In-Reply-To: References: Message-ID: <20120428215904.GA6524@dcvr.yhbt.net> Pico Aeterna wrote: > Hi, > > We've been getting reports that our users have been getting random 502 > Errors. After investigating I noticed 'upstream prematurely closed > connection while reading response header from upstream' in my nginx > error logs. From using strace it appears that there's numerous socket > SO_ERROR.. I temporarily switched out the domain sockets with TCP, > however am seeing the same result/issue. > > Strace and Nginx error log below: Can you show us an strace of a unicorn worker, too? (use "worker_processes 1" when debugging this) Also, the unicorn stderr log can be helpful. > 7366 epoll_ctl(16, EPOLL_CTL_ADD, 89, {EPOLLIN|EPOLLOUT|EPOLLET, > {u32=2473356496, u64=46963645770960}}) = 0 > 7366 connect(89, {sa_family=AF_FILE, > path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0 > 7366 getsockopt(89, SOL_SOCKET, SO_ERROR, [1422340301959200768], [4]) = 0 > 7366 writev(89, [{"GET /natalieliao HTTP/1.0\r\nX-Forwarded-For: > 94.224.228.239, 94.224.228.239\r\nHost"..., 1084}], 1) = 1084 Can you pass "-s 2048" (or larger value) when stracing? It'll be helpful to know what else is in the HTTP header. > Load balancer (HaProxy) => nginx => unicorn Does hitting unicorn directly with curl (TCP) give an expected result? (You can use the strace output above to have curl mimic what nginx sends, or even use printf | socat which works with Unix sockets). > Relevant sysctl.conf changes Probably not relevant since you switched to Unix sockets, but consider default sysctl values while debugging (especially tcp_tw_*) > net.ipv4.ip_local_port_range = 1024 65535 > net.ipv4.tcp_fin_timeout = 15 > #timewait sockets > net.ipv4.tcp_tw_reuse = 1 > net.ipv4.tcp_tw_recycle = 1 From normalperson at yhbt.net Sun Apr 29 07:48:49 2012 From: normalperson at yhbt.net (Eric Wong) Date: Sun, 29 Apr 2012 07:48:49 +0000 Subject: [ANN] unicorn 4.3.1 - shutdown() fixes Message-ID: <20120429074849.GA1291@dcvr.yhbt.net> Changes: * Call shutdown(2) if a client EOFs on us during upload. We can avoid holding a socket open if the Rack app forked a process during uploads. * ignore potential Errno::ENOTCONN errors (from shutdown(2)). Even on LANs, connections can occasionally be accept()-ed but be unusable afterwards. Thanks to Joel Nimety , Matt Smith and George on the mongrel-unicorn at rubyforge.org mailing list for their feedback and testing for this release. * http://unicorn.bogomips.org/ * mongrel-unicorn at rubyforge.org * git://bogomips.org/unicorn.git * http://unicorn.bogomips.org/NEWS.atom.xml From flippedootninja at gmail.com Sun Apr 29 16:34:54 2012 From: flippedootninja at gmail.com (Pico Aeterna) Date: Sun, 29 Apr 2012 09:34:54 -0700 Subject: Socket errors and nginx 502 gateway errors In-Reply-To: <20120428215904.GA6524@dcvr.yhbt.net> References: <20120428215904.GA6524@dcvr.yhbt.net> Message-ID: Eric, Thanks for the response. Unicorn Stderr Errorlog: reaped # worker=9 /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:172: [BUG] Segmentation fault ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.03 Exception `Errno::EAGAIN' at /var/www/domain/shared/bundle/ruby/1.8/gems/net-http-persistent-2.6/lib/net/http/faster.rb:15 - Resource temporarily unavailable Exception `EOFError' at /var/www/domain/shared/bundle/ruby/1.8/gems/net-http-persistent-2.6/lib/net/http/faster.rb:15 - end of file reached Exception `Errno::EAGAIN' at /var/www/domain/shared/bundle/ruby/1.8/gems/net-http-persistent-2.6/lib/net/http/faster.rb:15 - Resource temporarily unavailable Exception `EOFError' at /var/www/domain/shared/bundle/ruby/1.8/gems/net-http-persistent-2.6/lib/net/http/faster.rb:15 - end of file reached Exception `TypeError' at /var/www/domain/releases/20120427011507/app/views/search/_facet.html.erb:5 - can't convert String into Integer Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template widget/.erb in view path app/views Exception `EOFError' at /var/www/domain/shared/bundle/ruby/1.8/gems/newrelic_rpm-3.3.3/lib/new_relic/data_serialization.rb:111 - end of file reached Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template look/.erb in view path app/views Exception `Memcached::NotFound' at /var/www/domain/shared/bundle/ruby/1.8/gems/memcached-1.0.6/lib/memcached/memcached.rb:597 - Memcached::NotFound Exception `Memcached::NotFound' at /var/www/domain/shared/bundle/ruby/1.8/gems/memcached-1.0.6/lib/memcached/memcached.rb:525 - Memcached::NotFound Exception `EOFError' at /var/www/domain/shared/bundle/ruby/1.8/gems/newrelic_rpm-3.3.3/lib/new_relic/data_serialization.rb:111 - end of file reached Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template widget/.erb in view path app/views Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template widget/.erb in view path app/views Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template look/.erb in view path app/views Exception `NoMethodError' at /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:255 - undefined method `include?' for nil:NilClass Exception `ActionView::MissingTemplate' at /var/www/domain/shared/bundle/ruby/1.8/gems/actionpack-2.3.11/lib/action_view/paths.rb:74 - Missing template user/.erb in view path app/views Exception `Memcached::NotFound' at /var/www/domain/shared/bundle/ruby/1.8/gems/memcached-1.0.6/lib/memcached/memcached.rb:597 - Memcached::NotFound Exception `Memcached::NotFound' at /var/www/domain/shared/bundle/ruby/1.8/gems/memcached-1.0.6/lib/memcached/memcached.rb:597 - Memcached::NotFound Strace with 2048 on nginx 7369 connect(31, {sa_family=AF_FILE, path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0 7369 getsockopt(31, SOL_SOCKET, SO_ERROR, [1424310214479314944], [4]) = 0 7369 writev(31, [{"GET /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06 HTTP/1.0\r\nX-Forwarded -For: 83.9.20.243, 83.9.20.243\r\nHost: domain.nu\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 Safari/534.55.3\r\nAccept: */*\r\nReferer: http://www.xxxxx.com/\r\nAccept-Language: pl-PL\r\nAccept-Encoding: gzip, deflate\r\nCookie: __utma=78941577.1641012066.1319789648.1325094074.1325949868.3; __utmz=78941577.1325949868.3.3.utmcsr=xxxx.com|utmccn=(referral)|utmcmd=referral|utmcct=/l.php; __gads=ID=269c5cea8f09aea0:T=1319789648:S=ALNI_MaqwdMaIFS5I3FeqLnv86oivdVDuA; __qca=P0-1390452860-1319789648200\r\n\r\n", 745}], 1) = 7457369 epoll_wait(22, {{EPOLLOUT, {u32=2473377616, u64=46963645792080}}}, 512, 17146) = 17369 epoll_wait(22, {{EPOLLIN|EPOLLOUT|EPOLLHUP, {u32=2473377616, u64=46963645792080}}}, 512, 17146) = 17369 recvfrom(31, "", 4096, 0, NULL, NULL) = 0 7369 write(3, "2012/04/28 19:10:30 [error] 7369#0: *21954594 upstream prematurely closed connection while reading response header from upstream, client: 83.9.20.243, server: *.domain.nu, request: \"GET /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06 HTTP/1.1\", upstream: \"http://unix:/var/www/domain/shared/sockets/unicorn.sock:/widget/looks.js?id=1172821&thumbs=3&source=my_looks&nder=both&align=center&r=1320622693&hash=36189f21135680ef be2d076d0b56bb06\", host: \"domain.nu\", referrer: \"http://www.xxx.com/\"\n", 597) = 597 7369 close(31) = 0 7369 writev(20, [{"HTTP/1.1 502 Bad Gateway\r\nServer: nginx\r\nDate: Sat, 28 Apr 2012 23:10:30 GMT\r\nContent-Type: text/html\r\nContent-Length: 166\r\nConnection: close\r\n\r\n", 145}, {"\r\n502 Bad Gateway\r\n\r\n

502 Bad Gateway

\r\n", 120}, {"
nginx
\r\n\r\n\r\n", 46}], 3) = 311 Using the following I was able to get a 200 response req='GET / HTTP/1.1\r\nHost: domain.com\r\n\r\n' printf "$req" | socat - UNIX:/var/www/domain/shared/sockets/unicorn.sock HTTP/1.1 200 OK Date: Sun, 29 Apr 2012 16:29:08 GMT Status: 200 OK Connection: close ETag: "cc3c683b35f20547c69e8155279b3ecf" Content-Type: text/html; charset=utf-8 X-Runtime: 227 Content-Length: 144433 Set-Cookie: _domain_session=BAh7CToKZ2VvaXBGOg9zZXNzaW9uX2lkIiU4NGIzYzkzZjUxNTAwYzUyZDhmYjkzOGM1YjhhNzUxNDoQbW9iaWxlX3ZpZXdGOgtsb2NhbGUiB2Vu--524e4c395bcfa87ca0ce0ab5bfb2f305d4a5f12b; domain=.domain.com; path=/; HttpOnly Cache-Control: private, max-age=0, must-revalidate Sorry but i'm new to debugging Unicorn. Hopefully some of this is helpful. Thanks On Sat, Apr 28, 2012 at 2:59 PM, Eric Wong wrote: > Pico Aeterna wrote: >> Hi, >> >> We've been getting reports that our users have been getting random 502 >> Errors. ?After investigating I noticed 'upstream prematurely closed >> connection while reading response header from upstream' in my nginx >> error logs. ?From using strace it appears that there's numerous socket >> SO_ERROR.. ?I temporarily switched out the domain sockets with TCP, >> however am seeing the same result/issue. >> >> Strace and Nginx error log below: > > Can you show us an strace of a unicorn worker, too? ?(use > "worker_processes 1" when debugging this) > > Also, the unicorn stderr log can be helpful. > >> 7366 ?epoll_ctl(16, EPOLL_CTL_ADD, 89, {EPOLLIN|EPOLLOUT|EPOLLET, >> {u32=2473356496, u64=46963645770960}}) = 0 >> 7366 ?connect(89, {sa_family=AF_FILE, >> path="/var/www/domain/shared/sockets/unicorn.sock"...}, 110) = 0 >> 7366 ?getsockopt(89, SOL_SOCKET, SO_ERROR, [1422340301959200768], [4]) = 0 >> 7366 ?writev(89, [{"GET /natalieliao HTTP/1.0\r\nX-Forwarded-For: >> 94.224.228.239, 94.224.228.239\r\nHost"..., 1084}], 1) = 1084 > > Can you pass "-s 2048" (or larger value) when stracing? ?It'll be > helpful to know what else is in the HTTP header. > >> Load balancer (HaProxy) => nginx => unicorn > > Does hitting unicorn directly with curl (TCP) give an expected result? > (You can use the strace output above to have curl mimic what nginx > sends, or even use printf | socat which works with Unix sockets). > >> Relevant sysctl.conf changes > > Probably not relevant since you switched to Unix sockets, but > consider default sysctl values while debugging (especially tcp_tw_*) > >> net.ipv4.ip_local_port_range = 1024 65535 >> net.ipv4.tcp_fin_timeout = 15 >> #timewait sockets >> net.ipv4.tcp_tw_reuse = 1 >> net.ipv4.tcp_tw_recycle = 1 > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying From normalperson at yhbt.net Sun Apr 29 19:18:25 2012 From: normalperson at yhbt.net (Eric Wong) Date: Sun, 29 Apr 2012 12:18:25 -0700 Subject: Socket errors and nginx 502 gateway errors In-Reply-To: References: <20120428215904.GA6524@dcvr.yhbt.net> Message-ID: <20120429191825.GA14919@dcvr.yhbt.net> Pico Aeterna wrote: > reaped # worker=9 > /var/www/domain/releases/20120427011507/app/controllers/application_controller.rb:172: > [BUG] Segmentation fault OK, something (most likely your app) is causing Ruby to segfault. This is completely unrelated to nginx, so it'll be easier to hit unicorn directly to diagnose this. Since you're new to debugging, I'll try to walk you through dealing with core dumps: * Enable core dumps in the same shell process before starting unicorn (ulimit -c unlimited && unicorn -c /path/to/config ...) * On Linux, /proc/sys/kernel/core_* sysctls control where your core dumps are created. And when you get a core dump (a "core" or core.$PID file), you can run gdb on it: gdb /path/to/ruby /path/to/core Make sure you're using the same ruby binary that unicorn is using. Ruby should be compiled with debugging symbols ('-g' or '-ggdb') by default, but if not, you'll probably want to recompile. > 7369 writev(31, [{"GET > /widget/looks.js?id=1172821&thumbs=3&source=my_looks&gender=both&align=center&r=1320622693&hash=36189f21135680efbe2d076d0b56bb06 > HTTP/1.0\r\nX-Forwarded > -For: 83.9.20.243, 83.9.20.243\r\nHost: domain.nu\r\nConnection: > close\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) > AppleWebKit/534.55.3 (KHTML, like Gecko) Version/5.1.5 > Safari/534.55.3\r\nAccept: */*\r\nReferer: > http://www.xxxxx.com/\r\nAccept-Language: pl-PL\r\nAccept-Encoding: > gzip, deflate\r\nCookie: > __utma=78941577.1641012066.1319789648.1325094074.1325949868.3; > __utmz=78941577.1325949868.3.3.utmcsr=xxxx.com|utmccn=(referral)|utmcmd=referral|utmcct=/l.php; > __gads=ID=269c5cea8f09aea0:T=1319789648:S=ALNI_MaqwdMaIFS5I3FeqLnv86oivdVDuA; > __qca=P0-1390452860-1319789648200\r\n\r\n", 745}], 1) = 7457369 > Using the following I was able to get a 200 response > > req='GET / HTTP/1.1\r\nHost: domain.com\r\n\r\n' > printf "$req" | socat - UNIX:/var/www/domain/shared/sockets/unicorn.sock OK, I think your printf request needs to more closely match what nginx is sending to trigger the segfault. I would add/remove headers/parameters you see above until things start segfaulting. With socat, you may also want to wait a bit for the response for slower endpoints: (printf "$req"; sleep $SECONDS) | socat - ... Otherwise socat will disconnect immediately after its written to the socket without waiting for a response. (a better, but complicated way would be to setup a FIFO as the integration tests in unicorn do). If you're familiar with Rack::Mock*, you may also want to isolate this to just a Rack application. It's likely the segfault can be triggered for your app without unicorn, too. > Sorry but i'm new to debugging Unicorn. Hopefully some of this is helpful. It is helpful. Debugging unicorn isn't different than debugging anything else. The most important is to understand the problem, and that's done by isolating variables to get a small, reproducible test case.