' Application is running on rails 2.3.14/ruby 1.9.2p290 and maybe it has something to do with plugin being used, that is template_streaming (https://github.com/oggy/template_streaming) From normalperson at yhbt.net Tue Sep 6 15:39:23 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 6 Sep 2011 12:39:23 -0700 Subject: Workers getting broken after some time In-Reply-To: References: Message-ID: <20110906193923.GA3998@dcvr.yhbt.net> Kamil Kukura wrote: > I have strange error that happens after couple of hours. Some > instances start to respond on any request with status code 500. Log > file unicorn.stderr.log gets populated with this error: >From the backtrace, it looks like an application error of some sort. Unicorn itself never changes the application once it's loaded. Does an external service your application relies on get overloaded? This is a hunch: perhaps your Rails application is opening connections to a backend and not closing them explicitly (maybe in an error-handling path). > Application is running on rails 2.3.14/ruby 1.9.2p290 and maybe it > has something to do with plugin being used, that is > template_streaming (https://github.com/oggy/template_streaming) What happens when you disable the template_streaming? Streaming isn't likely to help with nginx in front right now (disabling proxy_buffering in nginx is dangerous to unicorn). -- Eric Wong From normalperson at yhbt.net Tue Sep 6 19:27:15 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 6 Sep 2011 16:27:15 -0700 Subject: [TAN] Unix systems programming in Ruby Message-ID: <20110906232715.GA17019@dcvr.yhbt.net> Since some folks interested in Unicorn may be interested in Unix systems programming in general, I'll do some generic Unix systems programming mailing list posts/writeups (not specific to Unicorn). I've started this topic in ruby-talk, but I don't know where it'll end up: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/387469 -- Eric Wong From jhughey at engineyard.com Thu Sep 8 05:04:37 2011 From: jhughey at engineyard.com (J. Austin Hughey) Date: Thu, 8 Sep 2011 03:04:37 -0600 Subject: Sending ABRT to timeout-errant process before KILL Message-ID: <6310128B-282E-40C0-801B-15DBBE9E6AC3@engineyard.com> Hello there - I've recently been working with a customer in my capacity as a support engineer at Engine Yard who's having some trouble with Unicorn. Basically, they're finding their application being force-killed once it reaches the default timeout. Rather than simply increasing the timeout, we're trying to find out _why_ their application is being killed. Unfortunately we can't quite do that because the application is being force-killed without warning, making it so that the customer's logging can't actually be written to the disk. (This is an intermittent issue as opposed to something that happens 100% of the time.) In discussing the matter internally and with our customer, I came up with a simple monkey patch to Unicorn that _sort of_ works, but I'm having some trouble with it once the number of Unicorn workers goes beyond one. I originally limited it to just one worker because I wanted to limit the possibility that multiple workers could cause problems in figuring things out -- turns out I was right. I'm going to show the patch in two ways: 1) inline, at the bottom of this post, and 2) by link to GitHub: https://github.com/jaustinhughey/unicorn/blob/abort_prior_to_kill_workers/lib/unicorn/http_server.rb#L438 The general idea is that I'd like to have some way to "warn" the application when it's about to be killed. I've patched murder_lazy_workers to send an abort signal via kill_worker, sleep for 5 seconds, then look and see if the process still exists by using Process.getpgid. If so, it sends the original kill command, and if not, a rescue block kicks in to prevent the raised error from Process.getpgid from making things explode. I've created a simulation app, built on Rails 3.0, that uses a generic "posts" controller to simulate a long-running request. Instead of just throwing a straight-up sleep 65 in there, I have it running through sleeping 1 second on a decrementing counter, and doing that 65 times. The reason is because, assuming I've read the code correctly, even with my "skip sleeping workers" commented line below, it'll skip over the process, thus rendering my simulation of a long-running process invalid. However, clarification on this point is certainly welcome. You can see the app here: https://github.com/jaustinhughey/unicorn_test/blob/master/app/controllers/posts_controller.rb The problem I'm running into, and where I could use some help, is when I increase the number of Unicorn workers from one to two. When running only one Unicorn worker, I can access my application's posts_controller's index action, which has the intentionally long-running code. At that time I tail -f unicorn.log and production.log. Those two logs look like this with one Unicorn worker: WITH ONE UNICORN WORKER: ======================== production.log: --------------- Sleeping 1 second (65 to go)? ... continued ... Sleeping 1 second (7 to go)... Sleeping 1 second (6 to go)... Sleeping 1 second (5 to go)... Caught Unicorn kill exception! Sleeping 1 second (4 to go)... Sleeping 1 second (3 to go)... Sleeping 1 second (2 to go)... Sleeping 1 second (1 to go)... Completed 500 Internal Server Error in 65131ms NoMethodError (undefined method `query_options' for nil:NilClass): app/controllers/posts_controller.rb:32:in `index' (I think the NoMethodError issue above is due to me calling a disconnect on ActiveRecord in the Signal.trap block, so I think that can be safely ignored.) As you can see, the Signal.trap block inside the aforementioned posts_controller is working in this case. Corresponding log entries in unicorn.log concur: unicorn.log: ------------ worker=0 ready master process ready [2011-09-08 00:31:01 PDT] worker=0 PID: 28921 timeout hit, sending ABRT to process 28921 then sleeping 5 seconds... [2011-09-08 00:31:06 PDT] worker=0 PID:28921 timeout (61s > 60s), killing reaped # worker=0 worker=0 ready So with one worker, everything seems cool. But with two workers? WITH TWO UNICORN WORKERS: ========================= production.log: --------------- Sleeping 1 second (8 to go)... Sleeping 1 second (7 to go)... Sleeping 1 second (6 to go)... Sleeping 1 second (5 to go)... Sleeping 1 second (4 to go)... Sleeping 1 second (3 to go)... Sleeping 1 second (2 to go)... Sleeping 1 second (1 to go)... Rendered posts/index.html.erb within layouts/application (13.2ms) Completed 200 OK in 65311ms (Views: 16.9ms | ActiveRecord: 0.5ms) Note that there is no notice that the ABRT signal was trapped, nor is there a NoMethodError (likely caused by disconnecting from the database) as above. Odd. unicorn.log: ------------ Nothing. No new data whatsoever. The only potential clue I can see at this point would be a start-up message in unicorn.log. After increasing the number of Unicorn workers to two, I examined unicorn.log again and found this: master complete I, [2011-09-08T00:34:40.499437 #29572] INFO -- : unlinking existing socket=/var/run/engineyard/unicorn_ut.sock I, [2011-09-08T00:34:40.499888 #29572] INFO -- : listening on addr=/var/run/engineyard/unicorn_ut.sock fd=5 I, [2011-09-08T00:34:40.504542 #29572] INFO -- : Refreshing Gem list worker=0 ready master process ready [2011-09-08 00:34:49 PDT] worker=1 PID: 29582 timeout hit, sending ABRT to process 29582 then sleeping 5 seconds... [2011-09-08 00:34:50 PDT] worker=1 PID:29582 timeout (1315467289s > 60s), killing reaped # worker=1 worker=1 ready So it looks like Worker 1 is hitting a strange/false timeout of 1315467289 seconds, which isn't really possible as it wasn't even running 1315467289 seconds prior to that (which equates to roughly 41 years ago if my math is right). --- Needless to say, I'm a bit stumped at this point, and would sincerely appreciate another point of view on this. Am I going about this all wrong? Is there a better approach I should consider? And if I'm on the right track, how can I get this to work regardless of how many Unicorn workers are running? Thank you very much for any assistance you can provide! -- INLINE VERSION OF PATCH -- diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index 78d80b4..8a2323f 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -429,6 +429,11 @@ class Unicorn::HttpServer proc_name 'master (old)' end + # A custom formatted timestamp for debugging + def custom_timestamp + return Time.now.strftime("[%Y-%m-%d %T %Z]") + end + # forcibly terminate all workers that haven't checked in in timeout seconds. The timeout is implemented using an unlinked File def murder_lazy_workers t = @timeout @@ -436,16 +441,40 @@ class Unicorn::HttpServer now = Time.now.to_i WORKERS.dup.each_pair do |wpid, worker| tick = worker.tick - 0 == tick and next # skip workers that are sleeping + + # REMOVE THE FOLLOWING COMMENT WHEN TESTING PRODUCTION +# 0 == tick and next # skip workers that are sleeping + # ^ needs to be active, commented here for simulation purposes + diff = now - tick tmp = t - diff if tmp >= 0 next_sleep < tmp and next_sleep = tmp next end - logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \ - "(#{diff}s > #{t}s), killing" - kill_worker(:KILL, wpid) # take no prisoners for timeout violations + + + # Send an ABRT signal to Unicorn and wait 5 seconds before attempting an + # actual kill, if and only if the process is still running. + + begin + # Send the ABRT signal. + logger.debug "#{custom_timestamp} worker=#{worker.nr} PID: #{wpid} timeout hit, sending ABRT to process #{wpid} then sleeping 5 seconds..." + kill_worker(:ABRT, wpid) + + sleep 5 + + # Now see if the process still exists after being given five + # seconds to terminate on its own, and if so, do a hard kill. + if Process.getpgid(wpid) + logger.error "#{custom_timestamp} worker=#{worker.nr} PID:#{wpid} timeout " \ + "(#{diff}s > #{@timeout}s), killing" + kill_worker(:KILL, wpid) # take no prisoners for timeout violations + end + rescue Errno::ESRCH => e + # No process identified - maybe it exited on its own? + logger.debug "#{custom_timestamp} worker=#{worker.nr} PID: #{wpid} responded to ABRT on its own and no longer exists. (Received message: #{e})" + end end next_sleep end -- END INLINE VERSION OF PATCH -- -- J. Austin Hughey Application Support Engineer - Engine Yard www.engineyard.com | jhughey at engineyard.com From normalperson at yhbt.net Thu Sep 8 15:13:52 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 8 Sep 2011 19:13:52 +0000 Subject: Sending ABRT to timeout-errant process before KILL In-Reply-To: <6310128B-282E-40C0-801B-15DBBE9E6AC3@engineyard.com> References: <6310128B-282E-40C0-801B-15DBBE9E6AC3@engineyard.com> Message-ID: <20110908191352.GA25251@dcvr.yhbt.net> "J. Austin Hughey" wrote: > The general idea is that I'd like to have some way to "warn" the > application when it's about to be killed. I've patched > murder_lazy_workers to send an abort signal via kill_worker, sleep for > 5 seconds, then look and see if the process still exists by using > Process.getpgid. If so, it sends the original kill command, and if > not, a rescue block kicks in to prevent the raised error from > Process.getpgid from making things explode. The problem with anything other than SIGKILL (or SIGSTOP) is that it assumes the Ruby VM is working and in a good state. > I've created a simulation app, built on Rails 3.0, that uses a generic > "posts" controller to simulate a long-running request. Instead of > just throwing a straight-up sleep 65 in there, I have it running > through sleeping 1 second on a decrementing counter, and doing that 65 > times. The reason is because, assuming I've read the code correctly, > even with my "skip sleeping workers" commented line below, it'll skip > over the process, thus rendering my simulation of a long-running > process invalid. However, clarification on this point is certainly > welcome. You can see the app here: > https://github.com/jaustinhughey/unicorn_test/blob/master/app/controllers/posts_controller.rb (purely for educational purposes, since I'll point you towards another approach I believe is better) Signal.trap(:ABRT) do # Write some stuff to the Rails log logger.info "Caught Unicorn kill exception!" If this is the logger that ships with Ruby, it locks a Mutex, so it'll deadlock if another SIGABRT is received while logging the above statement (a very small window, admittedly). # Do a controlled disconnect from ActiveRecord ActiveRecord::Base.connection.disconnect! Likewise, if AR needs to lock internal structures before disconnecting, it also must be reentrant. Ruby's normal Mutex implementation is not reentrant-safe. > So it looks like Worker 1 is hitting a strange/false timeout of > 1315467289 seconds, which isn't really possible as it wasn't even > running 1315467289 seconds prior to that (which equates to roughly 41 > years ago if my math is right). You're getting this because you removed the following line: 0 == tick and next # skip workers that are sleeping sleeping means they haven't accepted a client connection, yet. Not sleeping while processing a client request. I'll clarify that in the code. > Needless to say, I'm a bit stumped at this point, and would sincerely > appreciate another point of view on this. Am I going about this all > wrong? Is there a better approach I should consider? And if I'm on > the right track, how can I get this to work regardless of how many > Unicorn workers are running? Since it's an application error, it can be done as middleware. You can try something like the Rainbows::ThreadTimeout middleware, it's currently Rainbows! specific but can easily be made to work with Unicorn. git clone git://bogomips.org/rainbows cat rainbows/lib/rainbows/thread_timeout.rb This is conceptually similar to "timeout" in the Ruby standard library, but does not allow nesting. I'll try to clarify more later today if you have questions, in a bit of a rush right now. -- Eric Wong From normalperson at yhbt.net Thu Sep 8 21:36:29 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 8 Sep 2011 18:36:29 -0700 Subject: [TAN] Unix systems programming in Ruby In-Reply-To: <20110906232715.GA17019@dcvr.yhbt.net> References: <20110906232715.GA17019@dcvr.yhbt.net> Message-ID: <20110909013629.GC20151@dcvr.yhbt.net> Eric Wong wrote: > I've started this topic in ruby-talk, but I don't know where it'll end up: It'll be a separate list at usp.ruby at librelist.org A mailing list dedicated to Unix systems programming in Ruby ============================================================ Original discussion/introduction about this list started in ruby-talk: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/387469 The "separate mailing list" mentioned is usp.ruby at librelist.org Subscribe --------- To subscribe, send an email to usp.ruby at librelist.org and follow the directions in the response to confirm your subscription. This mailing list will officially start around the middle of September 2011. You may (and are encouraged to) subscribe before then. Mailing list rules ------------------ * Be nice to each other * Do not [top post][1] when replying, quote only what you reply to (or don't quote at all) * Text only; no HTML or attachments Mailing list archives --------------------- You will find yearly and monthly mbox archives in the "archives/" directory Nearly all mail user agents can open or import mbox format files. Mutt users can run the following command: mutt -f /path/to/downloaded/mbox [1]: http://catb.org/jargon/html/T/top-post.html -- Eric Wong From mohit.chawla.binary at gmail.com Fri Sep 9 06:26:04 2011 From: mohit.chawla.binary at gmail.com (Mohit Chawla) Date: Fri, 9 Sep 2011 15:56:04 +0530 Subject: [TAN] Unix systems programming in Ruby In-Reply-To: <20110909013629.GC20151@dcvr.yhbt.net> References: <20110906232715.GA17019@dcvr.yhbt.net> <20110909013629.GC20151@dcvr.yhbt.net> Message-ID: This is cool, thanks. From normalperson at yhbt.net Fri Sep 9 19:01:41 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 9 Sep 2011 16:01:41 -0700 Subject: Sending ABRT to timeout-errant process before KILL In-Reply-To: <20110908191352.GA25251@dcvr.yhbt.net> References: <6310128B-282E-40C0-801B-15DBBE9E6AC3@engineyard.com> <20110908191352.GA25251@dcvr.yhbt.net> Message-ID: <20110909230141.GA9521@dcvr.yhbt.net> Eric Wong wrote: > sleeping means they haven't accepted a client connection, yet. Not > sleeping while processing a client request. I'll clarify that in the > code. Pushed to git://bogomips.org/unicorn.git (commit d209910e29d4983f8346233262a49541464252c1) > git clone git://bogomips.org/rainbows > cat rainbows/lib/rainbows/thread_timeout.rb > > This is conceptually similar to "timeout" in the Ruby standard library, > but does not allow nesting. > > I'll try to clarify more later today if you have questions, in a bit of > a rush right now. Did you manage to get anything going based on this? I should add that this has the same chance of working as a SIGABRT handler written in Ruby (but is less intrusive). Hopefully the following diagram/chart from a previous post can explain why you can't rely on trappable signal handlers if the Ruby VM is in a bad state: http://mid.gmane.org/20110817201323.GA24581 at dcvr.yhbt.net -- Eric Wong From normalperson at yhbt.net Fri Sep 9 19:15:11 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 9 Sep 2011 16:15:11 -0700 Subject: [PATCH] Links: add a link to the UnXF middleware Message-ID: <20110909231511.GA11538@dcvr.yhbt.net> Since unicorn is designed to be deployed behind nginx (or similar), X-Forwarded-* headers are common and Rack applications may blindly trust spoofed X-Forwarded-* headers. UnXF provides a central place for managing that trust by using rpatricia. --- Pushed to git://bogomips.org/unicorn.git (commit db2cba26acc5748bcf9919e3184a667c46911f8c) and updated http://unicorn.bogomips.org/Links.html Links | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Links b/Links index e7f5e60..d78d00a 100644 --- a/Links +++ b/Links @@ -26,6 +26,9 @@ or services behind them. * {raindrops}[http://raindrops.bogomips.org/] - real-time stats for preforking Rack servers +* {UnXF}[http://bogomips.org/unxf/] Un-X-Forward* the Rack environment, + useful since unicorn is designed to be deployed behind a reverse proxy. + === \Unicorn is written to work with * {Rack}[http://rack.rubyforge.org/] - a minimal interface between webservers -- Eric Wong From gswallow at exacttarget.com Mon Sep 12 11:42:24 2011 From: gswallow at exacttarget.com (Greg Swallow) Date: Mon, 12 Sep 2011 11:42:24 -0400 Subject: Monitoring Unicorn processes with SNMP Message-ID: Hi, If I try to monitor Unicorn's processes with SNMP, I get this: snmpwalk -v2c -c public -Oa app7.org.org hrSWRun | grep 24051 HOST-RESOURCES-MIB::hrSWRunIndex.24051 = INTEGER: 24051 HOST-RESOURCES-MIB::hrSWRunName.24051 = STRING: "ruby" HOST-RESOURCES-MIB::hrSWRunID.24051 = OID: SNMPv2-SMI::zeroDotZero HOST-RESOURCES-MIB::hrSWRunParameters.24051 = STRING: " " HOST-RESOURCES-MIB::hrSWRunType.24051 = INTEGER: application(4) HOST-RESOURCES-MIB::hrSWRunStatus.24051 = INTEGER: runnable(2) That's not too helpful. :) Is there somewhere I can set a parameter that shows up in the hrSWRunParameters OID? (This is OID .1.3.6.1.2.1.25.4.2.1.5., though I'm really asking an OS-level question in general.) Also, I don't subscribe to this list; please reply to my e-mail address as well. Thanks! ExactTarget Greg Swallow System Administrator Phone | 317 524-5250 Yahoo IM | gswallow01 Email | gswallow at exacttarget.com From normalperson at yhbt.net Mon Sep 12 15:41:14 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 12 Sep 2011 19:41:14 +0000 Subject: Monitoring Unicorn processes with SNMP In-Reply-To: References: Message-ID: <20110912194114.GA25479@dcvr.yhbt.net> Greg Swallow wrote: > Is there somewhere I can set a parameter that shows up in the > hrSWRunParameters OID? (This is OID .1.3.6.1.2.1.25.4.2.1.5., though > I'm really asking an OS-level question in general.) Hopefully other folks here can help, I lack experience with SNMP setup. > Also, I don't subscribe to this list; please reply to my e-mail > address as well. Thanks for the heads up :) -- Eric Wong From mail-1 at pr-mail.daynight.jp Tue Sep 13 05:17:51 2011 From: mail-1 at pr-mail.daynight.jp (=?ISO-2022-JP?B?GyRCNWE/TT5wSnMkTi1qJVMlOCVNJTklSCVpJTklSCU3JTklRiVgJTobKEI=?=) Date: 13 Sep 2011 18:17:51 +0900 Subject: =?ISO-2022-JP?B?GyRCOl9CcCVvITwlKyE8NV5KZ0IuSnMbKEIhGyRCIVolRyE8JT9Gfk5PNkhMM0JnTkw8dUNtJE4kPyRhIVsheSVhJWslXiUsGyhCU09ITxskQkA4M2glSyVlITwlORsoQg==?= Message-ID: <20110913091751.13712.qmail@www9.bizmail.jp> ??????SOHO??????013,439??2011?9?? ??????:79,652? ????????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ?1? ????SOHO???????????????????????????????? ?PR???????????????????????????????? ?????????????????! SOHO???????????????! ????????????????????!!???????????????????! ????????????????????????????????!? ??? !! ?????????? ? http://community.rulez.jp/ ?PR???????????????????????????????? ???????9??????????????? ?????????????????????????? ??SOHO??????????3??????????????? ?????????????????????????? ?1????????????????! ????????????! ????????????????????????????? ?2?????????????????????????! ??????! ?SOHO????????????????! ?3??????????????????????!?????! ????????????????????! ? ? ????????? ? ???????! ??????????!! ? ? ????? http://community.rulez.jp/ ???????????????? ????????????????? ?????????????????????????? ????SOHO??????!! ?????????????? 1.???? ? 18,500 ? 5,550? 2.?????????????? ? 10,550 ? 45,900? 3.????????? ? 1,000 ? 31,531? 4.????????? 6 ? 42,000? 5.?????????? 4 ? 13,500? 6.???????????????? ? 90 ? 45,000? ????????183,481????? ???????????????????(1?2?4????)???????!! ? http://community.rulez.jp/ ? ?(??????????) ????????????????????????? ???????????????! ????????????!! ??????????????????????????????????? ?????????????????????????SOHO?????????? ?????????????????????????????????????????? ??????????????????????????????????? ??????????????? ??????????????????????????????????????? ?????????????????????? ?????????????????????????????????? ??????????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ??????????????????????????????????? ??????????????)??????????????Web?????????? ??????????????????????????????????? Copyright (c) 1999 ???????????????????????All rights reserved.??????? From bpo at somnambulance.net Wed Sep 14 10:29:05 2011 From: bpo at somnambulance.net (Brian P O'Rourke) Date: Wed, 14 Sep 2011 22:29:05 +0800 Subject: sigwinch and screen In-Reply-To: References: Message-ID: (my apologies if?this gets posted twice - don't think my earlier mail went through) Hi, I'm running unicorn within screen and have run into some trouble. Unicorn tries to ignore sigwinch when it's not daemonized. Unfortunately the check for daemonization fails when unicorn is exec'd from a process that already has a pgrp. You can simulate this problem from *within* a screen session with something like this: screen sleep 5 && screen bundle exec unicorn -c unicorn.conf config.ru sigwinch will be sent to the second screen session the first time you change to it. This doesn't happen unless you launch at least two screens in rapid succession, for reasons that escape me so far - haven't dug into that part much yet. It seems to me that this is a problem, and is occurring because Unicorn's check for daemonization is "is init my parent or is my group different from my pid?", which isn't necessarily the same as checking whether daemonization has happened. Here is the patch I would like to see applied: diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index ae0e175..65880d4 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -282,7 +282,7 @@ class Unicorn::HttpServer when :USR2 # exec binary, stay alive in case something went wrong reexec when :WINCH - if Process.ppid == 1 || Process.getpgrp != $$ + if Unicorn::Configurator::RACKUP[:daemonized] respawn = false logger.info "gracefully stopping all workers" kill_each_worker(:QUIT) From normalperson at yhbt.net Wed Sep 14 13:32:13 2011 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 14 Sep 2011 17:32:13 +0000 Subject: sigwinch and screen In-Reply-To: References:

Message-ID: <20110914173213.GA31955@dcvr.yhbt.net> Brian P O'Rourke wrote: > (my apologies if?this gets posted twice - don't think my earlier mail > went through) I've noticed rubyforge has occasional slowness :< It uses Postgrey so first time posters could be delayed... > I'm running unicorn within screen and have run into some trouble. > Unicorn tries to ignore sigwinch when it's not daemonized. > Unfortunately the check for daemonization fails when unicorn is exec'd > from a process that already has a pgrp. > > You can simulate this problem from *within* a screen session with > something like this: > > screen sleep 5 && screen bundle exec unicorn -c unicorn.conf config.ru So, start "screen", get a terminal + shell, /then/ type the above? > sigwinch will be sent to the second screen session the first time you > change to it. This doesn't happen unless you launch at least two > screens in rapid succession, for reasons that escape me so far - > haven't dug into that part much yet. I haven't been able to reproduce it on my end. It could be system-dependent (terminals are often wonky/inconsistent). > It seems to me that this is a problem, and is occurring because > Unicorn's check for daemonization is "is init my parent or is my group > different from my pid?", which isn't necessarily the same as checking > whether daemonization has happened. Agreed. Regardless of what the problem is, I like your check based on RACKUP[:daemonized]. It's much more readable and obvious :) > Here is the patch I would like to see applied: I'll apply it. Is there a commit message you'd like to use? (Otherwise I'll just edit something based on your email) Thanks! > --- a/lib/unicorn/http_server.rb > +++ b/lib/unicorn/http_server.rb > @@ -282,7 +282,7 @@ class Unicorn::HttpServer > when :USR2 # exec binary, stay alive in case something went wrong > reexec > when :WINCH > - if Process.ppid == 1 || Process.getpgrp != $$ > + if Unicorn::Configurator::RACKUP[:daemonized] > respawn = false > logger.info "gracefully stopping all workers" > kill_each_worker(:QUIT) -- Eric Wong From bpo at somnambulance.net Wed Sep 14 20:44:08 2011 From: bpo at somnambulance.net (Brian P O'Rourke) Date: Thu, 15 Sep 2011 08:44:08 +0800 Subject: sigwinch and screen In-Reply-To: <20110914173213.GA31955@dcvr.yhbt.net> References:

<20110914173213.GA31955@dcvr.yhbt.net> Message-ID: On Thu, Sep 15, 2011 at 1:32 AM, Eric Wong wrote: > Brian P O'Rourke wrote: >> You can simulate this problem from *within* a screen session with >> something like this: >> >> ? ? screen sleep 5 && screen bundle exec unicorn -c unicorn.conf config.ru > > So, start "screen", get a terminal + shell, /then/ type the above? > Correct. When running screen within screen like this, you just get a new screen that executes the command. It happens consistently for me on OSX when I launch several screens at once and the last active screen is *not* unicorn - could certainly be system-dependent. >> Here is the patch I would like to see applied: > > I'll apply it. ? Is there a commit message you'd like to use? > (Otherwise I'll just edit something based on your email) You can just pull from my public fork here: git://github.com/bpo/unicorn.git - branch name is 'daemonization_detection' Cheers, Brian P O'Rourke From normalperson at yhbt.net Wed Sep 14 20:54:52 2011 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 14 Sep 2011 17:54:52 -0700 Subject: sigwinch and screen In-Reply-To: References:

<20110914173213.GA31955@dcvr.yhbt.net> Message-ID: <20110915005452.GA10238@dcvr.yhbt.net> Brian P O'Rourke wrote: > You can just pull from my public fork here: > git://github.com/bpo/unicorn.git - branch name is > 'daemonization_detection' Acked and pushed out as commit b48c6659b294b37f2c6ff3e75c1c9245522d48d1 Thanks! From normalperson at yhbt.net Thu Sep 15 18:46:09 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 15 Sep 2011 22:46:09 +0000 Subject: SSL support pushed out to unicorn.git :x Message-ID: <20110915224608.GA21568@dcvr.yhbt.net> Consider this a joke until somebody who knows crypto well can review it (_and_ kgio-monkey[1]). I know it works with curl (test case included) and I can't see plain-text when I strace/tcpdump, that's about it :) Subject: [PATCH] add preliminary SSL support This will also be the foundation of SSL support in Rainbows! and Zbatery. Some users may also want to use this in Unicorn on LANs to meet certain security/auditing requirements. Of course, Nightmare! (in whatever form) should also be able to use it. --- The patch is a big, so you can view it here http://bogomips.org/unicorn.git/patch?id=ac346b5abc [1] - http://bogomips.org/kgio-monkey/ git clone git://bogomips.org/kgio-monkey.git This is absolutely NOT intended to be an endorsement of the current certificate authority system. Don't support or encourage it. lib/unicorn/configurator.rb | 13 +++-- lib/unicorn/http_server.rb | 3 + lib/unicorn/ssl_client.rb | 6 ++ lib/unicorn/ssl_configurator.rb | 104 +++++++++++++++++++++++++++++++++++++++ lib/unicorn/ssl_server.rb | 42 ++++++++++++++++ script/isolate_for_tests | 1 + t/.gitignore | 2 + t/sslgen.sh | 63 +++++++++++++++++++++++ t/t0600-https-server-basic.sh | 48 ++++++++++++++++++ test/unit/test_sni_hostnames.rb | 47 +++++++++++++++++ 10 files changed, 325 insertions(+), 4 deletions(-) create mode 100644 lib/unicorn/ssl_client.rb create mode 100644 lib/unicorn/ssl_configurator.rb create mode 100644 lib/unicorn/ssl_server.rb create mode 100755 t/sslgen.sh create mode 100755 t/t0600-https-server-basic.sh create mode 100644 test/unit/test_sni_hostnames.rb -- Eric Wong From russm-rubyforge at slofith.org Fri Sep 16 06:19:51 2011 From: russm-rubyforge at slofith.org (russell muetzelfeldt) Date: Fri, 16 Sep 2011 20:19:51 +1000 Subject: Rainbows! or unicorn? Message-ID: I'm putting together a small web frontend for a client to upload files into an existing application. It's trivial - there will never be more than a (small) handful of concurrent connections, but I need a streaming rack.input for upload progress on files up to 500MB or so. I was planning on using Rainbows! with ThreadSpawn and worker_connections=1, then noticed that unicorn is also listed as having streaming rack.input. While what I'm doing is pretty much the opposite of the unicorn design case, is there any reason in this scenario for me to use Rainbows!, or should I just go with unicorn and enough backends to handle a couple of concurrent uploads and the minimal other frontend bits? cheers Russell ----- Russell Muetzelfeldt Mundus vult decipi, ergo decipiatur. From normalperson at yhbt.net Fri Sep 16 07:11:07 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 16 Sep 2011 11:11:07 +0000 Subject: Rainbows! or unicorn? In-Reply-To: References: Message-ID: <20110916111107.GA15421@dcvr.yhbt.net> russell muetzelfeldt wrote: > I'm putting together a small web frontend for a client to upload files > into an existing application. It's trivial - there will never be more > than a (small) handful of concurrent connections, but I need a > streaming rack.input for upload progress on files up to 500MB or so. I > was planning on using Rainbows! with ThreadSpawn and > worker_connections=1, then noticed that unicorn is also listed as > having streaming rack.input. :ThreadSpawn + worker_connections=1 and the (default) :Base option are almost the same in Rainbows! if you don't want to worry about your app being thread-safe at all. > While what I'm doing is pretty much the opposite of the unicorn design > case, is there any reason in this scenario for me to use Rainbows!, or > should I just go with unicorn and enough backends to handle a couple > of concurrent uploads and the minimal other frontend bits? Rainbows! can (and does by default) limit upload sizes (client_max_body_size) for handling untrusted clients who may try to run you out of space. Since performance/scalability isn't your concern, it depends on whether you trust your clients to not upload until you run out of disk space. -- Eric Wong From russm-rubyforge at slofith.org Fri Sep 16 08:38:59 2011 From: russm-rubyforge at slofith.org (russell muetzelfeldt) Date: Fri, 16 Sep 2011 22:38:59 +1000 Subject: Rainbows! or unicorn? In-Reply-To: <20110916111107.GA15421@dcvr.yhbt.net> References: <20110916111107.GA15421@dcvr.yhbt.net> Message-ID: On 16/09/2011, at 9:11 PM, Eric Wong wrote: > :ThreadSpawn + worker_connections=1 and the (default) :Base option are > almost the same in Rainbows! if you don't want to worry about your app > being thread-safe at all. my reason for intending to run worker_connections=1 isn't avoiding threading issues, but to allow app reloads without leaving around old instances of the app in server processes that are handling a 2+ hour upload. I passed on Rainbows::Base just because of the "not intended for production use" comment in the docs. > Rainbows! can (and does by default) limit upload sizes > (client_max_body_size) for handling untrusted clients who may try to > run you out of space. ah, that's a good point. I'll be fronting through Pound for SSL, but having the server process limit upload size would be worthwhile. > Since performance/scalability isn't your concern, it depends on whether > you trust your clients to not upload until you run out of disk space. I trust them to not do that on purpose, but that doesn't mean a lot. :/ cheers Russell ----- Russell Muetzelfeldt Mundus vult decipi, ergo decipiatur. From normalperson at yhbt.net Fri Sep 16 13:00:12 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 16 Sep 2011 17:00:12 +0000 Subject: Rainbows! or unicorn? In-Reply-To: References: <20110916111107.GA15421@dcvr.yhbt.net> Message-ID: <20110916170012.GA18588@dcvr.yhbt.net> russell muetzelfeldt wrote: > On 16/09/2011, at 9:11 PM, Eric Wong wrote: > > :ThreadSpawn + worker_connections=1 and the (default) :Base option are > > almost the same in Rainbows! if you don't want to worry about your app > > being thread-safe at all. > I passed on Rainbows::Base just because of the "not intended > for production use" comment in the docs. Ah, Base and worker_connections=1 (anywhere) is risky for dealing with untrusted clients directly. From leehambley at me.com Tue Sep 20 04:18:21 2011 From: leehambley at me.com (Lee Hambley) Date: Tue, 20 Sep 2011 10:18:21 +0200 Subject: PID file ownership and group Message-ID: <6F8DA070-5CC5-4107-8706-34D3F61D57C2@me.com> I'm using unicorn in an environment with /very/ strict permissions (one might so as far as to say that the sysadmin is being too careful) and I've observed that when starting Unicorn via `upstart` (runs as root) with unicorn.rb configured to suid and sguid, the logs and other files are correctly owned by `selected user:group` but the pidfile is owned by root:root. Owing to very restrictive unmasking and other permissions, this file is not readable by any lower-level users, and thus one has to be root to read the pidfile. What's the logic here, is it a bug, an oversight or an intentional design, naturally one can use `ps` or any other number of ways to get a pid, so protecting the pidfile doesn't seem like a security concern/ Of course this is somewhat academic, as one must be root to signal the process anyway, but I'll cross that particular bridge when I come to it! Lee From normalperson at yhbt.net Tue Sep 20 14:03:14 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 20 Sep 2011 18:03:14 +0000 Subject: PID file ownership and group In-Reply-To: <6F8DA070-5CC5-4107-8706-34D3F61D57C2@me.com> References: <6F8DA070-5CC5-4107-8706-34D3F61D57C2@me.com> Message-ID: <20110920180314.GA30482@dcvr.yhbt.net> Lee Hambley wrote: > I'm using unicorn in an environment with /very/ strict permissions > (one might so as far as to say that the sysadmin is being too careful) > and I've observed that when starting Unicorn via `upstart` (runs as > root) with unicorn.rb configured to suid and sguid, the logs and other > files are correctly owned by `selected user:group` but the pidfile is > owned by root:root. Owing to very restrictive unmasking and other > permissions, this file is not readable by any lower-level users, and > thus one has to be root to read the pidfile. > > What's the logic here, is it a bug, an oversight or an intentional > design, naturally one can use `ps` or any other number of ways to get > a pid, so protecting the pidfile doesn't seem like a security concern/ I didn't think of it until now, but I don't think it's necessary to expose. With the exception of creating Unix domain sockets, unicorn always respects the umask given to it, and unicorn only changes permissions inside worker processes. Also, nginx (where unicorn draws many design elements from) does not change the permissions of the pid file, either. > Of course this is somewhat academic, as one must be root to signal the > process anyway, but I'll cross that particular bridge when I come to > it! Yeah, there's no point in knowing it unless you can send signals to it. -- Eric Wong From normalperson at yhbt.net Tue Sep 20 16:23:37 2011 From: normalperson at yhbt.net (Eric Wong) Date: Tue, 20 Sep 2011 13:23:37 -0700 Subject: PID file ownership and group In-Reply-To: References: <6F8DA070-5CC5-4107-8706-34D3F61D57C2@me.com> <20110920180314.GA30482@dcvr.yhbt.net> Message-ID: <20110920202337.GA6965@dcvr.yhbt.net> Lee Hambley wrote: > > Also, nginx (where unicorn draws many design elements from) does not > > change the permissions of the pid file, either. > > > >> Of course this is somewhat academic, as one must be root to signal the > >> process anyway, but I'll cross that particular bridge when I come to > >> it! > > > > Yeah, there's no point in knowing it unless you can send signals to it. > Surely though, with the pid being root:root when started via upstart > (and the restrictive u+rwx/g+rwx/o-) permissions enforced by my umask, > it would still make sense to own the pid to `unicorn:projectname` - > given that you can signal your own processes, then the "unicorn" user > should be able to read the pid? Not really, the pid file is only for the master process. The master always stays as the user it is started as (root in your case). Only the worker processes have their user:group changed and they can't signal the master after the change. > (Although, that said, most implementations I have seen of "monitors" > for unicorn seem to use `ps` to get the pid, which does seem somewhat > wasteful) Strange, wouldn't the monitors /not/ daemonize and thus be able to use the pid of the process they started? > Thanks for getting back to me so quickly on here Eric, with Github I'd > almost forgotten that mailing lists work! No problem. Signups/logins/passwords bother me, and I also dislike 99.999% of HTML, images, and JavaScript on the web; so I'll be sticking to mailing lists :) -- Eric Wong From joe at ownlocal.com Wed Sep 21 19:42:29 2011 From: joe at ownlocal.com (Joe Marty) Date: Wed, 21 Sep 2011 18:42:29 -0500 Subject: http_response seems to be using String.each Message-ID: I just installed Unicorn 4.1.1 to serve a Rails 3 project behind nginx on a server running Ruby 1.9.2 and I'm getting this error on every app request: app error: undefined method `each' for # (NoMethodError) /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_response.rb:41:in `http_response_write' I don't know for sure if 'body' is supposed to be an array, or if it's correctly a string and Unicorn is actually trying to read the string 1 line at a time using the each method, which was changed in ruby 1.9 to 'String.each_line'. Any ideas? The rails app does work fine in Passenger, and if I just swap out passenger for Unicorn, it fails with that message. -Joe From normalperson at yhbt.net Wed Sep 21 20:46:13 2011 From: normalperson at yhbt.net (Eric Wong) Date: Wed, 21 Sep 2011 17:46:13 -0700 Subject: http_response seems to be using String.each In-Reply-To: References: Message-ID: <20110922004613.GA16936@dcvr.yhbt.net> Joe Marty wrote: > I just installed Unicorn 4.1.1 to serve a Rails 3 project behind nginx > on a server running Ruby 1.9.2 and I'm getting this error on every app > request: > > app error: undefined method `each' for # (NoMethodError) > /usr/local/lib/ruby/gems/1.9.1/gems/unicorn-4.1.1/lib/unicorn/http_response.rb:41:in > `http_response_write' I suspect you have some buggy middleware that's returning Strings as the body (instead of Strings inside Arrays or something else) What middlewares do you have loaded? You can insert Rack::Lint in between every middleware to detect buggy ones. If Rack::Lint slows you down too much, you can also try just removing them one-at-a-time to isolate the buggy one. > I don't know for sure if 'body' is supposed to be an array, or if it's > correctly a string and Unicorn is actually trying to read the string 1 > line at a time using the each method, which was changed in ruby 1.9 to > 'String.each_line'. Any ideas? The Rack spec only requires the response body respond to the "each" method and that yields String (or String-like) objects. > The rails app does work fine in Passenger, and if I just swap out > passenger for Unicorn, it fails with that message. Maybe Passenger is more lenient with the Rack specs and/or working around bugs. -- Eric Wong From ajsharp at gmail.com Sun Sep 25 21:24:44 2011 From: ajsharp at gmail.com (Alex Sharp) Date: Sun, 25 Sep 2011 18:24:44 -0700 Subject: Timeout callback Message-ID: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> Would there be any support for a worker-level timeout callback, for workers that get killed by the master process for violating the Unicorn::Configurator.timeout setting? I'm thinking the method could be on the Unicorn::Configurator class, something like ".at_timeout_exit". My thinking here is I want to be able to invoke caller() and get a backtrace to figure out the code that's resulting in timeouts. -- Alex Sharp github.com/ajsharp twitter.com/ajsharp alexjsharp.com From normalperson at yhbt.net Sun Sep 25 21:42:59 2011 From: normalperson at yhbt.net (Eric Wong) Date: Mon, 26 Sep 2011 01:42:59 +0000 Subject: Timeout callback In-Reply-To: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> References: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> Message-ID: <20110926014259.GB19028@dcvr.yhbt.net> Alex Sharp wrote: > Would there be any support for a worker-level timeout callback, for > workers that get killed by the master process for violating the > Unicorn::Configurator.timeout setting? Something like this /cannot/ be done right. The unicorn timeout uses SIGKILL because SIGKILL is a last resort and not catchable/blockable/trappable in user space. (SIGSTOP is in the same boat as SIGKILL). > I'm thinking the method could be on the Unicorn::Configurator class, > something like ".at_timeout_exit". My thinking here is I want to be > able to invoke caller() and get a backtrace to figure out the code > that's resulting in timeouts. Getting a backtrace relies on Ruby being in a runnable state. If user space (and Ruby) is capable of accepting non-SIGKILL/SIGSTOP, you could already be using something along the lines of the Timeout module in Ruby stdlib, SystemTimeout, or the Rainbows::ThreadTimeout middleware. In other words, you can already use an application-level timeout (even around the entire app dispatch) if you could get a backtrace. From chris at cobaltedge.com Mon Sep 26 02:06:10 2011 From: chris at cobaltedge.com (Christopher Bailey) Date: Sun, 25 Sep 2011 23:06:10 -0700 Subject: Timeout callback In-Reply-To: <20110926014259.GB19028@dcvr.yhbt.net> References: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> <20110926014259.GB19028@dcvr.yhbt.net> Message-ID: Alex, we were having problems with timeouts, and it killing our logs, making it near impossible to figure out what was causing the timeout, etc. We too looked into a solution within Unicorn, but as Eric explains, it's not possible. What we wound up doing, which may or may not work for you, is to put an explicit Timeout wrapper around the code we knew caused this (we are lucky and knew a specific API call/controller action that was getting the timeouts). ?e.g. wrap with: ? Timeout::timeout(...) do This wound up actually being a far better solution for us, because our mobile clients that call this API timeout the HTTP request at 20 seconds anyway, so it was pointless to even get to 60. ?So, we now have a far better solution, one where we can time it out ourselves, handle the exception, and log the timeout/problem (e.g. we create a Zendesk ticket, etc.). ?Anyway, figured I'd mention that in case it could work in your case as well... On Sun, Sep 25, 2011 at 6:42 PM, Eric Wong wrote: > > Alex Sharp wrote: > > ?Would there be any support for a worker-level timeout callback, for > > ?workers that get killed by the master process for violating the > > ?Unicorn::Configurator.timeout setting? > > Something like this /cannot/ be done right. ?The unicorn timeout uses > SIGKILL because SIGKILL is a last resort and not > catchable/blockable/trappable in user space. ?(SIGSTOP is in the same > boat as SIGKILL). > > > I'm thinking the method could be on the Unicorn::Configurator class, > > something like ".at_timeout_exit". My thinking here is I want to be > > able to invoke caller() and get a backtrace to figure out the code > > that's resulting in timeouts. > > Getting a backtrace relies on Ruby being in a runnable state. > If user space (and Ruby) is capable of accepting non-SIGKILL/SIGSTOP, > you could already be using something along the lines of the Timeout > module in Ruby stdlib, SystemTimeout, or the Rainbows::ThreadTimeout > middleware. > > In other words, you can already use an application-level timeout > (even around the entire app dispatch) if you could get a backtrace. > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying -- Christopher Bailey Cobalt Edge LLC http://cobaltedge.com From ajsharp at gmail.com Mon Sep 26 15:02:52 2011 From: ajsharp at gmail.com (Alex Sharp) Date: Mon, 26 Sep 2011 12:02:52 -0700 Subject: Timeout callback In-Reply-To: <20110926014259.GB19028@dcvr.yhbt.net> References: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> <20110926014259.GB19028@dcvr.yhbt.net> Message-ID: <7FA6B098354247599A7DACDBC6F28EAF@gmail.com> Makes sense. Thanks for the explanation -- I'm pretty sure this exact topic has been discussed a few times on this list ;) - Alex From ajsharp at gmail.com Mon Sep 26 15:04:04 2011 From: ajsharp at gmail.com (Alex Sharp) Date: Mon, 26 Sep 2011 12:04:04 -0700 Subject: Timeout callback In-Reply-To: References: <4D4841AA1FE04C98BF883BDE8B17741E@gmail.com> <20110926014259.GB19028@dcvr.yhbt.net> Message-ID: <74BCCFBDF6C4414382DC540C16B2301B@gmail.com> On Sunday, September 25, 2011 at 11:06 PM, Christopher Bailey wrote: > What we wound up doing, which may or may not work for you, is to put > an explicit Timeout wrapper around the code we knew caused this (we > are lucky and knew a specific API call/controller action that was > getting the timeouts). e.g. wrap with: > > Timeout::timeout(...) do > > This wound up actually being a far better solution for us, because our > mobile clients that call this API timeout the HTTP request at 20 > seconds anyway, so it was pointless to even get to 60. So, we now > have a far better solution, one where we can time it out ourselves, > handle the exception, and log the timeout/problem (e.g. we create a > Zendesk ticket, etc.). Anyway, figured I'd mention that in case it > could work in your case as well... Great, thanks for sharing. - Alex From mail-1 at team66.thyme.jp Tue Sep 27 06:15:11 2011 From: mail-1 at team66.thyme.jp (=?ISO-2022-JP?B?GyRCNWE/TT5wSnMkTi1qJVMlOCVNJTklSCVpJTklSCU3JTklRiVgJTobKEI=?=) Date: 27 Sep 2011 19:15:11 +0900 Subject: =?ISO-2022-JP?B?GyRCOl9CcCVvITwlKyE8NV5KZ0IuSnMbKEIhGyRCIVolRyE8JT9Gfk5PNkhMM0JnTkw8dUNtJE4kPyRhIVsheSVhJWslXiUsGyhCU09ITxskQkA4M2glSyVlITwlORsoQg==?= Message-ID: <20110927101511.6270.qmail@www20.bizmail.jp> ??????SOHO??????13,447??2011?9?? ??????:56,725? ???????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ?1? ????SOHO???????????????????????????????? ?PR???????????????????????????????? ?????????????????! SOHO???????????????! ????????????????????!!???????????????????! ????????????????????????????????!? ??? !! ?????????? ? http://community.rulez.jp/ ?PR???????????????????????????????? ???????9??????????????? ?????????????????????????? ??SOHO??????????3??????????????? ?????????????????????????? ?1????????????????! ????????????! ????????????????????????????? ?2?????????????????????????! ??????! ?SOHO????????????????! ?3??????????????????????!?????! ????????????????????! ? ? ????????? ? ???????! ??????????!! ? ? ????? http://community.rulez.jp/ ???????????????? ????????????????? ?????????????????????????? ????SOHO??????!! ?????????????? 1.???? ? 18,500 ? 5,550? 2.?????????????? ? 10,550 ? 45,900? 3.????????? ? 1,000 ? 31,531? 4.????????? 6 ? 42,000? 5.?????????? 4 ? 13,500? 6.???????????????? ? 90 ? 45,000? ????????183,481????? ???????????????????(1?2?4????)???????!! ? http://community.rulez.jp/ ? ?(??????????) ????????????????????????? ???????????????! ????????????!! ??????????????????????????????????? ?????????????????????????SOHO?????????? ?????????????????????????????????????????? ??????????????????????????????????? ??????????????? ??????????????????????????????????????? ?????????????????????? ?????????????????????????????????? ??????????????????????????? ???????????????????????????????????????????? ???????? Please e-mail the delivery stop here.? kyo at biz-career.rulez.jp ??????????????????????????????????? ??????????????)??????????????Web?????????? ??????????????????????????????????? Copyright (c) 1999 ???????????????????????All rights reserved.??????? From j at jjb.cc Thu Sep 29 14:28:54 2011 From: j at jjb.cc (John Joseph Bachir) Date: Thu, 29 Sep 2011 14:28:54 -0400 Subject: large uploads Message-ID: My application accepts uploads from users, which can be quite huge in some cases. This of course requires setting the unicorn timeout to something much higher than 60 seconds, more like 10 minutes. Are there any drawbacks to doing this, other than the obvious drawback of not killing off long-running requests that are illegitimate? I've googled quite a bit about this and have found surprisingly little -- I guess people who have apps that receive uploads just generally don't use unicorn? Thanks, John From normalperson at yhbt.net Thu Sep 29 14:47:11 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 29 Sep 2011 18:47:11 +0000 Subject: large uploads In-Reply-To: References: Message-ID: <20110929184711.GA21222@dcvr.yhbt.net> John Joseph Bachir wrote: > My application accepts uploads from users, which can be quite huge in > some cases. This of course requires setting the unicorn timeout to > something much higher than 60 seconds, more like 10 minutes. > > Are there any drawbacks to doing this, other than the obvious drawback > of not killing off long-running requests that are illegitimate? Are your users remote? (outside of your immediate LAN). The upload speed (and thus timeout) for unicorn should be based on the nginx <-> unicorn transfer rate. Unicorn should never talk to users directly and users can upload slowly to nginx which buffers requests to the filesystem, first. > I've googled quite a bit about this and have found surprisingly little > -- I guess people who have apps that receive uploads just generally > don't use unicorn? I have a LAN-only application that regularly processes uploads several hundreds of megabytes (via PUT) directly to Unicorn. The local disk I/O is often the limiting factor since the parallel requests fill up the kernel buffers and wait on disk I/O. The Rack application itself unfortunately needs to seek/rewind so it must be in the filesystem. Rainbows! with ThreadSpawn/ThreadPool can process uploads without buffering them to disk first (but the Rack multipart parser may not). I often stream several hundred megabytes of data directly to apps on Rainbows! via PUT requests (curl -T), too. From cliftonk at gmail.com Thu Sep 29 14:48:44 2011 From: cliftonk at gmail.com (cliftonk at gmail.com) Date: Thu, 29 Sep 2011 13:48:44 -0500 Subject: large uploads In-Reply-To: References: Message-ID: <541300FF-748D-4E8F-9365-841FF4845173@gmail.com> You need to be buffering the upload before sending it to Unicorn. Are you using nginx? We're stuck on Apache, so I just did all my uploading / image-manipulation with nodejs instead. Clifton From j at jjb.cc Thu Sep 29 16:05:09 2011 From: j at jjb.cc (John Joseph Bachir) Date: Thu, 29 Sep 2011 16:05:09 -0400 Subject: large uploads In-Reply-To: <20110929184711.GA21222@dcvr.yhbt.net> References: <20110929184711.GA21222@dcvr.yhbt.net> Message-ID: On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong wrote: > The upload speed (and thus timeout) for unicorn should be based on > the nginx <-> unicorn transfer rate. ?Unicorn should never talk to users > directly and users can upload slowly to nginx which buffers requests to the > filesystem, first. On Thu, Sep 29, 2011 at 2:48 PM, wrote: > You need to be buffering the upload before sending it to Unicorn. Are > you using nginx? Thanks for the responses! So, I've see this module: https://github.com/vkholodkov/nginx-upload-module Is that what you are referring to? Are there other solutions? Does NGINX have inherent behavior which buffers requests? (Eric's email seemed to suggest that). Thanks, john From normalperson at yhbt.net Thu Sep 29 16:54:53 2011 From: normalperson at yhbt.net (Eric Wong) Date: Thu, 29 Sep 2011 13:54:53 -0700 Subject: large uploads In-Reply-To: References: <20110929184711.GA21222@dcvr.yhbt.net> Message-ID: <20110929205453.GA15148@dcvr.yhbt.net> John Joseph Bachir wrote: > On Thu, Sep 29, 2011 at 2:47 PM, Eric Wong wrote: > > The upload speed (and thus timeout) for unicorn should be based on > > the nginx <-> unicorn transfer rate. ?Unicorn should never talk to users > > directly and users can upload slowly to nginx which buffers requests to the > > filesystem, first. > > On Thu, Sep 29, 2011 at 2:48 PM, wrote: > > You need to be buffering the upload before sending it to Unicorn. Are > > you using nginx? > > Thanks for the responses! So, I've see this module: > https://github.com/vkholodkov/nginx-upload-module I think that's unrelated to what you're doing (it appears to be for nginx-only setups). > Is that what you are referring to? Are there other solutions? Does > NGINX have inherent behavior which buffers requests? (Eric's email > seemed to suggest that). Yes, nginx buffers entire requests by default before sending them to unicorn. From justin.giancola at gmail.com Thu Sep 29 16:58:19 2011 From: justin.giancola at gmail.com (Justin Giancola) Date: Thu, 29 Sep 2011 16:58:19 -0400 Subject: large uploads In-Reply-To: References: <20110929184711.GA21222@dcvr.yhbt.net> Message-ID: On Thu, Sep 29, 2011 at 4:05 PM, John Joseph Bachir wrote: > Does NGINX have inherent behavior which buffers requests? (Eric's email > seemed to suggest that). Nginx always buffers the entire body of requests before calling the upstream application. If the body exceeds client_body_buffer_size Nginx will store it in a tempfile. > Thanks for the responses! So, I've see this module: > https://github.com/vkholodkov/nginx-upload-module The upload module works a little differently. It allows you to configure specific upload endpoints that Nginx will preprocess. Rather than forwarding the entire request body to the upstream application, Nginx will parse the multipart/form-data encoded files, store them in tempfiles, and forward only the paths of the tempfiles to the upstream application. Justin From johnjosephbachir at gmail.com Fri Sep 30 13:04:26 2011 From: johnjosephbachir at gmail.com (John Joseph Bachir) Date: Fri, 30 Sep 2011 13:04:26 -0400 Subject: Multiple rack applications on the same server with unicorn Message-ID: If I'm running two rails apps on the same server using Unicorn, I have to run two instances of Unicorn, right? If so, then here's a place where passenger might win in terms of memory use, as the rails code will be loaded into memory twice, right? I'm still probably going with Unicorn, but just exploring this first. Thanks, John From normalperson at yhbt.net Fri Sep 30 13:16:08 2011 From: normalperson at yhbt.net (Eric Wong) Date: Fri, 30 Sep 2011 17:16:08 +0000 Subject: Multiple rack applications on the same server with unicorn In-Reply-To: References: Message-ID: <20110930171608.GA10578@dcvr.yhbt.net> John Joseph Bachir wrote: > If I'm running two rails apps on the same server using Unicorn, I have > to run two instances of Unicorn, right? I'm not sure if Rails (or any other frameworks) has issues, but plain Rack applications will work. Rack lets you do something like this in config.ru: --------------------- config.ru ------------------ map("http://foo.example.com/") do run FooApp.new end map("http://bar.example.com/") do run BarApp.new end ----------------------- 8< ----------------------- Both applications need to be able to work from the same working directory, though. > If so, then here's a place where passenger might win in terms of > memory use, as the rails code will be loaded into memory twice, right? It's all dependent on whether Rails lets you run two apps in the same process or not (in which case Unicorn can do it). The following example is pretty ugly, but lets you split your workers between two applications: ----------------- unicorn.conf.rb ----------------- preload_app false worker_processes 2 # need more than one after_fork do |server,worker| $unicorn_worker_nr = worker.nr end --------------------------------------------------- --------------------- config.ru ------------------ if ($unicorn_worker_nr % 2) == 0 run FooApp.new else run BarApp.new end ----------------------- 8< ----------------------- -- Eric Wong From hongli at phusion.nl Fri Sep 30 15:50:33 2011 From: hongli at phusion.nl (Hongli Lai) Date: Fri, 30 Sep 2011 21:50:33 +0200 Subject: Multiple rack applications on the same server with unicorn In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 7:04 PM, John Joseph Bachir wrote: > If I'm running two rails apps on the same server using Unicorn, I have > to run two instances of Unicorn, right? > > If so, then here's a place where passenger might win in terms of > memory use, as the rails code will be loaded into memory twice, right? If you have two apps then Phusion Passenger also loads Rails into memory twice. Unless if you're using the 'smart' spawning method (instead of the default 'smart-lv2' spawning method). The Phusion Passenger manual explains the difference between these two spawning methods. However 'smart' is practically useless these days because everybody uses Bundler. It is pretty much impossible nowadays to preload the Rails framework and sharing it between multiple apps transparently. We're planning on removing the 'smart' spawning method and only supporting 'smart-lv2' in the future. In the mean time, if you want to share the same Rails memory between apps, then you should merge them together into a single app. -- Phusion | Ruby & Rails deployment, scaling and tuning solutions Web: http://www.phusion.nl/ E-mail: info at phusion.nl Chamber of commerce no: 08173483 (The Netherlands) From j at jjb.cc Fri Sep 30 23:34:27 2011 From: j at jjb.cc (John Joseph Bachir) Date: Fri, 30 Sep 2011 23:34:27 -0400 Subject: Multiple rack applications on the same server with unicorn In-Reply-To: References:

Message-ID: I didn't realize that about bundler and smart spawning. Very helpful, thanks! From j at jjb.cc Fri Sep 30 23:39:17 2011 From: j at jjb.cc (John Joseph Bachir) Date: Fri, 30 Sep 2011 23:39:17 -0400 Subject: large uploads In-Reply-To: <20110929205453.GA15148@dcvr.yhbt.net> References: <20110929184711.GA21222@dcvr.yhbt.net> <20110929205453.GA15148@dcvr.yhbt.net> Message-ID: So buffering the entire request and then sending it to the backend is the default behavior of HttpUpstreamModule? That's fascinating. It almost seems like that's a bad design choice, because it inhibits the possibility of parallel work being done as the request comes in (like, it's the opposite of how unix pipes work).