A barrage of unexplained timeouts
normalperson at yhbt.net
Tue Aug 20 16:37:45 UTC 2013
nick at auger.net wrote:
> We've been running unicorn-3.6.2 on REE 1.8.7 2011.12 in production
> for quite some time and we use monit to monitor each unicorn worker.
> Occasionally, I'll get a notification that a worker has timed-out and
> has been re-spawned. In all these cases, when I look at the rails
> logs, I can see the last request that the worker handled, and they all
> have appeared to complete successfully from the client's perspective
> (rails and nginx respond with 200), but the unicorn log shows that it
> was killed due to timeout. This has always been relatively rare and I
> thought it was a non-problem.
> During this time, our servers (Web, PG DB, and redis) were not under
> load and IO was normal. After the last monit notification at 8:30,
> everything went back to normal. I understand why unicorns would
> timeout if they were waiting (>120 secs) on IO, but there aren't any
> orphaned requests in the rails log. For each request line, there's a
> corresponding completion line. No long running queries to blame on
> PG, either.
Can you take a look at the nginx error and access logs? From what
you're saying, there's a chance a request never even got to the Rails
layer. However, nginx should be logging failed/long-running requests to
More information about the mongrel-unicorn