A barrage of unexplained timeouts

Eric Wong normalperson at yhbt.net
Tue Aug 20 16:37:45 UTC 2013


nick at auger.net wrote:
> We've been running unicorn-3.6.2 on REE 1.8.7 2011.12 in production
> for quite some time and we use monit to monitor each unicorn worker.
> Occasionally, I'll get a notification that a worker has timed-out and
> has been re-spawned.  In all these cases, when I look at the rails
> logs, I can see the last request that the worker handled, and they all
> have appeared to complete successfully from the client's perspective
> (rails and nginx respond with 200), but the unicorn log shows that it
> was killed due to timeout.  This has always been relatively rare and I
> thought it was a non-problem.
> 
> During this time, our servers (Web, PG DB, and redis) were not under
> load and IO was normal.  After the last monit notification at 8:30,
> everything went back to normal.  I understand why unicorns would
> timeout if they were waiting (>120 secs) on IO, but there aren't any
> orphaned requests in the rails log.  For each request line, there's a
> corresponding completion line.  No long running queries to blame on
> PG, either.

Can you take a look at the nginx error and access logs?  From what
you're saying, there's a chance a request never even got to the Rails
layer.  However, nginx should be logging failed/long-running requests to
unicorn.


More information about the mongrel-unicorn mailing list