[Mongrel] Recommentation: Sessions and PStore
zedshaw at zedshaw.com
Sun Sep 3 12:18:06 EDT 2006
As most of you know there were a few people who had the following three
* CLOSE_WAIT: Lots of sockets in CLOSE_WAIT state.
* 99% CPU: Mongrel's getting "stuck" pegged at 99% CPU.
* LEAK: Memory leak.
I've successfully fixed these bugs or attributed them to one main cause:
First, the memory leak was because of a bug in how the GC in Ruby was
collecting threads after they used a Mutex. Don't ask me why, but
switching to Sync fixed it. Problem is, this also causes the leak on
Win32. Still working on that problem.
Now for the recommendations:
1) DON'T USE PSTORE AS A SESSION STORE!
All of the people who had Mongrel processes going crazy, blocking,
locking, or just generally doing bad things had a pstore as their
session storage option. What happens is the pstore isn't that great
when multiple processes access it, and if one Mongrel has problems
pulling an object from the pstore then it'll explode. Use the file
store when you develop, and use the database or memcached store in
You should also be looking for other ways processes are sharing
resources when they lock up or do weird things. Try running with just
one process, and if you don't have any problems then it's a resource
locking problem (most likely).
2) DON'T STORE COMPLEX OBJECTS IN THE SESSION!
This especially goes for you Java people who think the session store is
a temporary database. The session really only works for simple types
like integers, strings, etc. You *must* follow the mantra of "Share
Nothing" and either store everything in the database or use form fields
and GET parameters to make it stateless. As I've said repeatedly, "If
rails-core don't do it, you don't do it." Rails-core does not do this.
You don't do it.
3) USE THE USR1 LOGGING!
Every person who had a problem with Mongrel processes getting stuck
could have used the new USR1 logging to quickly find out what Rails
action was blocking the process. Once I got a good USR1 log it was
trivial to see a message stating that 981 threads were waiting for one
Rails action. Once I pointed this out, they went back and figured out
what the problem was.
Use "killall -USR1 mongrel_rails" to turn this on temporarily, and
Mongrel will log the numbers of threads waiting for each request and
which are remaining to be processed by Mongrel. It's very handy.
Hitting the mongrels with USR1 again turns it off. It's also pretty
fast so you could leave it on all the time and not see a big performance
4) YOUR RAILS CODE SUCKS!
I hate to say it folks, and I'll gladly help people when they've got a
problem, but please check your Rails code first, and assume it's broken.
Lots and lots and lots and lots of sites are running high traffic
Mongrel installs without any problems. If you have a problem, then
you've got something different. I know, you're absolutely perfect and
there's nothing you're doing strange, but really look. In all the cases
mentioned above they were also doing something weird. Things to look
* Using extensions to libraries you didn't write. If you have a library
that you bought and then wrapped with Ruby's extensions, then chances
are it's *not* designed for multi-threaded Ruby action. Move this
library to a sequentially run DRb process shared by all your Mongrel
* Trying to resolve hosts against broken DNS. Many people have slow
rails actions because they're doing tons of DNS resolving of hosts.
Make sure your internal DNS is fast, reliable and consider using dnsmasq
running locally to cache the results. Main problem is Ruby's resolving
blocks the process.
* Use pstore as the session store, so probably using pstore for nearly
anything is bad.
* Using your own custom network protocols. If you just wrote it then
it's broken. #1, #2, and #3 rule of programming. It's not Mongrel.
* Using the mysql.rb instead of installing the actual mysql compiled
gem. The mysql.rb is just for development, and it sucks horribly.
Thanks for your time folks, and keep coming with those bugs.
Zed A. Shaw
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.
More information about the Mongrel-users