[Mongrel] production-izing a popular site

Ezra Zygmuntowicz ezmobius at gmail.com
Thu Dec 28 22:24:14 EST 2006

On Dec 28, 2006, at 5:45 PM, Rob Sanheim wrote:

> Lets say you have a site that is serving, oh, around 100k unique
> visitors a day (plain ole' browser requests) - plus probably ~ 40k
> uniques to feeds.  Assume this site is 90% read like most of the web,
> so the traffic looks like your typical news or portal site.  There are
> two web boxes behind a hardware load balancer, each doing apache 2.2.3
> -> mongrel_proxy_balancer -> mongrel clusters.  One db machine, but
> that shouldn't be stressed at all due to heavy caching of the content.
> Where to being with performance testing something like this?  How many
> mongrels per web box to start with?  Any comparable sites in
> production to compare it to (Ezra?)
> thanks,
> - Rob

Hey Rob-

	I am hosting a site that does about 110k unique visitors per day and  
about 250k page views/day. I would say it is a typical rails app with  
about 85% read and then some write and user contributed uploads and  
such. This app is pretty well optimized and doesn't have any requests  
except for file uploads that take longer then about 1.5 seconds to  
complete with most requests going out at about 35-45req/sec. Pretty  
typical rails app.

	It is running on two of our slices which are Xen VM's where there is  
a hardware load balancer up front, then nginx proxying to mongrel  
cluster on each node. It runs 3 mongrels on each VM for a total of 6.  
The database is on another super beefy machine but is not dedicated  
entirely to this app and is shared by others. I think this will be a  
good place for you to start load testing, 3 - 5 mongrels on each of  
your app servers. First get a baseline for your hardware by using ab  
or httperf on a fresh rails app that just does render :text =>  
'hello' to get the best case scenario for a rails request on your  
hardware. This is the fastest you can expect rails to serve a   
dynamic request on your boxes. It will probably be around 100-140req/ 
sec with sessions turned on. Then do a few more of this same  
benchmark against one mongrel and a few of your rails actions that  
represent short and long requests for your app. Write these results  
down or save them as you go. This is important because I discovered  
that 3 mongrels performs *better* then 5 mongrels on our  
architecture. This will vary widely though over different os and  
hardware configurations.

	Now its time to add more mongrels until you hit a point of  
diminishing returns or you are confident in your ability to handle  
the traffic you expect plus some. ounds like you have two nice boxes  
for your app servers so start with 3 mongrels on one of the servers.  
Get this benchmark for the render :text action as well as the other  
actions you benched in your real app. At this point you should get  
more req/sec when you measure with concurrent users at 20-50. Then  
add one mongrel at a time and measure until adding one more mongrel  
does not buy you any more performance. This is your sweet spot where  
you are maxing out your proxy_balancers efficiency.

	Now that you hit the sweet spot for one app server setup the second  
one the same way and do some more benches at high concurrency. If you  
have good hardware load balancers you should get a good improvement  
with the two app servers over just one of them. In reality software  
load balancers don't multiply the perf of a single mongrel by the  
number of mongrels. Its more like adding another mongrel may get you  
20-50% more until you hit the sweet spot and can't get anymore. But  
the good hardware load balancers are more efficient and get closer to  
50-80% improvement when adding more nodes. Sometimes it's nice to go  
straight from the hardware load balancers right to mongrel and use an  
asset_host for static content. I don't think you will need this for  
the app you are talking about.

	I suspect you will end up with 4-6 mongrels per app server and this  
will leave you some overhead for what you need to serve. Another  
point of reference is the http://yakimaherald.com gets about 75-90k  
page views/day on 3 mongrels behind apache22 on an xserve.


-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Mongrel-users mailing list