[Mongrel] Sharing Mongrel Log Files on SAN Storage

Ezra Zygmuntowicz ezmobius at gmail.com
Thu Nov 30 15:54:41 EST 2006

On Nov 30, 2006, at 10:36 AM, Steven Hansen wrote:

> Hi,
> My department has a SAN and I'm wondering if it is safe to have all of
> my mongrel's share log files stored on the SAN.
> More specifically, we have 4 machines, each running a couple of  
> mongrel
> processes.  Right now each machine has its own set of log files for
> mongrel.  However, I am thinking that it might be better to put the  
> log
> files on the SAN and let all the mongrels on all the different  
> machines
> share 1 set of log files.  I'd be interested to gets some feedback  
> from
> other mongrelians on whether this is a good idea or a bad idea and  
> why.
> Thanks,
> Steven

Hey Steven-

	What filesystem re you using n the SAN? We have many Xen nodes  
running rails apps that share a GFS filesystem partition off the SAN.  
With GFS you can make hostname specific symlinks so that even though  
the nodes of the cluster are all still writing their logs to logs/ 
production.log, each node is actually writing to a symlink based on  

	Let me illustrate. Say we have 2 Xen instances for one rails app.  
They both share a GFS partition mounted off the SAN at /data. So  
anything under /data is shared between these nodes. With a capistrano  
setup you make sure to deploy yoru apps to the /data partition. So to  
setup the hostname logging on both slices you woudl log into each one  
and run the following commands:

app is here:

logs are here:

	We want both nodes to have their own log files but still have them  
both be writing to the same log file path. So we make hostname  
symlinks like this:

$ cd /data/railsapp/shared
$ rm -rf log
$ mkdir `hostname`
$ ln -s @hostname log

	You do that on all nodes that share the same /data partition. This  
gives a very dramatic increase in performance over all nodes writing  
to the same log file. Getting a file lock on shared storage is more  
troublesome then on non shared storage. So with multiple nodes all  
writing to the same log file, each node has to get a lock before it  
can write and other nodes wait their turn for the lock. Using this  
@hostname trick, each node has its own log file that no other nodes  
write to. But you don't have to change the way rails or capistrano  
sets up the log files. Each node writes its logs to /data/railsapp/ 
shared/log/production.log, but in reality they are all writing to a  
symlink to a directory with the same name as the nodes hostname.

	We tried NFS and a few other filesystems and they all sucked. We  
ended up using the red hat clustering suite. But even though its a  
red hat suite it runs on any linux, we use it with gentoo. It  
includes GFS and the cluster fencing stuff that allows nodes to  
dynamically fence off any other nodes that are misbehaving. GFS works  
really well for shared storage off a SAN.

	I would definitely recommend not having all your nodes write their  
logs to the same file on the SAN. See if you can find a way to do  
like GFS does or as a last resort you may have to do some hacking to  
get rails to write its log files with the pid or some other unique  
identifier so that each node writes to its own log files. Doing this  
gave us a whopping 25% better performance when compared with all  
nodes sharing the same log file.


-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

More information about the Mongrel-users mailing list