[Mongrel] Sharing Mongrel Log Files on SAN Storage

Steven Hansen runner at berkeley.edu
Thu Nov 30 19:01:12 EST 2006


Ezra Zygmuntowicz wrote:
> On Nov 30, 2006, at 10:36 AM, Steven Hansen wrote:
>
>   
>> Hi,
>>
>> My department has a SAN and I'm wondering if it is safe to have all of
>> my mongrel's share log files stored on the SAN.
>>
>> More specifically, we have 4 machines, each running a couple of  
>> mongrel
>> processes.  Right now each machine has its own set of log files for
>> mongrel.  However, I am thinking that it might be better to put the  
>> log
>> files on the SAN and let all the mongrels on all the different  
>> machines
>> share 1 set of log files.  I'd be interested to gets some feedback  
>> from
>> other mongrelians on whether this is a good idea or a bad idea and  
>> why.
>>
>> Thanks,
>> Steven
>>
>>     
>
> Hey Steven-
>
> 	What filesystem re you using n the SAN? We have many Xen nodes  
> running rails apps that share a GFS filesystem partition off the SAN.  
> With GFS you can make hostname specific symlinks so that even though  
> the nodes of the cluster are all still writing their logs to logs/ 
> production.log, each node is actually writing to a symlink based on  
> hostname.
>
> 	Let me illustrate. Say we have 2 Xen instances for one rails app.  
> They both share a GFS partition mounted off the SAN at /data. So  
> anything under /data is shared between these nodes. With a capistrano  
> setup you make sure to deploy yoru apps to the /data partition. So to  
> setup the hostname logging on both slices you woudl log into each one  
> and run the following commands:
>
> app is here:
> /data/railsapp/current
>
> logs are here:
> /data/railsapp/shared/log
>
> 	We want both nodes to have their own log files but still have them  
> both be writing to the same log file path. So we make hostname  
> symlinks like this:
>
> $ cd /data/railsapp/shared
> $ rm -rf log
> $ mkdir `hostname`
> $ ln -s @hostname log
>
> 	You do that on all nodes that share the same /data partition. This  
> gives a very dramatic increase in performance over all nodes writing  
> to the same log file. Getting a file lock on shared storage is more  
> troublesome then on non shared storage. So with multiple nodes all  
> writing to the same log file, each node has to get a lock before it  
> can write and other nodes wait their turn for the lock. Using this  
> @hostname trick, each node has its own log file that no other nodes  
> write to. But you don't have to change the way rails or capistrano  
> sets up the log files. Each node writes its logs to /data/railsapp/ 
> shared/log/production.log, but in reality they are all writing to a  
> symlink to a directory with the same name as the nodes hostname.
>
> 	We tried NFS and a few other filesystems and they all sucked. We  
> ended up using the red hat clustering suite. But even though its a  
> red hat suite it runs on any linux, we use it with gentoo. It  
> includes GFS and the cluster fencing stuff that allows nodes to  
> dynamically fence off any other nodes that are misbehaving. GFS works  
> really well for shared storage off a SAN.
>
> 	I would definitely recommend not having all your nodes write their  
> logs to the same file on the SAN. See if you can find a way to do  
> like GFS does or as a last resort you may have to do some hacking to  
> get rails to write its log files with the pid or some other unique  
> identifier so that each node writes to its own log files. Doing this  
> gave us a whopping 25% better performance when compared with all  
> nodes sharing the same log file.
>
> 	
> Cheers-
>
> -- Ezra Zygmuntowicz 
> -- Lead Rails Evangelist
> -- ez at engineyard.com
> -- Engine Yard, Serious Rails Hosting
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Mongrel-users mailing list
> Mongrel-users at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-users
>   

Ezra and Zed,

Thanks for responding.  I feel a little foolish for even asking the 
question now, better safe than sorry though :-P

I guess originally I was thinking that if I wanted to check logs for 
traffic or something, then having it all in one log would be easier.  
But as Zed pointed out, this would make tracking down problems with an 
offending mongrel instance, much more difficult.  Not to mention that 
the issue with more mongrel threads fighting for the lock on the log files.

Anyway, I'm not sure what type of file system we are using as I really 
don't have much contact with the guys who are in charge of our 
machines.  Their setup seems similar to yours, Ezra.  When I ssh to a 
machine, I end up on 1 of (n) machines.  My entire account's file 
system--except a few directories--is stored on the SAN (at least this is 
how I understand it).  The non SAN directories are symbolic links to the 
specific machine that I'm logged on to.  So in my situation the 
directory /apps/local/#{account_name} will always point to the directory 
of a specific machine.  I think I'll be ok and have no problem giving 
each mongrel access to logs on its specific machine.  In case, you're 
interested, below is an overly detailed diagram of the setup.

Thanks so much!

-Steven


'%' means filesystem is local to current machine, all other directories 
are on the SAN

|--apps %
|
|--users
|  +
|  |--#{account_name}
|  |  +
|  |  |--bin
|  |  |--rails
|  |  |  +
|  |  |  |--#{app_name}
|  |  |  |  +
|  |  |  |  |--current -> 
/users/#{account_name}/rails/#{app_name}/releases/rel_x
|  |  |  |  |  +
|  |  |  |  |  |--app
|  |  |  |  |  |--config
|  |  |  |  |  |--db
|  |  |  |  |  |--doc
|  |  |  |  |  |--lib
|  |  |  |  |  |--log -> /apps/local/#{account_name}/mongrel/log
|  |  |  |  |  |--public
|  |  |  |  |  |--script
|  |  |  |  |  |--test
|  |  |  |  |  |--tmp
|  |  |  |  |  |  +
|  |  |  |  |  |  |--pids -> /apps/local/#{account_name}/mongrel/pids  %
|  |  |  |  |  |
|  |  |  |  |  |--vendor
|  |  |  |  |
|  |  |  |  |--releases
|  |  |  |  |  +
|  |  |  |  |  |--rel_1
|  |  |  |  |  |--rel_2
|  |  |  |  |  |--rel_3
|  |  |  |  |
|  |  |
|  |  |--apache2
|  |  |  +
|  |  |  |--conf
|  |  |  |--lib
|  |  |  |--libexec -> /opt/csw/apache2/libexec  %
|  |  |  |--logs -> /apps/local/#{account_name}/apache2/logs  %
|  |  |  |--bin -> /opt/csw/apache2/bin  %
|  |  | 
|  |  |--include
|  |  |--var
|  |  |--local -> /apps/local/#{account_name}  %
|  |  |  +
|  |  |  |--apache2  %
|  |  |  |  +
|  |  |  |  |--logs  %
|  |  |  |  |--tmp  %
|  |  |  |
|  |  |  |--mongrel  %
|  |  |  |  +
|  |  |  |  |--log  %
|  |  |  |  |--pids  %
|  |  |  |
















































More information about the Mongrel-users mailing list