From scot.dalton at nyu.edu  Tue Apr  5 16:56:41 2011
From: scot.dalton at nyu.edu (Scot Dalton)
Date: Tue, 5 Apr 2011 16:56:41 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
Message-ID: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>

Hi all,

We're experiencing an excessive number of TCP connections in a CLOSE_WAIT state which are bogging down our Umlaut server.  Has anyone else experienced this problem or does anybody have any advice on how to troubleshoot the issue.  A quick Google search indicates this may be a common problem in mongrel_rails, but we're not sure how to resolve it.

Thanks,
Scot



--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From rochkind at jhu.edu  Tue Apr  5 17:24:47 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Tue, 05 Apr 2011 17:24:47 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
In-Reply-To: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
References: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
Message-ID: <4D9B889F.7010700@jhu.edu>

Ugh, that sucks. I haven't run into it.

You might try deploying under passenger instead of mongrel_rails. i 
haven't tried it yet.

I REALLY want to find time to do a major overhaul of Umlaut, upgrade it 
to Rails3, deploying under passenger, using thread-friendly mysql2 
adapter.  Not sure when that's going to happen, sigh.

Hacky, but you could always cronjob shutting down your mongrels 
regularly. Shut down half of em, bring em back up, wait for em to be 
back up, restart and shut down the other half.  The threading in Umlaut 
makes a straight 'restart' of mongrels not work so well, since the 
mongrels won't shut down until all the threads are complete (or after 
some timeout when it's sick of waiting) (good), but this makes 'restart' 
get messed up trying to shut down and restart em (bad). So I 'stop', 
wait a minute, then 'start' instead.

On 4/5/2011 4:56 PM, Scot Dalton wrote:
> Hi all,
>
> We're experiencing an excessive number of TCP connections in a CLOSE_WAIT state which are bogging down our Umlaut server.  Has anyone else experienced this problem or does anybody have any advice on how to troubleshoot the issue.  A quick Google search indicates this may be a common problem in mongrel_rails, but we're not sure how to resolve it.
>
> Thanks,
> Scot
>
>
>
> --
> Scot Dalton
> Phone:  (212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general

From rossfsinger at gmail.com  Tue Apr  5 22:36:52 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Tue, 5 Apr 2011 22:36:52 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
In-Reply-To: <4D9B889F.7010700@jhu.edu>
References: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
	<4D9B889F.7010700@jhu.edu>
Message-ID: <BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>

On Tue, Apr 5, 2011 at 5:24 PM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
> You might try deploying under passenger instead of mongrel_rails. i haven't
> tried it yet.
>
Another option that would require less infrastructure change would be
to swap out mongrel for thin:
http://code.macournoyer.com/thin/

heroku uses thin, which should give you some confidence in it.

(If you're already itching to migrate to Passenger, nevermind, I'm
just putting this out there since you wouldn't have to change the way
your existing stack works)

It works basically the same way as mongrel, you deploy a cluster and
proxy them via Apache or Nginx or whatever:

http://code.macournoyer.com/thin/usage/

Anyway, you could deploy a thin cluster of the same size, on the same
ports as you're currently deploying mongrel and no one would be the
wiser (except it would probably be faster and more stable).

> I REALLY want to find time to do a major overhaul of Umlaut, upgrade it to
> Rails3, deploying under passenger, using thread-friendly mysql2 adapter.
> ?Not sure when that's going to happen, sigh.

In the meantime, I think you'd get the quickest wins fairly painlessly
via JRuby with Trinidad (https://github.com/trinidad/trinidad) or
Torquebox (http://torquebox.org/).  JRuby 1.6 is so much faster than
MRI 1.8.7 and it would be ideal for Umlaut's threads.

I know it doesn't address what you actually want to do, Jonathan, but
it also doesn't preclude it (Rails 3 runs in JRuby 1.5.x+ and it also
has 1.9 support).  I just think it would give you some significant
performance boosts without the massive refactoring that Rails 3 or 1.9
would require (while also not standing in the way of any refactoring).

But, Scot, I'm still recommending Thin for you right now :)

-Ross.

From std5 at nyu.edu  Wed Apr  6 09:04:42 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 6 Apr 2011 09:04:42 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
In-Reply-To: <BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>
References: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
	<4D9B889F.7010700@jhu.edu>
	<BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>
Message-ID: <6B9F4C64-257D-4574-8E18-1554637665DB@nyu.edu>

Thanks for the suggestions.  I think we'll try thin for now and see how it works.  Once we have it up and running, I'll report back to the list with our experience.

Thanks,
Scot

On Apr 5, 2011, at Apr 5, 10:36 PM, Ross Singer wrote:

> On Tue, Apr 5, 2011 at 5:24 PM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
>> You might try deploying under passenger instead of mongrel_rails. i haven't
>> tried it yet.
>> 
> Another option that would require less infrastructure change would be
> to swap out mongrel for thin:
> http://code.macournoyer.com/thin/
> 
> heroku uses thin, which should give you some confidence in it.
> 
> (If you're already itching to migrate to Passenger, nevermind, I'm
> just putting this out there since you wouldn't have to change the way
> your existing stack works)
> 
> It works basically the same way as mongrel, you deploy a cluster and
> proxy them via Apache or Nginx or whatever:
> 
> http://code.macournoyer.com/thin/usage/
> 
> Anyway, you could deploy a thin cluster of the same size, on the same
> ports as you're currently deploying mongrel and no one would be the
> wiser (except it would probably be faster and more stable).
> 
>> I REALLY want to find time to do a major overhaul of Umlaut, upgrade it to
>> Rails3, deploying under passenger, using thread-friendly mysql2 adapter.
>>  Not sure when that's going to happen, sigh.
> 
> In the meantime, I think you'd get the quickest wins fairly painlessly
> via JRuby with Trinidad (https://github.com/trinidad/trinidad) or
> Torquebox (http://torquebox.org/).  JRuby 1.6 is so much faster than
> MRI 1.8.7 and it would be ideal for Umlaut's threads.
> 
> I know it doesn't address what you actually want to do, Jonathan, but
> it also doesn't preclude it (Rails 3 runs in JRuby 1.5.x+ and it also
> has 1.9 support).  I just think it would give you some significant
> performance boosts without the massive refactoring that Rails 3 or 1.9
> would require (while also not standing in the way of any refactoring).
> 
> But, Scot, I'm still recommending Thin for you right now :)
> 
> -Ross.
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general


From rochkind at jhu.edu  Wed Apr  6 11:12:09 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Wed, 06 Apr 2011 11:12:09 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
In-Reply-To: <BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>
References: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
	<4D9B889F.7010700@jhu.edu>
	<BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>
Message-ID: <4D9C82C9.2000102@jhu.edu>

On 4/5/2011 10:36 PM, Ross Singer wrote:
>
> In the meantime, I think you'd get the quickest wins fairly painlessly
> via JRuby with Trinidad (https://github.com/trinidad/trinidad) or

Um, that's only the 'quickest' win if Umlaut works out of the box under 
JRuby.  I am not confident enough in that to be confident this is the 
'quickest', especially since evaluating whether Umlaut works properly 
under JRuby is a chore of it's own (Umlaut sadly lacks good unit tests).


From std5 at nyu.edu  Thu Apr  7 10:44:36 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Thu, 7 Apr 2011 10:44:36 -0400
Subject: [Umlaut-general] Excessive CLOSE_WAITS
In-Reply-To: <6B9F4C64-257D-4574-8E18-1554637665DB@nyu.edu>
References: <AF307FBB-B288-4496-A657-E6811E1D9470@nyu.edu>
	<4D9B889F.7010700@jhu.edu>
	<BANLkTinMghoxw5QStyUEjAR2HfU6k4MCtg@mail.gmail.com>
	<6B9F4C64-257D-4574-8E18-1554637665DB@nyu.edu>
Message-ID: <D2E731FE-BA3E-495B-B87E-37E08EFFD662@nyu.edu>

Hi all, 

We implemented thin and as Ross suggested, it was straightforward and painless.  Seems to be working well so far.  I'll update the wiki with our thin configuration.

We still see a lot of TCP connections in the CLOSE_WAIT state, but they seem to be closing properly after a few seconds.  
Are other institutions seeing a large number of CLOSE_WAITS during high traffic periods?  These connections are slowing down the server and causing some performance issues.  We're looking into tuning the OS (Solaris) params for TCP, and are open to any other suggestions.

Thanks,
Scot

On Apr 6, 2011, at Apr 6, 9:04 AM, Scot Dalton wrote:

> Thanks for the suggestions.  I think we'll try thin for now and see how it works.  Once we have it up and running, I'll report back to the list with our experience.
> 
> Thanks,
> Scot
> 
> On Apr 5, 2011, at Apr 5, 10:36 PM, Ross Singer wrote:
> 
>> On Tue, Apr 5, 2011 at 5:24 PM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
>>> You might try deploying under passenger instead of mongrel_rails. i haven't
>>> tried it yet.
>>> 
>> Another option that would require less infrastructure change would be
>> to swap out mongrel for thin:
>> http://code.macournoyer.com/thin/
>> 
>> heroku uses thin, which should give you some confidence in it.
>> 
>> (If you're already itching to migrate to Passenger, nevermind, I'm
>> just putting this out there since you wouldn't have to change the way
>> your existing stack works)
>> 
>> It works basically the same way as mongrel, you deploy a cluster and
>> proxy them via Apache or Nginx or whatever:
>> 
>> http://code.macournoyer.com/thin/usage/
>> 
>> Anyway, you could deploy a thin cluster of the same size, on the same
>> ports as you're currently deploying mongrel and no one would be the
>> wiser (except it would probably be faster and more stable).
>> 
>>> I REALLY want to find time to do a major overhaul of Umlaut, upgrade it to
>>> Rails3, deploying under passenger, using thread-friendly mysql2 adapter.
>>> Not sure when that's going to happen, sigh.
>> 
>> In the meantime, I think you'd get the quickest wins fairly painlessly
>> via JRuby with Trinidad (https://github.com/trinidad/trinidad) or
>> Torquebox (http://torquebox.org/).  JRuby 1.6 is so much faster than
>> MRI 1.8.7 and it would be ideal for Umlaut's threads.
>> 
>> I know it doesn't address what you actually want to do, Jonathan, but
>> it also doesn't preclude it (Rails 3 runs in JRuby 1.5.x+ and it also
>> has 1.9 support).  I just think it would give you some significant
>> performance boosts without the massive refactoring that Rails 3 or 1.9
>> would require (while also not standing in the way of any refactoring).
>> 
>> But, Scot, I'm still recommending Thin for you right now :)
>> 
>> -Ross.
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
> 


From scot.dalton at nyu.edu  Mon Apr 11 10:11:29 2011
From: scot.dalton at nyu.edu (Scot Dalton)
Date: Mon, 11 Apr 2011 10:11:29 -0400
Subject: [Umlaut-general] Scopus Service
Message-ID: <2007D658-3A20-4F49-A973-B43F56A5D61B@nyu.edu>

Hi all,

We're interested in turning on the Scopus service in Umlaut.  Are there any issues we should be aware of (other than the need for an API key)?

Thanks,
Scot

--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From rochkind at jhu.edu  Mon Apr 11 10:44:00 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 11 Apr 2011 10:44:00 -0400
Subject: [Umlaut-general] Scopus Service
In-Reply-To: <2007D658-3A20-4F49-A973-B43F56A5D61B@nyu.edu>
References: <2007D658-3A20-4F49-A973-B43F56A5D61B@nyu.edu>
Message-ID: <4DA313B0.1020902@jhu.edu>

Not that I know of, it works pretty well for me, and I'm encouraged to 
have other people giving it a try, I think it's actually awfully useful.

One issue not with the service in general but with the way I think 
you've set up your umlaut --  the way you put your bX results right on 
the page in the "Related" list --- the Scopus service adds a link to the 
Scopus results in the "related" list.  So that's going to display a bit 
weirdly, the link out to Scopus will be just antoher item in your list 
of actual related articles.   You and me had different ideas of how to 
use this "related" list.  So you might need to make some choices about 
how you want all this to appear in the UI, and then figure out how to 
make Umlaut do that, happy to help with that.

If you have trouble figuring out how to get an API key from Scopus, I 
can look up my Scopus contacts and share them with you.

Jonathan

On 4/11/2011 10:11 AM, Scot Dalton wrote:
> Hi all,
>
> We're interested in turning on the Scopus service in Umlaut.  Are there any issues we should be aware of (other than the need for an API key)?
>
> Thanks,
> Scot
>
> --
> Scot Dalton
> Phone:  (212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general

From scot.dalton at nyu.edu  Mon Apr 11 10:48:39 2011
From: scot.dalton at nyu.edu (Scot Dalton)
Date: Mon, 11 Apr 2011 10:48:39 -0400
Subject: [Umlaut-general] Scopus Service
In-Reply-To: <4DA313B0.1020902@jhu.edu>
References: <2007D658-3A20-4F49-A973-B43F56A5D61B@nyu.edu>
	<4DA313B0.1020902@jhu.edu>
Message-ID: <693D335F-9967-48C1-B690-AA0B112953DA@nyu.edu>

I think we'll be on the same page with the UI for now, since our bX implementation was only a test and we haven't chosen to go forward in production.  I'll let you know if I encounter any issues with the API key.

Thanks,
Scot

On Apr 11, 2011, at Apr 11, 10:44 AM, Jonathan Rochkind wrote:

> Not that I know of, it works pretty well for me, and I'm encouraged to have other people giving it a try, I think it's actually awfully useful.
> 
> One issue not with the service in general but with the way I think you've set up your umlaut --  the way you put your bX results right on the page in the "Related" list --- the Scopus service adds a link to the Scopus results in the "related" list.  So that's going to display a bit weirdly, the link out to Scopus will be just antoher item in your list of actual related articles.   You and me had different ideas of how to use this "related" list.  So you might need to make some choices about how you want all this to appear in the UI, and then figure out how to make Umlaut do that, happy to help with that.
> 
> If you have trouble figuring out how to get an API key from Scopus, I can look up my Scopus contacts and share them with you.
> 
> Jonathan
> 
> On 4/11/2011 10:11 AM, Scot Dalton wrote:
>> Hi all,
>> 
>> We're interested in turning on the Scopus service in Umlaut.  Are there any issues we should be aware of (other than the need for an API key)?
>> 
>> Thanks,
>> Scot
>> 
>> --
>> Scot Dalton
>> Phone:  (212) 998-2674
>> Web Services
>> Division of Libraries
>> New York University
>> 
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general


--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From scot.Dalton at nyu.edu  Tue Apr 12 14:11:29 2011
From: scot.Dalton at nyu.edu (Scot Dalton)
Date: Tue, 12 Apr 2011 14:11:29 -0400
Subject: [Umlaut-general] Umlaut Performance
Message-ID: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>

Hi all,

Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.

Any help would be greatly appreciated.

Thanks,
Scot

--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From rochkind at jhu.edu  Tue Apr 12 14:32:12 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Tue, 12 Apr 2011 18:32:12 +0000
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
Message-ID: <665DBC51D0250A47B4F9306CE71E5FB70252EB@JHEMTEBEX5.win.ad.jhu.edu>

I suspect it may sadly be that the threading we're doing in Rails is kind of not so great. I hope to do some performance tuning and improvement with the hoped-for Rails3 migration, as well as the fact that ruby 1.8.7/Rails3 do better with threading (let alone jruby or ruby 1.9). 

The easiest thing to do is certainly just to add additional instances of your app. I forget if you are still running mongrel or not, but each mongrel instance only takes about 150Megs of RAM max, if you just add a few more for peak times that may take care of it. I run like 7 mongrel instances myself, just in case. 

However, the next thing I'd try , or maybe even try first -- if you are running ruby 1.8.7, try the 'mysql2' ActiveRecord adapter, instead of the mysql one that you are probably using now. 'mysql2' is the default in Rails3, but should work under Rails2 too, I think. (But it doesn't work under ruby 1.8.6, it needs 1.8.7). The original 'mysql' adapter we are using does some crazy things under threaded concurrency, that the mysql2 adapter fixes. 

But I suspect it _probably_ isn't something in your code specifically -- although it may be something in Umlaut code, which will be easier to fix in Rails3. 

Jonathan

-----Original Message-----
From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
Sent: Tuesday, April 12, 2011 2:11 PM
To: umlaut-general at rubyforge.org
Subject: [Umlaut-general] Umlaut Performance

Hi all,

Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.

Any help would be greatly appreciated.

Thanks,
Scot

--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University

_______________________________________________
Umlaut-general mailing list
Umlaut-general at rubyforge.org
http://rubyforge.org/mailman/listinfo/umlaut-general

From rossfsinger at gmail.com  Tue Apr 12 14:38:00 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Tue, 12 Apr 2011 14:38:00 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
Message-ID: <BANLkTinWLMGbi4k_NU4FkPLF5iTqu5QDXQ@mail.gmail.com>

How many thin servers do you run in your cluster?

Performance, unfortunately, is a bit of a "known known", not helped in
the slightest by the fact that the entire process is kicked off by
what is, in any reasonable estimation, an incredibly slow SFX API.

I guess a couple of good questions would be:

How many services run in an average request?  How many "waves" of threads run?

Do you know how many concurrent requests seem to be happening at these
peak times?

-Ross.

On Tue, Apr 12, 2011 at 2:11 PM, Scot Dalton <scot.Dalton at nyu.edu> wrote:
> Hi all,
>
> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scot
>
> --
> Scot Dalton
> Phone: ?(212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From dale.poulter at Vanderbilt.Edu  Tue Apr 12 14:14:59 2011
From: dale.poulter at Vanderbilt.Edu (Poulter,   Dale)
Date: Tue, 12 Apr 2011 13:14:59 -0500
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
Message-ID: <7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>

We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly. 

-----Original Message-----
From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
Sent: Tuesday, April 12, 2011 1:11 PM
To: umlaut-general at rubyforge.org
Subject: [Umlaut-general] Umlaut Performance

Hi all,

Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.

Any help would be greatly appreciated.

Thanks,
Scot

--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University

_______________________________________________
Umlaut-general mailing list
Umlaut-general at rubyforge.org
http://rubyforge.org/mailman/listinfo/umlaut-general

From std5 at nyu.edu  Wed Apr 13 09:46:27 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 09:46:27 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
Message-ID: <04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>

Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.

Thanks again,
Scot

On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:

> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly. 
> 
> -----Original Message-----
> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
> Sent: Tuesday, April 12, 2011 1:11 PM
> To: umlaut-general at rubyforge.org
> Subject: [Umlaut-general] Umlaut Performance
> 
> Hi all,
> 
> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
> 
> Any help would be greatly appreciated.
> 
> Thanks,
> Scot
> 
> --
> Scot Dalton
> Phone:  (212) 998-2674
> Web Services
> Division of Libraries
> New York University
> 
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general


From rossfsinger at gmail.com  Wed Apr 13 10:12:16 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Wed, 13 Apr 2011 10:12:16 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
Message-ID: <BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>

Scot,

How many were you running?

-Ross.

On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
> Thanks for the suggestions. ?I increased the number of processes running to 8 and will continue to monitor. ?We'll also be moving to a new server at the end of the month, which will hopefully help.
>
> Thanks again,
> Scot
>
> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>
>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>
>> -----Original Message-----
>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>> Sent: Tuesday, April 12, 2011 1:11 PM
>> To: umlaut-general at rubyforge.org
>> Subject: [Umlaut-general] Umlaut Performance
>>
>> Hi all,
>>
>> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>
>> Any help would be greatly appreciated.
>>
>> Thanks,
>> Scot
>>
>> --
>> Scot Dalton
>> Phone: ?(212) 998-2674
>> Web Services
>> Division of Libraries
>> New York University
>>
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From std5 at nyu.edu  Wed Apr 13 11:40:41 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 11:40:41 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
Message-ID: <F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>

We were previously running 4.

Thanks,
Scot

On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:

> Scot,
> 
> How many were you running?
> 
> -Ross.
> 
> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>> 
>> Thanks again,
>> Scot
>> 
>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>> 
>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>> 
>>> -----Original Message-----
>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>> To: umlaut-general at rubyforge.org
>>> Subject: [Umlaut-general] Umlaut Performance
>>> 
>>> Hi all,
>>> 
>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>> 
>>> Any help would be greatly appreciated.
>>> 
>>> Thanks,
>>> Scot
>>> 
>>> --
>>> Scot Dalton
>>> Phone:  (212) 998-2674
>>> Web Services
>>> Division of Libraries
>>> New York University
>>> 
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>> 
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>> 


From rossfsinger at gmail.com  Wed Apr 13 12:13:42 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Wed, 13 Apr 2011 12:13:42 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
Message-ID: <BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>

Yeah, I think expanding to 8 will make a huge difference.  Between the
initial request and the subsequent AJAX requests, I can imagine 4
getting used up by one or two resolution requests.

-Ross.

On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton <std5 at nyu.edu> wrote:
> We were previously running 4.
>
> Thanks,
> Scot
>
> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>
>> Scot,
>>
>> How many were you running?
>>
>> -Ross.
>>
>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>> Thanks for the suggestions. ?I increased the number of processes running to 8 and will continue to monitor. ?We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>
>>> Thanks again,
>>> Scot
>>>
>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>
>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>
>>>> -----Original Message-----
>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>> To: umlaut-general at rubyforge.org
>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>
>>>> Hi all,
>>>>
>>>> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>
>>>> Any help would be greatly appreciated.
>>>>
>>>> Thanks,
>>>> Scot
>>>>
>>>> --
>>>> Scot Dalton
>>>> Phone: ?(212) 998-2674
>>>> Web Services
>>>> Division of Libraries
>>>> New York University
>>>>
>>>> _______________________________________________
>>>> Umlaut-general mailing list
>>>> Umlaut-general at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>> _______________________________________________
>>>> Umlaut-general mailing list
>>>> Umlaut-general at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>
>
>

From rochkind at jhu.edu  Wed Apr 13 12:34:38 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Wed, 13 Apr 2011 12:34:38 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
Message-ID: <4DA5D09E.1020805@jhu.edu>

Part of why Umlaut may need more instances than a typical Rails app is 
because Umlaut's requests can take so long to complete -- mainly because 
of waiting on SFX.  Another possible reason is that Umlaut's background 
processing makes an instance somewhat busier than a typical Rails 
instance, which does no background threaded processing like this -- I am 
not entirely sure, but think that background processing probably means 
more instances than usual are indicated, especially on a multi-CPU box, 
to spread the load out.

So I haven't actually done the work to profile, but these are some 
logical reasons that you might need to give more instances to Umlaut 
than would be suggested by general purpose Rails guidelines you may find 
on the web.  Umlaut instances are both slower to process a 
request-response cycle, as well as busier with background threads, than 
a typical Rails app instance.  Which may indicate a requirement for more 
instances in the pool than typical for Rails apps.  (And if so, even if 
you were deploying under Passenger, would probably suggest instructing 
Passenger to spin up more instances than typical. Passenger too has 
multiple instances of your app, it just handles it all transparently, 
spinning them up and down as needed. However, you can give Passenger 
instructions instead of using defaults).

One thing to be aware of when adding mongrel instances is whether you 
have enough physical RAM to contain them all.  If you add more instances 
than you have physical RAM to contain, then the swapping in and out from 
disk to RAM, between the various instances MAY negate any advantage you 
get by adding the instance.

Jonathan

On 4/13/2011 12:13 PM, Ross Singer wrote:
> Yeah, I think expanding to 8 will make a huge difference.  Between the
> initial request and the subsequent AJAX requests, I can imagine 4
> getting used up by one or two resolution requests.
>
> -Ross.
>
> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>> We were previously running 4.
>>
>> Thanks,
>> Scot
>>
>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>
>>> Scot,
>>>
>>> How many were you running?
>>>
>>> -Ross.
>>>
>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>
>>>> Thanks again,
>>>> Scot
>>>>
>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>
>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>
>>>>> -----Original Message-----
>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>> To: umlaut-general at rubyforge.org
>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>
>>>>> Any help would be greatly appreciated.
>>>>>
>>>>> Thanks,
>>>>> Scot
>>>>>
>>>>> --
>>>>> Scot Dalton
>>>>> Phone:  (212) 998-2674
>>>>> Web Services
>>>>> Division of Libraries
>>>>> New York University
>>>>>
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>> _______________________________________________
>>>> Umlaut-general mailing list
>>>> Umlaut-general at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>
>>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general


From std5 at nyu.edu  Wed Apr 13 12:34:20 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 12:34:20 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
Message-ID: <0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>

Great.  Do you think an additional few processes (say going from 8 to 12) would help?

Thanks,
Scot


On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:

> Yeah, I think expanding to 8 will make a huge difference.  Between the
> initial request and the subsequent AJAX requests, I can imagine 4
> getting used up by one or two resolution requests.
> 
> -Ross.
> 
> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton <std5 at nyu.edu> wrote:
>> We were previously running 4.
>> 
>> Thanks,
>> Scot
>> 
>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>> 
>>> Scot,
>>> 
>>> How many were you running?
>>> 
>>> -Ross.
>>> 
>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>> 
>>>> Thanks again,
>>>> Scot
>>>> 
>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>> 
>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>> 
>>>>> -----Original Message-----
>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>> To: umlaut-general at rubyforge.org
>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>> 
>>>>> Any help would be greatly appreciated.
>>>>> 
>>>>> Thanks,
>>>>> Scot
>>>>> 
>>>>> --
>>>>> Scot Dalton
>>>>> Phone:  (212) 998-2674
>>>>> Web Services
>>>>> Division of Libraries
>>>>> New York University
>>>>> 
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>> 
>>>> _______________________________________________
>>>> Umlaut-general mailing list
>>>> Umlaut-general at rubyforge.org
>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>> 
>> 
>> 


From std5 at nyu.edu  Wed Apr 13 12:37:58 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 12:37:58 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <4DA5D09E.1020805@jhu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<4DA5D09E.1020805@jhu.edu>
Message-ID: <9CBDA079-6DFF-4CEA-BD6C-C3DE9B8418DA@nyu.edu>

Thanks for the explanation.  That definitely helps.  Are there significant changes in Rails 3 that would change anything explained below?

-Scot

On Apr 13, 2011, at Apr 13, 12:34 PM, Jonathan Rochkind wrote:

> Part of why Umlaut may need more instances than a typical Rails app is because Umlaut's requests can take so long to complete -- mainly because of waiting on SFX.  Another possible reason is that Umlaut's background processing makes an instance somewhat busier than a typical Rails instance, which does no background threaded processing like this -- I am not entirely sure, but think that background processing probably means more instances than usual are indicated, especially on a multi-CPU box, to spread the load out.
> 
> So I haven't actually done the work to profile, but these are some logical reasons that you might need to give more instances to Umlaut than would be suggested by general purpose Rails guidelines you may find on the web.  Umlaut instances are both slower to process a request-response cycle, as well as busier with background threads, than a typical Rails app instance.  Which may indicate a requirement for more instances in the pool than typical for Rails apps.  (And if so, even if you were deploying under Passenger, would probably suggest instructing Passenger to spin up more instances than typical. Passenger too has multiple instances of your app, it just handles it all transparently, spinning them up and down as needed. However, you can give Passenger instructions instead of using defaults).
> 
> One thing to be aware of when adding mongrel instances is whether you have enough physical RAM to contain them all.  If you add more instances than you have physical RAM to contain, then the swapping in and out from disk to RAM, between the various instances MAY negate any advantage you get by adding the instance.
> 
> Jonathan
> 
> On 4/13/2011 12:13 PM, Ross Singer wrote:
>> Yeah, I think expanding to 8 will make a huge difference.  Between the
>> initial request and the subsequent AJAX requests, I can imagine 4
>> getting used up by one or two resolution requests.
>> 
>> -Ross.
>> 
>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>> We were previously running 4.
>>> 
>>> Thanks,
>>> Scot
>>> 
>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>> 
>>>> Scot,
>>>> 
>>>> How many were you running?
>>>> 
>>>> -Ross.
>>>> 
>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>> 
>>>>> Thanks again,
>>>>> Scot
>>>>> 
>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>> 
>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>> To: umlaut-general at rubyforge.org
>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>> 
>>>>>> Any help would be greatly appreciated.
>>>>>> 
>>>>>> Thanks,
>>>>>> Scot
>>>>>> 
>>>>>> --
>>>>>> Scot Dalton
>>>>>> Phone:  (212) 998-2674
>>>>>> Web Services
>>>>>> Division of Libraries
>>>>>> New York University
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Umlaut-general mailing list
>>>>>> Umlaut-general at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>> _______________________________________________
>>>>>> Umlaut-general mailing list
>>>>>> Umlaut-general at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>> 
>>> 
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
> 


From rossfsinger at gmail.com  Wed Apr 13 13:22:37 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Wed, 13 Apr 2011 13:22:37 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
Message-ID: <BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>

On Wed, Apr 13, 2011 at 12:34 PM, Scot Dalton <std5 at nyu.edu> wrote:
> Great. ?Do you think an additional few processes (say going from 8 to 12) would help?

Not necessarily.  You only need as many processes as you have peak
simultaneous blocking requests.  Unused processes are sitting there
wasting memory that other things could be using.

It's definitely worth monitoring apache's mod_proxy_balancer (assuming
you're using Apache, of course) and thin/mongrel's cluster usage.  But
if you're seeing improvement in moderately high usage but still
finding performance degradation at peak (and you have memory to
spare), sure, up it to 12.

Googling on this (unfortunately, I can't find anything that helps
monitor cluster usage) did lead me to the fact that Apache's
mod_proxy_balancer apparently isn't very sophisticated in how it
chooses to balance among mongrels/thins.  Basically if it's up
(regardless of whether or not it's busy) mpb *may* try to send a
request to the same port (because it's unaware that it's occupied with
another request and unable to process another).

One possible remedy, if you're using Apache >=2.2.10 is to use the
proxy method lbmethod=bybusyness which will automatically queue to the
worker with the fewest pending requests.

Of course, another approach, as Jonathan pointed out, would be to
migrate to Passenger.

Do you have an idea how long an average request takes?  Is there a
kind of request that takes noticeably longer than others (that is, is
the initial browser request the longest?  XHR requests?)?

-Ross.
>
> Thanks,
> Scot
>
>
> On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:
>
>> Yeah, I think expanding to 8 will make a huge difference. ?Between the
>> initial request and the subsequent AJAX requests, I can imagine 4
>> getting used up by one or two resolution requests.
>>
>> -Ross.
>>
>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>> We were previously running 4.
>>>
>>> Thanks,
>>> Scot
>>>
>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>>
>>>> Scot,
>>>>
>>>> How many were you running?
>>>>
>>>> -Ross.
>>>>
>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>>>> Thanks for the suggestions. ?I increased the number of processes running to 8 and will continue to monitor. ?We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>>
>>>>> Thanks again,
>>>>> Scot
>>>>>
>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>>
>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>> To: umlaut-general at rubyforge.org
>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>>
>>>>>> Any help would be greatly appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Scot
>>>>>>
>>>>>> --
>>>>>> Scot Dalton
>>>>>> Phone: ?(212) 998-2674
>>>>>> Web Services
>>>>>> Division of Libraries
>>>>>> New York University
>>>>>>
>>>>>> _______________________________________________
>>>>>> Umlaut-general mailing list
>>>>>> Umlaut-general at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>> _______________________________________________
>>>>>> Umlaut-general mailing list
>>>>>> Umlaut-general at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>
>>>>> _______________________________________________
>>>>> Umlaut-general mailing list
>>>>> Umlaut-general at rubyforge.org
>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>
>>>
>>>
>
>

From jronallo at gmail.com  Wed Apr 13 14:01:00 2011
From: jronallo at gmail.com (Jason Ronallo)
Date: Wed, 13 Apr 2011 14:01:00 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
Message-ID: <BANLkTi=C9jd8vZhO6jAVHVmGcwOaYW4M5Q@mail.gmail.com>

You may also check out performance monitoring like newrelic [1]. It
can tell you  which requests and methods are slowest and what the
average response time is. You can also see whether any bottleneck is
something like the database request or something else. Even running
this in developer mode on your own machine can help you find problems.

Jason

[1] http://newrelic.com/

On Tue, Apr 12, 2011 at 2:11 PM, Scot Dalton <scot.Dalton at nyu.edu> wrote:
> Hi all,
>
> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>
> Any help would be greatly appreciated.
>
> Thanks,
> Scot
>
> --
> Scot Dalton
> Phone: ?(212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From rochkind at jhu.edu  Wed Apr 13 13:16:48 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Wed, 13 Apr 2011 13:16:48 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <9CBDA079-6DFF-4CEA-BD6C-C3DE9B8418DA@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<4DA5D09E.1020805@jhu.edu>
	<9CBDA079-6DFF-4CEA-BD6C-C3DE9B8418DA@nyu.edu>
Message-ID: <4DA5DA80.4000408@jhu.edu>

On 4/13/2011 12:37 PM, Scot Dalton wrote:
> Thanks for the explanation.  That definitely helps.  Are there significant changes in Rails 3 that would change anything explained below?

Only a bit, sort of.  Rails3 ActiveRecord can handle concurrent database 
access in more clearly documented ways, such that I can re-architect the 
"background" database access a bit so accessing the database in one 
thread will be less likely to 'stop the world' for other threads, or at 
least have the interruption be brief.

Really that's available even in later versions of Rails2 than Umlaut 
runs, but I haven't had time to upgrade, plan at this point to just go 
all the way to Rails3.

Additionally, as I mentioned before, the "mysql2" adapter, instead of 
the "mysql" one we use now, behaves better under multi-threaded 
concurrency. Which I think theoretically could be used in Rails2 too, 
but again I haven't had time to try it, and it _definitely_ works with 
Rails3.  (It does require ruby 1.8.7, as does Rails3 in general. I am 
still running Umlaut on ruby 1.8.6 here, and haven't tested it under 
1.8.7).

And on top of that, Rails3 is known to work under both "MRI" ruby 1.9 
and jruby (1.8 or maybe 1.9 too) -- both of which handle threading 
better than ruby 1.8.   I don't know if I'll have time to investigate 
Umlaut under ruby 1.9 or jruby myself, but the Rails3 upgrade will set 
the stage for such an investigation with higher chance of success than 
currently. (Although Ross has told us that at some point in the past he 
did run Umlaut under jruby, I can't say for sure if the current Umlaut 
will do fine under jruby or not. I don't believe Rails2 will work under 
MRI ruby 1.9).

So, phew, right?  But basically there are some things that can be done 
to make the multi-threaded concurrency perform better, that will be 
easier under Rails3.

Nothing much that can be done about SFX being so slow.


From scot.dalton at nyu.edu  Wed Apr 13 16:19:49 2011
From: scot.dalton at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 16:19:49 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
	<BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>
Message-ID: <1C0649FC-4D08-490C-A4C5-E0B4B2CF40DE@nyu.edu>

With 8 processes running, we're definitely seeing an improvement in performance.  We may bump it up to 10, but are still investigating.

On average (very rudimentary investigation), it looks like the initial browser request is taking about 6 seconds (slowest was 10 seconds) and the XHR requests are around 3 seconds each.  This analysis is a combination of looking at packets sniffed with wireshark[1] and using browser developer tools.  Any other tools you can recommend to get a better idea of what's happening?

Thanks,
Scot

[1] http://www.wireshark.org/

On Apr 13, 2011, at Apr 13, 1:22 PM, Ross Singer wrote:

> On Wed, Apr 13, 2011 at 12:34 PM, Scot Dalton <std5 at nyu.edu> wrote:
>> Great.  Do you think an additional few processes (say going from 8 to 12) would help?
> 
> Not necessarily.  You only need as many processes as you have peak
> simultaneous blocking requests.  Unused processes are sitting there
> wasting memory that other things could be using.
> 
> It's definitely worth monitoring apache's mod_proxy_balancer (assuming
> you're using Apache, of course) and thin/mongrel's cluster usage.  But
> if you're seeing improvement in moderately high usage but still
> finding performance degradation at peak (and you have memory to
> spare), sure, up it to 12.
> 
> Googling on this (unfortunately, I can't find anything that helps
> monitor cluster usage) did lead me to the fact that Apache's
> mod_proxy_balancer apparently isn't very sophisticated in how it
> chooses to balance among mongrels/thins.  Basically if it's up
> (regardless of whether or not it's busy) mpb *may* try to send a
> request to the same port (because it's unaware that it's occupied with
> another request and unable to process another).
> 
> One possible remedy, if you're using Apache >=2.2.10 is to use the
> proxy method lbmethod=bybusyness which will automatically queue to the
> worker with the fewest pending requests.
> 
> Of course, another approach, as Jonathan pointed out, would be to
> migrate to Passenger.
> 
> Do you have an idea how long an average request takes?  Is there a
> kind of request that takes noticeably longer than others (that is, is
> the initial browser request the longest?  XHR requests?)?
> 
> -Ross.
>> 
>> Thanks,
>> Scot
>> 
>> 
>> On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:
>> 
>>> Yeah, I think expanding to 8 will make a huge difference.  Between the
>>> initial request and the subsequent AJAX requests, I can imagine 4
>>> getting used up by one or two resolution requests.
>>> 
>>> -Ross.
>>> 
>>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>>> We were previously running 4.
>>>> 
>>>> Thanks,
>>>> Scot
>>>> 
>>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>>> 
>>>>> Scot,
>>>>> 
>>>>> How many were you running?
>>>>> 
>>>>> -Ross.
>>>>> 
>>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton <std5 at nyu.edu> wrote:
>>>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>>> 
>>>>>> Thanks again,
>>>>>> Scot
>>>>>> 
>>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>>> 
>>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>>> To: umlaut-general at rubyforge.org
>>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>>> 
>>>>>>> Any help would be greatly appreciated.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Scot
>>>>>>> 
>>>>>>> --
>>>>>>> Scot Dalton
>>>>>>> Phone:  (212) 998-2674
>>>>>>> Web Services
>>>>>>> Division of Libraries
>>>>>>> New York University
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Umlaut-general mailing list
>>>>>>> Umlaut-general at rubyforge.org
>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>> _______________________________________________
>>>>>>> Umlaut-general mailing list
>>>>>>> Umlaut-general at rubyforge.org
>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Umlaut-general mailing list
>>>>>> Umlaut-general at rubyforge.org
>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>> 
>>>> 
>>>> 
>> 
>> 




--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From std5 at nyu.edu  Wed Apr 13 16:21:01 2011
From: std5 at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 16:21:01 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <BANLkTi=C9jd8vZhO6jAVHVmGcwOaYW4M5Q@mail.gmail.com>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<BANLkTi=C9jd8vZhO6jAVHVmGcwOaYW4M5Q@mail.gmail.com>
Message-ID: <80904791-90A1-4202-B10C-810136964531@nyu.edu>

Thanks, Jason.  We'll definitely investigate newrelic.

-Scot

On Apr 13, 2011, at Apr 13, 2:01 PM, Jason Ronallo wrote:

> You may also check out performance monitoring like newrelic [1]. It
> can tell you  which requests and methods are slowest and what the
> average response time is. You can also see whether any bottleneck is
> something like the database request or something else. Even running
> this in developer mode on your own machine can help you find problems.
> 
> Jason
> 
> [1] http://newrelic.com/
> 
> On Tue, Apr 12, 2011 at 2:11 PM, Scot Dalton <scot.Dalton at nyu.edu> wrote:
>> Hi all,
>> 
>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>> 
>> Any help would be greatly appreciated.
>> 
>> Thanks,
>> Scot
>> 
>> --
>> Scot Dalton
>> Phone:  (212) 998-2674
>> Web Services
>> Division of Libraries
>> New York University
>> 
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>> 
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general


From rochkind at jhu.edu  Wed Apr 13 16:46:07 2011
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Wed, 13 Apr 2011 16:46:07 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <1C0649FC-4D08-490C-A4C5-E0B4B2CF40DE@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
	<BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>
	<1C0649FC-4D08-490C-A4C5-E0B4B2CF40DE@nyu.edu>
Message-ID: <4DA60B8F.80501@jhu.edu>

Hmm, that's weird. For me, initial browser request is around 3-4 seconds 
(and 2-3 of that, from previous investigation, is just waiting on SFX).

And subsequent xhr's are only 1s or less.

Not sure why you are so much slower. could you have that much more 
traffic than me? I do robots.txt my entire umlaut installation (there 
are virtually infinite pages and no good reason to crawl them).

Do you run the nightly_maintenance task?  It deletes old un-needed 
things from the db, if you never run it your db tables can get gigantic 
and make mysql slow.

I'm not sure what else could be going on, weird.

No matter what you'll occasionally get pathologically slow initial 
requests when SFX itself is pathologically slow. SFX has some bugs and 
mis-designs where for some requests it is horribly slow.  But your 
subsequent xhr requests don't involve waiting on SFX, and should always 
be speedy.

Disturbing, sorry.

On 4/13/2011 4:19 PM, Scot Dalton wrote:
> With 8 processes running, we're definitely seeing an improvement in performance.  We may bump it up to 10, but are still investigating.
>
> On average (very rudimentary investigation), it looks like the initial browser request is taking about 6 seconds (slowest was 10 seconds) and the XHR requests are around 3 seconds each.  This analysis is a combination of looking at packets sniffed with wireshark[1] and using browser developer tools.  Any other tools you can recommend to get a better idea of what's happening?
>
> Thanks,
> Scot
>
> [1] http://www.wireshark.org/
>
> On Apr 13, 2011, at Apr 13, 1:22 PM, Ross Singer wrote:
>
>> On Wed, Apr 13, 2011 at 12:34 PM, Scot Dalton<std5 at nyu.edu>  wrote:
>>> Great.  Do you think an additional few processes (say going from 8 to 12) would help?
>> Not necessarily.  You only need as many processes as you have peak
>> simultaneous blocking requests.  Unused processes are sitting there
>> wasting memory that other things could be using.
>>
>> It's definitely worth monitoring apache's mod_proxy_balancer (assuming
>> you're using Apache, of course) and thin/mongrel's cluster usage.  But
>> if you're seeing improvement in moderately high usage but still
>> finding performance degradation at peak (and you have memory to
>> spare), sure, up it to 12.
>>
>> Googling on this (unfortunately, I can't find anything that helps
>> monitor cluster usage) did lead me to the fact that Apache's
>> mod_proxy_balancer apparently isn't very sophisticated in how it
>> chooses to balance among mongrels/thins.  Basically if it's up
>> (regardless of whether or not it's busy) mpb *may* try to send a
>> request to the same port (because it's unaware that it's occupied with
>> another request and unable to process another).
>>
>> One possible remedy, if you're using Apache>=2.2.10 is to use the
>> proxy method lbmethod=bybusyness which will automatically queue to the
>> worker with the fewest pending requests.
>>
>> Of course, another approach, as Jonathan pointed out, would be to
>> migrate to Passenger.
>>
>> Do you have an idea how long an average request takes?  Is there a
>> kind of request that takes noticeably longer than others (that is, is
>> the initial browser request the longest?  XHR requests?)?
>>
>> -Ross.
>>> Thanks,
>>> Scot
>>>
>>>
>>> On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:
>>>
>>>> Yeah, I think expanding to 8 will make a huge difference.  Between the
>>>> initial request and the subsequent AJAX requests, I can imagine 4
>>>> getting used up by one or two resolution requests.
>>>>
>>>> -Ross.
>>>>
>>>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>>> We were previously running 4.
>>>>>
>>>>> Thanks,
>>>>> Scot
>>>>>
>>>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>>>>
>>>>>> Scot,
>>>>>>
>>>>>> How many were you running?
>>>>>>
>>>>>> -Ross.
>>>>>>
>>>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>>>>
>>>>>>> Thanks again,
>>>>>>> Scot
>>>>>>>
>>>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>>>>
>>>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>>>> To: umlaut-general at rubyforge.org
>>>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>>>>
>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Scot
>>>>>>>>
>>>>>>>> --
>>>>>>>> Scot Dalton
>>>>>>>> Phone:  (212) 998-2674
>>>>>>>> Web Services
>>>>>>>> Division of Libraries
>>>>>>>> New York University
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Umlaut-general mailing list
>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>> _______________________________________________
>>>>>>>> Umlaut-general mailing list
>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>> _______________________________________________
>>>>>>> Umlaut-general mailing list
>>>>>>> Umlaut-general at rubyforge.org
>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>
>>>>>
>>>
>
>
>
> --
> Scot Dalton
> Phone:  (212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general


From scot.dalton at nyu.edu  Wed Apr 13 17:02:43 2011
From: scot.dalton at nyu.edu (Scot Dalton)
Date: Wed, 13 Apr 2011 17:02:43 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <4DA60B8F.80501@jhu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
	<BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>
	<1C0649FC-4D08-490C-A4C5-E0B4B2CF40DE@nyu.edu>
	<4DA60B8F.80501@jhu.edu>
Message-ID: <50A08BF6-FAAE-4E1D-9017-6B56B9B5308D@nyu.edu>

We have a lot of other services running on that box, so Umlaut may be competing for limited resources.  I'm hoping the new server will help with that.  We do run the nightly cron, so db size shouldn't be an issue and at non-peak times it's definitely quicker.  We have the Umlaut robot.txt in place.  Are there other paths your disallowing other than /link_router and /resolve?

Regarding my analysis, the sample was quite small.  We're looking into tools that will give us a better measure of request times, but we don't want to deploy them until we're on the new hardware, since we don't want to increase traffic at this time. (I assume these latency tools would need to increase traffic in order to get a measure of performance).

As far as traffic goes, we had 47000 page views on Monday and 42000 page views on Tuesday, and are expecting similar numbers for the rest of the semester, probably peaking near finals at 50 - 60K.  Not sure how that compares with other institutions.

Thanks,
Scot


On Apr 13, 2011, at Apr 13, 4:46 PM, Jonathan Rochkind wrote:

> Hmm, that's weird. For me, initial browser request is around 3-4 seconds (and 2-3 of that, from previous investigation, is just waiting on SFX).
> 
> And subsequent xhr's are only 1s or less.
> 
> Not sure why you are so much slower. could you have that much more traffic than me? I do robots.txt my entire umlaut installation (there are virtually infinite pages and no good reason to crawl them).
> 
> Do you run the nightly_maintenance task?  It deletes old un-needed things from the db, if you never run it your db tables can get gigantic and make mysql slow.
> 
> I'm not sure what else could be going on, weird.
> 
> No matter what you'll occasionally get pathologically slow initial requests when SFX itself is pathologically slow. SFX has some bugs and mis-designs where for some requests it is horribly slow.  But your subsequent xhr requests don't involve waiting on SFX, and should always be speedy.
> 
> Disturbing, sorry.
> 
> On 4/13/2011 4:19 PM, Scot Dalton wrote:
>> With 8 processes running, we're definitely seeing an improvement in performance.  We may bump it up to 10, but are still investigating.
>> 
>> On average (very rudimentary investigation), it looks like the initial browser request is taking about 6 seconds (slowest was 10 seconds) and the XHR requests are around 3 seconds each.  This analysis is a combination of looking at packets sniffed with wireshark[1] and using browser developer tools.  Any other tools you can recommend to get a better idea of what's happening?
>> 
>> Thanks,
>> Scot
>> 
>> [1] http://www.wireshark.org/
>> 
>> On Apr 13, 2011, at Apr 13, 1:22 PM, Ross Singer wrote:
>> 
>>> On Wed, Apr 13, 2011 at 12:34 PM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>> Great.  Do you think an additional few processes (say going from 8 to 12) would help?
>>> Not necessarily.  You only need as many processes as you have peak
>>> simultaneous blocking requests.  Unused processes are sitting there
>>> wasting memory that other things could be using.
>>> 
>>> It's definitely worth monitoring apache's mod_proxy_balancer (assuming
>>> you're using Apache, of course) and thin/mongrel's cluster usage.  But
>>> if you're seeing improvement in moderately high usage but still
>>> finding performance degradation at peak (and you have memory to
>>> spare), sure, up it to 12.
>>> 
>>> Googling on this (unfortunately, I can't find anything that helps
>>> monitor cluster usage) did lead me to the fact that Apache's
>>> mod_proxy_balancer apparently isn't very sophisticated in how it
>>> chooses to balance among mongrels/thins.  Basically if it's up
>>> (regardless of whether or not it's busy) mpb *may* try to send a
>>> request to the same port (because it's unaware that it's occupied with
>>> another request and unable to process another).
>>> 
>>> One possible remedy, if you're using Apache>=2.2.10 is to use the
>>> proxy method lbmethod=bybusyness which will automatically queue to the
>>> worker with the fewest pending requests.
>>> 
>>> Of course, another approach, as Jonathan pointed out, would be to
>>> migrate to Passenger.
>>> 
>>> Do you have an idea how long an average request takes?  Is there a
>>> kind of request that takes noticeably longer than others (that is, is
>>> the initial browser request the longest?  XHR requests?)?
>>> 
>>> -Ross.
>>>> Thanks,
>>>> Scot
>>>> 
>>>> 
>>>> On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:
>>>> 
>>>>> Yeah, I think expanding to 8 will make a huge difference.  Between the
>>>>> initial request and the subsequent AJAX requests, I can imagine 4
>>>>> getting used up by one or two resolution requests.
>>>>> 
>>>>> -Ross.
>>>>> 
>>>>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>>>> We were previously running 4.
>>>>>> 
>>>>>> Thanks,
>>>>>> Scot
>>>>>> 
>>>>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>>>>> 
>>>>>>> Scot,
>>>>>>> 
>>>>>>> How many were you running?
>>>>>>> 
>>>>>>> -Ross.
>>>>>>> 
>>>>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton<std5 at nyu.edu>  wrote:
>>>>>>>> Thanks for the suggestions.  I increased the number of processes running to 8 and will continue to monitor.  We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>>>>> 
>>>>>>>> Thanks again,
>>>>>>>> Scot
>>>>>>>> 
>>>>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>>>>> 
>>>>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>>>>> 
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>>>>> To: umlaut-general at rubyforge.org
>>>>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> Any recommendations on analyzing Umlaut performance?  We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times.  I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>>>>> 
>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Scot
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Scot Dalton
>>>>>>>>> Phone:  (212) 998-2674
>>>>>>>>> Web Services
>>>>>>>>> Division of Libraries
>>>>>>>>> New York University
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> Umlaut-general mailing list
>>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>>> _______________________________________________
>>>>>>>>> Umlaut-general mailing list
>>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>> _______________________________________________
>>>>>>>> Umlaut-general mailing list
>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>> 
>>>>>> 
>>>> 
>> 
>> 
>> 
>> --
>> Scot Dalton
>> Phone:  (212) 998-2674
>> Web Services
>> Division of Libraries
>> New York University
>> 
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
> 


--
Scot Dalton
Phone:  (212) 998-2674
Web Services
Division of Libraries
New York University


From rossfsinger at gmail.com  Wed Apr 13 21:26:51 2011
From: rossfsinger at gmail.com (Ross Singer)
Date: Wed, 13 Apr 2011 21:26:51 -0400
Subject: [Umlaut-general] Umlaut Performance
In-Reply-To: <50A08BF6-FAAE-4E1D-9017-6B56B9B5308D@nyu.edu>
References: <959D73BA-C5F9-4F6F-92D9-79567ADF2C74@nyu.edu>
	<7288D49C0448E840B752320350AFA94E867EE16F@its-hcwnem03.ds.Vanderbilt.edu>
	<04097D2E-3838-42D8-B0A8-A8AF997B2452@nyu.edu>
	<BANLkTimby7c5TigT9icmAtJoY+dNYCzghw@mail.gmail.com>
	<F1A81699-C376-47C9-8A06-74EACDF17E77@nyu.edu>
	<BANLkTimViPfwXnzaT4zJUmiDKo0QhQYbvA@mail.gmail.com>
	<0C7FA03C-487C-4E32-9CDE-443D46F20EC7@nyu.edu>
	<BANLkTinOHsT_FkkJCyxC29S3Xv9QJ-xDXw@mail.gmail.com>
	<1C0649FC-4D08-490C-A4C5-E0B4B2CF40DE@nyu.edu>
	<4DA60B8F.80501@jhu.edu>
	<50A08BF6-FAAE-4E1D-9017-6B56B9B5308D@nyu.edu>
Message-ID: <BANLkTikjB6UFeheBMJvcZhGmd8J1OE9W=A@mail.gmail.com>

Ok, so about 1 page view/FTE/day :)  About what we had at GaTech.

Really, I'd be surprised if the other services were the problem
(unless there's a lot of swapping going on).

For your initial request, that seems way slow.

Just to be sure, is Umlaut definitely running in production mode?

What version of Ruby are you running?

What kind of responses do you get if you "time curl" an openurl to SFX's API?

It just took forever for https://getit.library.nyu.edu/ to load for me
(although resolving the journals themselves didn't take that long).

One request for the "home page" took 15s.

time curl "https://getit.library.nyu.edu/resolve?url_ver=Z39.88-2004&url_ctx_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Actx&ctx_tim=2011-04-13T20%3A54%3A38-04%3A00&ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&ctx_id=&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft_id=info%3Adoi%2F10.1086%2F300510&rfr_id=info%3Asid%2Fsfxit.com%3Acitation"

Usually returned in ~4.5-5 seconds.  Once it took 13s.  A low of 3.7s.

Weird.

So ignore my question about SFX, it seems fine - it seems like your
thin cluster is either getting starved for some reason or not utilized
properly.  I guess I would check to make sure all 8-10 (whatever) are
actually being proxied (and used).

One neat thing about Thin is that it's evented, so when you use curl
you can sort of see where it's getting hung up, if it's in the view
stage.

When I get hanging like this (which, admittedly, isn't consistent), it
happens around this part of the view:

<li class="response_item">

  <!-- fancy schmancy ajax? But not if we're generating embed partials,
       won't work. -->

    <a href="/link_router/index/26198633" id="svctype_link_26198633"
target="_blank">to RefWorks </a>











<!-- debug info, hidden with CSS, can be made visible with javascript. -->
<div class="debug_info" style="display:none">
  <b>Debug info:</b> NYU_SFX


    ;  Target: REFWORKS_EXPORT_TOOL ; SFX object ID: 954922828207


</div>


</li>


<li class="response_item">

  <!-- fancy schmancy ajax? But not if we're generating embed partials,
       won't work. -->

    <a href="/link_router/index/26198635" id="svctype_link_26198635"
target="_blank">to EndNote </a>

Somewhere in there.

I don't really know if there's a helper or something firing at this
point, but I would say that there is *some* clue there.

Majority of requests just vomit all the HTML in a burst, though,
regardless of their delay.  And usually that's in the sub 4 sec.
range.

So, yeah, I'm going to say that a lot of your problems will be solved
with your new server :)

If the time curl/SFX API results come out about near the same as
Umlaut's initial request, I think it might be worthwhile to reopen an
idea I had a long time ago, which is to bypass the SFX API in favor of
semantic xhtml in the regular SFX templates.  After all, if the URLs
route through SFX, that helps statistics, anyway, right?

-Ross.

On Wed, Apr 13, 2011 at 5:02 PM, Scot Dalton <scot.dalton at nyu.edu> wrote:
> We have a lot of other services running on that box, so Umlaut may be competing for limited resources. ?I'm hoping the new server will help with that. ?We do run the nightly cron, so db size shouldn't be an issue and at non-peak times it's definitely quicker. ?We have the Umlaut robot.txt in place. ?Are there other paths your disallowing other than /link_router and /resolve?
>
> Regarding my analysis, the sample was quite small. ?We're looking into tools that will give us a better measure of request times, but we don't want to deploy them until we're on the new hardware, since we don't want to increase traffic at this time. (I assume these latency tools would need to increase traffic in order to get a measure of performance).
>
> As far as traffic goes, we had 47000 page views on Monday and 42000 page views on Tuesday, and are expecting similar numbers for the rest of the semester, probably peaking near finals at 50 - 60K. ?Not sure how that compares with other institutions.
>
> Thanks,
> Scot
>
>
> On Apr 13, 2011, at Apr 13, 4:46 PM, Jonathan Rochkind wrote:
>
>> Hmm, that's weird. For me, initial browser request is around 3-4 seconds (and 2-3 of that, from previous investigation, is just waiting on SFX).
>>
>> And subsequent xhr's are only 1s or less.
>>
>> Not sure why you are so much slower. could you have that much more traffic than me? I do robots.txt my entire umlaut installation (there are virtually infinite pages and no good reason to crawl them).
>>
>> Do you run the nightly_maintenance task? ?It deletes old un-needed things from the db, if you never run it your db tables can get gigantic and make mysql slow.
>>
>> I'm not sure what else could be going on, weird.
>>
>> No matter what you'll occasionally get pathologically slow initial requests when SFX itself is pathologically slow. SFX has some bugs and mis-designs where for some requests it is horribly slow. ?But your subsequent xhr requests don't involve waiting on SFX, and should always be speedy.
>>
>> Disturbing, sorry.
>>
>> On 4/13/2011 4:19 PM, Scot Dalton wrote:
>>> With 8 processes running, we're definitely seeing an improvement in performance. ?We may bump it up to 10, but are still investigating.
>>>
>>> On average (very rudimentary investigation), it looks like the initial browser request is taking about 6 seconds (slowest was 10 seconds) and the XHR requests are around 3 seconds each. ?This analysis is a combination of looking at packets sniffed with wireshark[1] and using browser developer tools. ?Any other tools you can recommend to get a better idea of what's happening?
>>>
>>> Thanks,
>>> Scot
>>>
>>> [1] http://www.wireshark.org/
>>>
>>> On Apr 13, 2011, at Apr 13, 1:22 PM, Ross Singer wrote:
>>>
>>>> On Wed, Apr 13, 2011 at 12:34 PM, Scot Dalton<std5 at nyu.edu> ?wrote:
>>>>> Great. ?Do you think an additional few processes (say going from 8 to 12) would help?
>>>> Not necessarily. ?You only need as many processes as you have peak
>>>> simultaneous blocking requests. ?Unused processes are sitting there
>>>> wasting memory that other things could be using.
>>>>
>>>> It's definitely worth monitoring apache's mod_proxy_balancer (assuming
>>>> you're using Apache, of course) and thin/mongrel's cluster usage. ?But
>>>> if you're seeing improvement in moderately high usage but still
>>>> finding performance degradation at peak (and you have memory to
>>>> spare), sure, up it to 12.
>>>>
>>>> Googling on this (unfortunately, I can't find anything that helps
>>>> monitor cluster usage) did lead me to the fact that Apache's
>>>> mod_proxy_balancer apparently isn't very sophisticated in how it
>>>> chooses to balance among mongrels/thins. ?Basically if it's up
>>>> (regardless of whether or not it's busy) mpb *may* try to send a
>>>> request to the same port (because it's unaware that it's occupied with
>>>> another request and unable to process another).
>>>>
>>>> One possible remedy, if you're using Apache>=2.2.10 is to use the
>>>> proxy method lbmethod=bybusyness which will automatically queue to the
>>>> worker with the fewest pending requests.
>>>>
>>>> Of course, another approach, as Jonathan pointed out, would be to
>>>> migrate to Passenger.
>>>>
>>>> Do you have an idea how long an average request takes? ?Is there a
>>>> kind of request that takes noticeably longer than others (that is, is
>>>> the initial browser request the longest? ?XHR requests?)?
>>>>
>>>> -Ross.
>>>>> Thanks,
>>>>> Scot
>>>>>
>>>>>
>>>>> On Apr 13, 2011, at Apr 13, 12:13 PM, Ross Singer wrote:
>>>>>
>>>>>> Yeah, I think expanding to 8 will make a huge difference. ?Between the
>>>>>> initial request and the subsequent AJAX requests, I can imagine 4
>>>>>> getting used up by one or two resolution requests.
>>>>>>
>>>>>> -Ross.
>>>>>>
>>>>>> On Wed, Apr 13, 2011 at 11:40 AM, Scot Dalton<std5 at nyu.edu> ?wrote:
>>>>>>> We were previously running 4.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Scot
>>>>>>>
>>>>>>> On Apr 13, 2011, at Apr 13, 10:12 AM, Ross Singer wrote:
>>>>>>>
>>>>>>>> Scot,
>>>>>>>>
>>>>>>>> How many were you running?
>>>>>>>>
>>>>>>>> -Ross.
>>>>>>>>
>>>>>>>> On Wed, Apr 13, 2011 at 9:46 AM, Scot Dalton<std5 at nyu.edu> ?wrote:
>>>>>>>>> Thanks for the suggestions. ?I increased the number of processes running to 8 and will continue to monitor. ?We'll also be moving to a new server at the end of the month, which will hopefully help.
>>>>>>>>>
>>>>>>>>> Thanks again,
>>>>>>>>> Scot
>>>>>>>>>
>>>>>>>>> On Apr 12, 2011, at Apr 12, 2:14 PM, Poulter, Dale wrote:
>>>>>>>>>
>>>>>>>>>> We experienced some performance increase by increasing the number of mongrels running and run the night maintenance --well nightly.
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: umlaut-general-bounces at rubyforge.org [mailto:umlaut-general-bounces at rubyforge.org] On Behalf Of Scot Dalton
>>>>>>>>>> Sent: Tuesday, April 12, 2011 1:11 PM
>>>>>>>>>> To: umlaut-general at rubyforge.org
>>>>>>>>>> Subject: [Umlaut-general] Umlaut Performance
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Any recommendations on analyzing Umlaut performance? ?We're getting upwards of 4K sessions a day and are seeing serious performance degradation at peak times. ?I'd like to be able to see if there is anything in our code that is causing a delay, or if it's just a question of load and beefing up the resources on the box and/or getting the web server configured correctly.
>>>>>>>>>>
>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Scot
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Scot Dalton
>>>>>>>>>> Phone: ?(212) 998-2674
>>>>>>>>>> Web Services
>>>>>>>>>> Division of Libraries
>>>>>>>>>> New York University
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Umlaut-general mailing list
>>>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Umlaut-general mailing list
>>>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>>> _______________________________________________
>>>>>>>>> Umlaut-general mailing list
>>>>>>>>> Umlaut-general at rubyforge.org
>>>>>>>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Scot Dalton
>>> Phone: ?(212) 998-2674
>>> Web Services
>>> Division of Libraries
>>> New York University
>>>
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>
>
>
> --
> Scot Dalton
> Phone: ?(212) 998-2674
> Web Services
> Division of Libraries
> New York University
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From dale.poulter at Vanderbilt.Edu  Mon Apr 25 08:56:13 2011
From: dale.poulter at Vanderbilt.Edu (Poulter,   Dale)
Date: Mon, 25 Apr 2011 07:56:13 -0500
Subject: [Umlaut-general] non-latin display problem
Message-ID: <7288D49C0448E840B752320350AFA94E89DCE624@its-hcwnem03.ds.Vanderbilt.edu>

All,

Has anyone encountered the title displaying as ?????????? for a title that is using non-latin1 characters?  This seems to display correctly in the native SFX screen but not umlaut.  Thanks for any suggestions.

Example:
Mezhdunarodnai?a zhizn?

http://umlaut.library.vanderbilt.edu/go/131692


--Dale

---------------------------------------
Dale Poulter
Automation Coordinator
Library Information Technology Services
Vanderbilt University
419 21st Avenue South, Room 812
Nashville, TN  37203-2427
(615)343-5388
(615)343-8834 (fax)
(615)207-9705 (cell)
dale.poulter at vanderbilt.edu<mailto:dale.poulter at vanderbilt.edu>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://rubyforge.org/pipermail/umlaut-general/attachments/20110425/4f8f23d7/attachment-0001.html>