From jronallo at gmail.com  Sat May 10 16:02:16 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Sat, 10 May 2008 16:02:16 -0400
Subject: [Umlaut-general] 1st resolution
Message-ID: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>

Hi, Jonathan,
Just wanted to let you know that I set up Umlaut and got my first link
to resolve. I turned off all services except SFX and PubMed. As I get
API keys to other services, I'll be turning those on as well. Is there
any publicly accessible catalog that would work with the Umlaut for
testing purposes?

Because I wasn't using a direct connection to the SFX database, things
like title search (with auto suggestions?) and A-Z list failed. For a
journal title search I get the error that the table
umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
load_sfx_urls rake task which draws in data directly from the SFX
database? If I set config.app_config.use_umlaut_journal_index to true,
then it fails because acts_as_ferret isn't loaded. I suppose the
needed info hasn't been pre-fetched from the SFX server database. Was
acts_as_ferret used by Ross to do things like categories?

I know the above is confusing, but I'm a bit confused about how A-Z
list and journal search works right now.

Would it be possible to have a third option which would let Umalut do
title searches and A-Z list dynamically by querying the SFX server
without direct database access or preloading a database? Would the SFX
API help out here? Might be slow, but also the quickest way to get up
and testing it out.

In any case A-Z list and journal search were my biggest problems
getting it up and running. Well, because those failed I wasn't sure it
was up and running right. I could see commenting out many more of the
services in the config files so it's quicker to get that first link
resolved.

Would it be helpful to include a step by step on creating a mysql
development database?

Now that I've got it up once, I'm going to try installation and
configuration again on a fresh install of Ubuntu.

Jason

From rossfsinger at gmail.com  Mon May 12 16:31:25 2008
From: rossfsinger at gmail.com (Ross Singer)
Date: Mon, 12 May 2008 16:31:25 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
Message-ID: <23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>

Wow, I think is, like, our first ever email...  And it's (mostly) a
success story!

Jason, I'll reply inline.

On Sat, May 10, 2008 at 4:02 PM, Jason Ronallo <jronallo at gmail.com> wrote:
> Hi, Jonathan,
>  Just wanted to let you know that I set up Umlaut and got my first link
>  to resolve. I turned off all services except SFX and PubMed. As I get
>  API keys to other services, I'll be turning those on as well. Is there
>  any publicly accessible catalog that would work with the Umlaut for
>  testing purposes?

You can probably also turn on Crossref -- the API key is optional and
there's the possibility that they can throttle you, but I think it's
probably pretty unlikely.
>
>  Because I wasn't using a direct connection to the SFX database, things
>  like title search (with auto suggestions?) and A-Z list failed. For a
>  journal title search I get the error that the table
>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>  load_sfx_urls rake task which draws in data directly from the SFX
>  database? If I set config.app_config.use_umlaut_journal_index to true,
>  then it fails because acts_as_ferret isn't loaded. I suppose the
>  needed info hasn't been pre-fetched from the SFX server database. Was
>  acts_as_ferret used by Ross to do things like categories?

What SFX server are you using?  I didn't directly go off the DB either
(since SFX was hosted consortially for Tech, this wasn't an option).
Instead, I had a python script to import the tab delimited export
files into mysql (doing this in python took about 1/5 the time of the
equivalent ruby script) and then I'd reindex the ferret index after it
completed.  At Tech, we didn't use the categories, since the
librarians didn't like them.
>
>  I know the above is confusing, but I'm a bit confused about how A-Z
>  list and journal search works right now.

In my setup it was shaky and fragile.
>
>  Would it be possible to have a third option which would let Umalut do
>  title searches and A-Z list dynamically by querying the SFX server
>  without direct database access or preloading a database? Would the SFX
>  API help out here? Might be slow, but also the quickest way to get up
>  and testing it out.

This won't work, sadly.  The API will time out before you could
possibly load the 'j' page on even the smallest of SFX collections.
You'd also have problems with autocomplete.
>
>  In any case A-Z list and journal search were my biggest problems
>  getting it up and running. Well, because those failed I wasn't sure it
>  was up and running right. I could see commenting out many more of the
>  services in the config files so it's quicker to get that first link
>  resolved.

This makes sense.
>
>  Would it be helpful to include a step by step on creating a mysql
>  development database?

Do the migration scripts not work right?
>
>  Now that I've got it up once, I'm going to try installation and
>  configuration again on a fresh install of Ubuntu.

While you're at it, it might be interesting to try it with postgres or
firebird or something.  I mean, if you're up to it.

-Ross.

From rochkind at jhu.edu  Mon May 12 16:57:45 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 12 May 2008 16:57:45 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
Message-ID: <4828AF49.8010406@jhu.edu>

Ross Singer wrote:
> This won't work, sadly. The API will time out before you could
> possibly load the 'j' page on even the smallest of SFX collections.
> You'd also have problems with autocomplete.
>   
Interestingly, I know have autocomplete targetted directly against the 
SFX db, and it seems to be working fine and not slowing down the SFX 
server too much. I was emboldened to do this when I realized that SFX 
itself now includes optional autocomplete functionality (no doubt they 
got the idea from Ross), which goes directly against the SFX db.

But yeah, at present, the A-Z list functionality _I_ use goes against 
the db, and I can't vouch for the fucntionality that relied on an 
imported list instead. I would actually like to get that functionality 
working again at some point (although if you HAVE direct db access, you 
could do the local index importing directly from the db, instead of the 
SFX export tab deliminted file. You might still want a local index to 
add other entries to it that do not come from SFX; that's what I'd like 
to do some day).

Jonathan

From rochkind at jhu.edu  Mon May 12 16:58:09 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 12 May 2008 16:58:09 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
Message-ID: <4828AF61.3070408@jhu.edu>

Oops, didn't realize that Jason sent this to the umlaut list! Here was
my reply this morning, that I should have sent here (and thanks too to
Ross for the reply! Hopefully our replies agree. :) ).

Hi Jason, yeah I can see that being confusing, sorry.

The answer is basically that Umlaut, when I got it, did build it's own
journal index based on the SFX "journal export" file. I didn't like
this, I wanted Umlaut to consult the SFX db directly instead, so I added
that functionality. And didn't pay any attention to the original
functionality, so it's probably stopped working---but even if it is
working, it needed to be 'primed' by importing an SFX export file, and I
didn't write any documentation for that, because I was ignoring it.
(Almost all documentation was written by me).
If you're interested, that could be a project for you to make sure that
functionality is working, and document it. But I wouldn't recommend that
as an initial project.  To start out with, it should be quite fine to
use Umlaut without A-Z list lookup functionality. Are you pointing it at
your own university's SFX for now?

Your idea of a 'third' option--the reason that Ross originally had
Umlaut build it's own index from an SFX export file, is that the SFX API
turns out _not_ to work for A-Z list lookup. The more "hits" there are
in SFX API results, the slower SFX is to respond. With 100 hits
returned---SFX is so slow that apache starts timing out. So there only
good way to build a local index in Umlaut is with either an SFX journal
export file (david walker for Xerxes uses the one intended for Google
Scholar; Ross for Umlaut originally used the 'standard' one), or direct
db access. I would be interested in  Umlaut working with a local journal
index that could be built from _any_ of these sources at your option.
But just the API isn't going to work.

I thought I _had_ step-by-step instructions for creating a mysql dev
database!  The instructions in the docs can be used to create a dev db
and a production db, no?    I guess we might want to make more prominent
instructions that at present Umlaut A-Z and journal-name lookup is only
tested with direct connection to SFX db.

For catalogs, pretty much only the Horizon adapter is actually tested
and working. Ross had a voyager adapter too, but I haven't kept it up,
and I"m sure it doesn't work out of the box right now.  You are welcome
to point your Umlaut at my Horizon OPAC instance. It doesn't actually
require any special authorization, just config the service like this:

JH_HIP:
name: JH Libraries Catalog
display_name: JH Libraries Catalog
type: Hip3Service
priority: c # After SFX, to get SFX metadata enhancement
base_path: http://catalog.library.jhu.edu/ipac20/ipac.jsp


Hope this helps clear things up. Feel free to update any documentation
any way you think is appropriate. Happy to have phone conversation or IM
conversation if you would find it helpful.

I'm also realizing that the view code I am using is included with the
distro, but is not the default one. I left Ross's view code the default
one, when GATech was still using it. But I'm not completely sure the
default one works completely! It might make more sense to change mine to
be the default one.  But if you want to configure your instance to use
mine, but this in in your umlaut_config.environment.rb:
config.app_config.resolve_view = "alternate/resolve_alternate"

PPS: Feel free to change the sample config/umlaut_distribution services,
institutions, and environment files in whatever way you think is best
and commit them.


Ross Singer wrote:
> Wow, I think is, like, our first ever email...  And it's (mostly) a
> success story!
>
> Jason, I'll reply inline.
>
> On Sat, May 10, 2008 at 4:02 PM, Jason Ronallo <jronallo at gmail.com> wrote:
>   
>> Hi, Jonathan,
>>  Just wanted to let you know that I set up Umlaut and got my first link
>>  to resolve. I turned off all services except SFX and PubMed. As I get
>>  API keys to other services, I'll be turning those on as well. Is there
>>  any publicly accessible catalog that would work with the Umlaut for
>>  testing purposes?
>>     
>
> You can probably also turn on Crossref -- the API key is optional and
> there's the possibility that they can throttle you, but I think it's
> probably pretty unlikely.
>   
>>  Because I wasn't using a direct connection to the SFX database, things
>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>  journal title search I get the error that the table
>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>  load_sfx_urls rake task which draws in data directly from the SFX
>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>  acts_as_ferret used by Ross to do things like categories?
>>     
>
> What SFX server are you using?  I didn't directly go off the DB either
> (since SFX was hosted consortially for Tech, this wasn't an option).
> Instead, I had a python script to import the tab delimited export
> files into mysql (doing this in python took about 1/5 the time of the
> equivalent ruby script) and then I'd reindex the ferret index after it
> completed.  At Tech, we didn't use the categories, since the
> librarians didn't like them.
>   
>>  I know the above is confusing, but I'm a bit confused about how A-Z
>>  list and journal search works right now.
>>     
>
> In my setup it was shaky and fragile.
>   
>>  Would it be possible to have a third option which would let Umalut do
>>  title searches and A-Z list dynamically by querying the SFX server
>>  without direct database access or preloading a database? Would the SFX
>>  API help out here? Might be slow, but also the quickest way to get up
>>  and testing it out.
>>     
>
> This won't work, sadly.  The API will time out before you could
> possibly load the 'j' page on even the smallest of SFX collections.
> You'd also have problems with autocomplete.
>   
>>  In any case A-Z list and journal search were my biggest problems
>>  getting it up and running. Well, because those failed I wasn't sure it
>>  was up and running right. I could see commenting out many more of the
>>  services in the config files so it's quicker to get that first link
>>  resolved.
>>     
>
> This makes sense.
>   
>>  Would it be helpful to include a step by step on creating a mysql
>>  development database?
>>     
>
> Do the migration scripts not work right?
>   
>>  Now that I've got it up once, I'm going to try installation and
>>  configuration again on a fresh install of Ubuntu.
>>     
>
> While you're at it, it might be interesting to try it with postgres or
> firebird or something.  I mean, if you're up to it.
>
> -Ross.
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>   

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886
rochkind (at) jhu.edu



From jronallo at gmail.com  Sat May 17 20:56:40 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Sat, 17 May 2008 20:56:40 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
Message-ID: <763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>

Hi, Ross,
Response inline.

On Mon, May 12, 2008 at 4:31 PM, Ross Singer <rossfsinger at gmail.com> wrote:
> Wow, I think is, like, our first ever email...  And it's (mostly) a
> success story!

Yeah, I was quite happy to have gotten it to basically run.

> You can probably also turn on Crossref -- the API key is optional and
> there's the possibility that they can throttle you, but I think it's
> probably pretty unlikely.

I uncommented the Crossref service in services.yml and uncommented
Crossref in institutions.yml as well. But it times out. It looks not
to be working right now.


>>  Because I wasn't using a direct connection to the SFX database, things
>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>  journal title search I get the error that the table
>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>  load_sfx_urls rake task which draws in data directly from the SFX
>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>  acts_as_ferret used by Ross to do things like categories?
>
> What SFX server are you using?  I didn't directly go off the DB either
> (since SFX was hosted consortially for Tech, this wasn't an option).
> Instead, I had a python script to import the tab delimited export
> files into mysql (doing this in python took about 1/5 the time of the
> equivalent ruby script) and then I'd reindex the ferret index after it
> completed.  At Tech, we didn't use the categories, since the
> librarians didn't like them.

For now I'll probably just skip A-Z functionality and forge ahead.



>>  Would it be helpful to include a step by step on creating a mysql
>>  development database?
>
> Do the migration scripts not work right?

The migrations seem to work just fine. I was talking more about the
more mundane details of creating the mysql database. For instance I
issued the following so that it wasn't using the root user. I thought
it might be good to include something fuller like this in the docs,
but before I did I wanted another set of eyes on it.

CREATE DATABASE umlaut_dev DEFAULT CHARACTER SET 'utf8';
CREATE USER umlaut;
SET PASSWORD FOR 'umlaut'@'localhost' = PASSWORD('umlaut');
GRANT ALL ON umlaut_dev.* TO 'umlaut'@'localhost';

>>  Now that I've got it up once, I'm going to try installation and
>>  configuration again on a fresh install of Ubuntu.
>
> While you're at it, it might be interesting to try it with postgres or
> firebird or something.  I mean, if you're up to it.

Sounds like a good plan. I'll give postgres a try.

Jason

From jronallo at gmail.com  Sat May 17 20:57:00 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Sat, 17 May 2008 20:57:00 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <4828AF61.3070408@jhu.edu>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<4828AF61.3070408@jhu.edu>
Message-ID: <763570460805171757m5f32548ds3c4082c8f8f8199c@mail.gmail.com>

On Mon, May 12, 2008 at 4:58 PM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
> I thought I _had_ step-by-step instructions for creating a mysql dev
> database!  The instructions in the docs can be used to create a dev db
> and a production db, no?    I guess we might want to make more prominent
> instructions that at present Umlaut A-Z and journal-name lookup is only
> tested with direct connection to SFX db.

I added a note to the wiki under set up your institutions and services.


> For catalogs, pretty much only the Horizon adapter is actually tested
> and working. Ross had a voyager adapter too, but I haven't kept it up,
> and I"m sure it doesn't work out of the box right now.  You are welcome
> to point your Umlaut at my Horizon OPAC instance.

I made a note about this on the wiki as well.

> It doesn't actually
> require any special authorization, just config the service like this:
>
> JH_HIP:
> name: JH Libraries Catalog
> display_name: JH Libraries Catalog
> type: Hip3Service
> priority: c # After SFX, to get SFX metadata enhancement
> base_path: http://catalog.library.jhu.edu/ipac20/ipac.jsp

I updated my local config to use your catalog. Should I change the
default config to include your Horizon?

> I'm also realizing that the view code I am using is included with the
> distro, but is not the default one. I left Ross's view code the default
> one, when GATech was still using it. But I'm not completely sure the
> default one works completely! It might make more sense to change mine to
> be the default one.  But if you want to configure your instance to use
> mine, but this in in your umlaut_config.environment.rb:
> config.app_config.resolve_view = "alternate/resolve_alternate"

Yes, I think we should put that on the todo list.

> PPS: Feel free to change the sample config/umlaut_distribution services,
> institutions, and environment files in whatever way you think is best
> and commit them.

I am working on changes to these files.  This leads me to the question
of which services are working. I've been looking at different service
adapters trying to get them set up so I can learn how the different
services work for writing my own. But sometimes I'm not certain which
are working. I'm sure this will all make more sense when I read the
technical documentation closer.

Can we go through the services and make a list of those known to be
working and those that are questionable or definitely broken? For
instance the Crossref service currently fails because the
service_types_generated method is missing. This method is missing from
some other services, but I'm not completely sure whether they might
still be working or not.

Once I know which ones aren't working I can add FIXME notes to the
source. Then I can create config files with just the services that are
known to work. As others are fixed then I can include sample configs
for those as well in services.yml. In institutions.yml I can uncomment
the ones that need no further configuration and comment the ones where
services.yml must be changed. Do you think that's a good way forward?

Here I'll start the list based on the filenames in lib/service_adapters:
amazon: working
citeulike:
connotea:
crossref: fixme
delicious:
ezproxy:
google_search:
hip3_service: working
hip_holding_search:
isbn_db:
opac:
opencontent_search:
pubmed:
sfx_backchannel_record:
sfx: working
ulrichs_link: working
voyager_native: fixme/unknown
voyager: fixme/unknown
worldcat: working
yahoo_my_web:
yahoo_search:


What may also be helpful is collecting those links which are known to
invoke and display particular services. It seems some background
services might show up and then are hidden by javascript if there is a
surer source of fulltext? I can see from the logs that YahooSearch
returns results, but they don't show on the page.  (Any way to have
these web links always show up if there are hits?) In lieu of a test
suite these kinds of links could help us troubleshoot different
services.

Let me know when rubyforge is set up to send commit emails. I tried to
admin the general list but don't have permissions.

Jason

From rochkind at jhu.edu  Mon May 19 10:31:24 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 19 May 2008 10:31:24 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>
Message-ID: <48318F3C.5010400@jhu.edu>

I have my SFX itself configured to do both CrossRef and Pubmed lookups.  
So I don't use the Umlaut CrossRef and Pubmed services---instead my 
Umlaut makes the request to SFX, SFX enhances metadata with crossref and 
pubmed, and returns this metadata to umlaut.  The current Umlaut SFX 
adaptor does take advantage of any added metadata from SFX to enhance 
the Umlaut metadata structures.

Jason, I would not recommend trying to use Postgres as an initial 
project.   I think there are currently some MySQL specific things in 
there. That could certainly be changed, and that would be an improvement 
to Umlaut, but I think there are bigger bang-for-the-buck projects that 
I'd recommend as a starting point.

[For those who don't know, Jason is 'interning' with me this summer 
working on Umlaut].

Jonathan

Jason Ronallo wrote:
> Hi, Ross,
> Response inline.
>
> On Mon, May 12, 2008 at 4:31 PM, Ross Singer <rossfsinger at gmail.com> wrote:
>   
>> Wow, I think is, like, our first ever email...  And it's (mostly) a
>> success story!
>>     
>
> Yeah, I was quite happy to have gotten it to basically run.
>
>   
>> You can probably also turn on Crossref -- the API key is optional and
>> there's the possibility that they can throttle you, but I think it's
>> probably pretty unlikely.
>>     
>
> I uncommented the Crossref service in services.yml and uncommented
> Crossref in institutions.yml as well. But it times out. It looks not
> to be working right now.
>
>
>   
>>>  Because I wasn't using a direct connection to the SFX database, things
>>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>>  journal title search I get the error that the table
>>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>>  load_sfx_urls rake task which draws in data directly from the SFX
>>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>>  acts_as_ferret used by Ross to do things like categories?
>>>       
>> What SFX server are you using?  I didn't directly go off the DB either
>> (since SFX was hosted consortially for Tech, this wasn't an option).
>> Instead, I had a python script to import the tab delimited export
>> files into mysql (doing this in python took about 1/5 the time of the
>> equivalent ruby script) and then I'd reindex the ferret index after it
>> completed.  At Tech, we didn't use the categories, since the
>> librarians didn't like them.
>>     
>
> For now I'll probably just skip A-Z functionality and forge ahead.
>
>
>
>   
>>>  Would it be helpful to include a step by step on creating a mysql
>>>  development database?
>>>       
>> Do the migration scripts not work right?
>>     
>
> The migrations seem to work just fine. I was talking more about the
> more mundane details of creating the mysql database. For instance I
> issued the following so that it wasn't using the root user. I thought
> it might be good to include something fuller like this in the docs,
> but before I did I wanted another set of eyes on it.
>
> CREATE DATABASE umlaut_dev DEFAULT CHARACTER SET 'utf8';
> CREATE USER umlaut;
> SET PASSWORD FOR 'umlaut'@'localhost' = PASSWORD('umlaut');
> GRANT ALL ON umlaut_dev.* TO 'umlaut'@'localhost';
>
>   
>>>  Now that I've got it up once, I'm going to try installation and
>>>  configuration again on a fresh install of Ubuntu.
>>>       
>> While you're at it, it might be interesting to try it with postgres or
>> firebird or something.  I mean, if you're up to it.
>>     
>
> Sounds like a good plan. I'll give postgres a try.
>
> Jason
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>   

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu


From rochkind at jhu.edu  Mon May 19 10:36:21 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 19 May 2008 10:36:21 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <763570460805171757m5f32548ds3c4082c8f8f8199c@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<4828AF61.3070408@jhu.edu>
	<763570460805171757m5f32548ds3c4082c8f8f8199c@mail.gmail.com>
Message-ID: <48319065.2030602@jhu.edu>

Yeah, the service_types_generated method was an added API requirement 
that services I don't use probably don't have yet. It's easy enough to 
add that method--but the services may or may not be working otherwise.  
Here's a note of the ones I know work, vs the ones I don't.  Which is 
really the ones I use here at JHU vs the ones leftover from Ross that I 
don't.

> amazon: working
> citeulike:  don't use
> connotea: dont' use 
> crossref: fixme (don't use)
> delicious: don't use
> ezproxy: working
> google_search: don't use
> hip3_service: working 
> hip_holding_search: working
> isbn_db: working
> opac: don't use
> opencontent_search: working sort of, but misconceived, needs to be replaced. 
> pubmed: don't use. 
> sfx_backchannel_record: working. 
> sfx: working
> ulrichs_link: working
> voyager_native: fixme/unknown
> voyager: fixme/unknown
> worldcat: working
> yahoo_my_web: don't use
> yahoo_search: dont' use
>
>
>   


> What may also be helpful is collecting those links which are known to
> invoke and display particular services. 

Yes, that's a good idea.


> It seems some background
> services might show up and then are hidden by javascript if there is a
> surer source of fulltext? I can see from the logs that YahooSearch
> returns results, but they don't show on the page. 

I'm not sure what's going on there, I simply dont' use YahooSearch. It 
may be that the 'view' you are using just ignores the data returned by 
YahooSearch, that's my guess. Are you using the Umlaut 'default' view, 
or my 'alt' view I use with JHU?  I have no idea what the default view 
does; like I suggested before, now that GA Tech isn't using Umlaut 
anyway, we should probably make my 'alt' (actually maintained) view the 
default standard.


> Let me know when rubyforge is set up to send commit emails. I tried to
> admin the general list but don't have permissions.
>   

I have no idea how to do this. You have any idea?  I can give you full 
permissions on our rubyforge project if that would help.

Jonathan
> Jason
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>   

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu


From rossfsinger at gmail.com  Mon May 19 10:56:26 2008
From: rossfsinger at gmail.com (Ross Singer)
Date: Mon, 19 May 2008 10:56:26 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <48318F3C.5010400@jhu.edu>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>
	<48318F3C.5010400@jhu.edu>
Message-ID: <23b83f160805190756r7fb4980ah51d838a9a4be3dd6@mail.gmail.com>

What is MySQL specific?  It seems worthwhile (in my mind, at least) to
remove any dependencies to a particular stack since that could
potentially be prohibitive to broader adoption.

I know the migrations have (or at least had, I can't remember if I
checked in the changes) some MySQL-isms in them, but I successfully
got the Umlaut running with Postgres last year with pretty minor
changes to the migrations.

Another useful endeavor in my mind would be getting Umlaut to run in JRuby.

My fear is that a lack of stack independence points to some bigger
flaws under the hood that will expose themselves later.

Maybe it would be worthwhile for us to come up with a priority list
and justification as to what we hope the outcomes of the priorities
are?

-Ross.

On Mon, May 19, 2008 at 10:31 AM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
> I have my SFX itself configured to do both CrossRef and Pubmed lookups.  So
> I don't use the Umlaut CrossRef and Pubmed services---instead my Umlaut
> makes the request to SFX, SFX enhances metadata with crossref and pubmed,
> and returns this metadata to umlaut.  The current Umlaut SFX adaptor does
> take advantage of any added metadata from SFX to enhance the Umlaut metadata
> structures.
>
> Jason, I would not recommend trying to use Postgres as an initial project.
> I think there are currently some MySQL specific things in there. That could
> certainly be changed, and that would be an improvement to Umlaut, but I
> think there are bigger bang-for-the-buck projects that I'd recommend as a
> starting point.
>
> [For those who don't know, Jason is 'interning' with me this summer working
> on Umlaut].
>
> Jonathan
>
> Jason Ronallo wrote:
>>
>> Hi, Ross,
>> Response inline.
>>
>> On Mon, May 12, 2008 at 4:31 PM, Ross Singer <rossfsinger at gmail.com>
>> wrote:
>>
>>>
>>> Wow, I think is, like, our first ever email...  And it's (mostly) a
>>> success story!
>>>
>>
>> Yeah, I was quite happy to have gotten it to basically run.
>>
>>
>>>
>>> You can probably also turn on Crossref -- the API key is optional and
>>> there's the possibility that they can throttle you, but I think it's
>>> probably pretty unlikely.
>>>
>>
>> I uncommented the Crossref service in services.yml and uncommented
>> Crossref in institutions.yml as well. But it times out. It looks not
>> to be working right now.
>>
>>
>>
>>>>
>>>>  Because I wasn't using a direct connection to the SFX database, things
>>>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>>>  journal title search I get the error that the table
>>>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>>>  load_sfx_urls rake task which draws in data directly from the SFX
>>>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>>>  acts_as_ferret used by Ross to do things like categories?
>>>>
>>>
>>> What SFX server are you using?  I didn't directly go off the DB either
>>> (since SFX was hosted consortially for Tech, this wasn't an option).
>>> Instead, I had a python script to import the tab delimited export
>>> files into mysql (doing this in python took about 1/5 the time of the
>>> equivalent ruby script) and then I'd reindex the ferret index after it
>>> completed.  At Tech, we didn't use the categories, since the
>>> librarians didn't like them.
>>>
>>
>> For now I'll probably just skip A-Z functionality and forge ahead.
>>
>>
>>
>>
>>>>
>>>>  Would it be helpful to include a step by step on creating a mysql
>>>>  development database?
>>>>
>>>
>>> Do the migration scripts not work right?
>>>
>>
>> The migrations seem to work just fine. I was talking more about the
>> more mundane details of creating the mysql database. For instance I
>> issued the following so that it wasn't using the root user. I thought
>> it might be good to include something fuller like this in the docs,
>> but before I did I wanted another set of eyes on it.
>>
>> CREATE DATABASE umlaut_dev DEFAULT CHARACTER SET 'utf8';
>> CREATE USER umlaut;
>> SET PASSWORD FOR 'umlaut'@'localhost' = PASSWORD('umlaut');
>> GRANT ALL ON umlaut_dev.* TO 'umlaut'@'localhost';
>>
>>
>>>>
>>>>  Now that I've got it up once, I'm going to try installation and
>>>>  configuration again on a fresh install of Ubuntu.
>>>>
>>>
>>> While you're at it, it might be interesting to try it with postgres or
>>> firebird or something.  I mean, if you're up to it.
>>>
>>
>> Sounds like a good plan. I'll give postgres a try.
>>
>> Jason
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>
>
> --
> Jonathan Rochkind
> Digital Services Software Engineer
> The Sheridan Libraries
> Johns Hopkins University
> 410.516.8886 rochkind (at) jhu.edu
>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From jronallo at gmail.com  Mon May 19 11:02:10 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Mon, 19 May 2008 11:02:10 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <48319065.2030602@jhu.edu>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<4828AF61.3070408@jhu.edu>
	<763570460805171757m5f32548ds3c4082c8f8f8199c@mail.gmail.com>
	<48319065.2030602@jhu.edu>
Message-ID: <763570460805190802h5dbd4e15t5e8776ed1d6cad4e@mail.gmail.com>

Thanks, Jonathan,
At least for now I'll update all the adapters you're not using with a
FIXME note. Could save someone lots of time wondering why an adapter
doesn't work.

If you gave me proper privileges on the project, I could set up a
commit email list. I think I could create an svn post-commit hook to
email that list.

Jason

From rochkind at jhu.edu  Mon May 19 11:08:49 2008
From: rochkind at jhu.edu (Jonathan Rochkind)
Date: Mon, 19 May 2008 11:08:49 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <23b83f160805190756r7fb4980ah51d838a9a4be3dd6@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>
	<48318F3C.5010400@jhu.edu>
	<23b83f160805190756r7fb4980ah51d838a9a4be3dd6@mail.gmail.com>
Message-ID: <48319801.3050507@jhu.edu>

I can't remember what may be MySQL specific, I just worry some things 
might be. Other than the migrations, I think probably just my "nightly 
maintenance" tasks that clean up the database.  I think MySQL 
dependencies elsewhere should be minimal, or not exist at all.

We can certainly come up with a shared priority list for Umlaut, but 
since you and I are using it in _entirely_ different ways, I think our 
priorities are probably entirely different. My biggest priorities right 
now are adding some fairly "low hanging fruit" services that will have 
an immediate impact on my users, but will probably be entirely 
irrelevant to Ross's needs. The first examples are OCA and Google Books.

An unprioritized random jumble list of all the things I'd like done to 
Umlaut can be found here:
http://wiki.code4lib.org/index.php/Umlaut_wishlist

Jonathan

Ross Singer wrote:
> What is MySQL specific?  It seems worthwhile (in my mind, at least) to
> remove any dependencies to a particular stack since that could
> potentially be prohibitive to broader adoption.
>
> I know the migrations have (or at least had, I can't remember if I
> checked in the changes) some MySQL-isms in them, but I successfully
> got the Umlaut running with Postgres last year with pretty minor
> changes to the migrations.
>
> Another useful endeavor in my mind would be getting Umlaut to run in JRuby.
>
> My fear is that a lack of stack independence points to some bigger
> flaws under the hood that will expose themselves later.
>
> Maybe it would be worthwhile for us to come up with a priority list
> and justification as to what we hope the outcomes of the priorities
> are?
>
> -Ross.
>
> On Mon, May 19, 2008 at 10:31 AM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
>   
>> I have my SFX itself configured to do both CrossRef and Pubmed lookups.  So
>> I don't use the Umlaut CrossRef and Pubmed services---instead my Umlaut
>> makes the request to SFX, SFX enhances metadata with crossref and pubmed,
>> and returns this metadata to umlaut.  The current Umlaut SFX adaptor does
>> take advantage of any added metadata from SFX to enhance the Umlaut metadata
>> structures.
>>
>> Jason, I would not recommend trying to use Postgres as an initial project.
>> I think there are currently some MySQL specific things in there. That could
>> certainly be changed, and that would be an improvement to Umlaut, but I
>> think there are bigger bang-for-the-buck projects that I'd recommend as a
>> starting point.
>>
>> [For those who don't know, Jason is 'interning' with me this summer working
>> on Umlaut].
>>
>> Jonathan
>>
>> Jason Ronallo wrote:
>>     
>>> Hi, Ross,
>>> Response inline.
>>>
>>> On Mon, May 12, 2008 at 4:31 PM, Ross Singer <rossfsinger at gmail.com>
>>> wrote:
>>>
>>>       
>>>> Wow, I think is, like, our first ever email...  And it's (mostly) a
>>>> success story!
>>>>
>>>>         
>>> Yeah, I was quite happy to have gotten it to basically run.
>>>
>>>
>>>       
>>>> You can probably also turn on Crossref -- the API key is optional and
>>>> there's the possibility that they can throttle you, but I think it's
>>>> probably pretty unlikely.
>>>>
>>>>         
>>> I uncommented the Crossref service in services.yml and uncommented
>>> Crossref in institutions.yml as well. But it times out. It looks not
>>> to be working right now.
>>>
>>>
>>>
>>>       
>>>>>  Because I wasn't using a direct connection to the SFX database, things
>>>>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>>>>  journal title search I get the error that the table
>>>>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>>>>  load_sfx_urls rake task which draws in data directly from the SFX
>>>>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>>>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>>>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>>>>  acts_as_ferret used by Ross to do things like categories?
>>>>>
>>>>>           
>>>> What SFX server are you using?  I didn't directly go off the DB either
>>>> (since SFX was hosted consortially for Tech, this wasn't an option).
>>>> Instead, I had a python script to import the tab delimited export
>>>> files into mysql (doing this in python took about 1/5 the time of the
>>>> equivalent ruby script) and then I'd reindex the ferret index after it
>>>> completed.  At Tech, we didn't use the categories, since the
>>>> librarians didn't like them.
>>>>
>>>>         
>>> For now I'll probably just skip A-Z functionality and forge ahead.
>>>
>>>
>>>
>>>
>>>       
>>>>>  Would it be helpful to include a step by step on creating a mysql
>>>>>  development database?
>>>>>
>>>>>           
>>>> Do the migration scripts not work right?
>>>>
>>>>         
>>> The migrations seem to work just fine. I was talking more about the
>>> more mundane details of creating the mysql database. For instance I
>>> issued the following so that it wasn't using the root user. I thought
>>> it might be good to include something fuller like this in the docs,
>>> but before I did I wanted another set of eyes on it.
>>>
>>> CREATE DATABASE umlaut_dev DEFAULT CHARACTER SET 'utf8';
>>> CREATE USER umlaut;
>>> SET PASSWORD FOR 'umlaut'@'localhost' = PASSWORD('umlaut');
>>> GRANT ALL ON umlaut_dev.* TO 'umlaut'@'localhost';
>>>
>>>
>>>       
>>>>>  Now that I've got it up once, I'm going to try installation and
>>>>>  configuration again on a fresh install of Ubuntu.
>>>>>
>>>>>           
>>>> While you're at it, it might be interesting to try it with postgres or
>>>> firebird or something.  I mean, if you're up to it.
>>>>
>>>>         
>>> Sounds like a good plan. I'll give postgres a try.
>>>
>>> Jason
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>
>>>       
>> --
>> Jonathan Rochkind
>> Digital Services Software Engineer
>> The Sheridan Libraries
>> Johns Hopkins University
>> 410.516.8886 rochkind (at) jhu.edu
>>
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>
>>     

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu


From jronallo at gmail.com  Mon May 19 11:21:06 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Mon, 19 May 2008 11:21:06 -0400
Subject: [Umlaut-general] 1st resolution
In-Reply-To: <23b83f160805190756r7fb4980ah51d838a9a4be3dd6@mail.gmail.com>
References: <763570460805101302t5f22ff9dye303aff5306594c7@mail.gmail.com>
	<23b83f160805121331g742429d4q21c06f4979ba0f48@mail.gmail.com>
	<763570460805171756g2f97e63bq223b934d9155a949@mail.gmail.com>
	<48318F3C.5010400@jhu.edu>
	<23b83f160805190756r7fb4980ah51d838a9a4be3dd6@mail.gmail.com>
Message-ID: <763570460805190821p69805fa2oc87d29ac30f11211@mail.gmail.com>

Where I'd like to start is with writing new service adapters and try
to improve the documentation on how to write service adapters. I think
that'd be a good introduction to the Umlaut and doable in the amount
of time I have. It would also highlight how the Umlaut can be extended
by others to better meet their needs.

>From there I'm open to other projects including removing any
dependencies on mysql. Eventually (outside this internship probably)
I'd like to create a SerialsSolutions 360 Link adapter so it is clear
that the Umlaut can be decoupled from any particular knowledgebase.

I've also thought about JRuby deployment, but wouldn't want to tackle
that one until further down the line.

Jason

On Mon, May 19, 2008 at 10:56 AM, Ross Singer <rossfsinger at gmail.com> wrote:
> What is MySQL specific?  It seems worthwhile (in my mind, at least) to
> remove any dependencies to a particular stack since that could
> potentially be prohibitive to broader adoption.
>
> I know the migrations have (or at least had, I can't remember if I
> checked in the changes) some MySQL-isms in them, but I successfully
> got the Umlaut running with Postgres last year with pretty minor
> changes to the migrations.
>
> Another useful endeavor in my mind would be getting Umlaut to run in JRuby.
>
> My fear is that a lack of stack independence points to some bigger
> flaws under the hood that will expose themselves later.
>
> Maybe it would be worthwhile for us to come up with a priority list
> and justification as to what we hope the outcomes of the priorities
> are?
>
> -Ross.
>
> On Mon, May 19, 2008 at 10:31 AM, Jonathan Rochkind <rochkind at jhu.edu> wrote:
>> I have my SFX itself configured to do both CrossRef and Pubmed lookups.  So
>> I don't use the Umlaut CrossRef and Pubmed services---instead my Umlaut
>> makes the request to SFX, SFX enhances metadata with crossref and pubmed,
>> and returns this metadata to umlaut.  The current Umlaut SFX adaptor does
>> take advantage of any added metadata from SFX to enhance the Umlaut metadata
>> structures.
>>
>> Jason, I would not recommend trying to use Postgres as an initial project.
>> I think there are currently some MySQL specific things in there. That could
>> certainly be changed, and that would be an improvement to Umlaut, but I
>> think there are bigger bang-for-the-buck projects that I'd recommend as a
>> starting point.
>>
>> [For those who don't know, Jason is 'interning' with me this summer working
>> on Umlaut].
>>
>> Jonathan
>>
>> Jason Ronallo wrote:
>>>
>>> Hi, Ross,
>>> Response inline.
>>>
>>> On Mon, May 12, 2008 at 4:31 PM, Ross Singer <rossfsinger at gmail.com>
>>> wrote:
>>>
>>>>
>>>> Wow, I think is, like, our first ever email...  And it's (mostly) a
>>>> success story!
>>>>
>>>
>>> Yeah, I was quite happy to have gotten it to basically run.
>>>
>>>
>>>>
>>>> You can probably also turn on Crossref -- the API key is optional and
>>>> there's the possibility that they can throttle you, but I think it's
>>>> probably pretty unlikely.
>>>>
>>>
>>> I uncommented the Crossref service in services.yml and uncommented
>>> Crossref in institutions.yml as well. But it times out. It looks not
>>> to be working right now.
>>>
>>>
>>>
>>>>>
>>>>>  Because I wasn't using a direct connection to the SFX database, things
>>>>>  like title search (with auto suggestions?) and A-Z list failed. For a
>>>>>  journal title search I get the error that the table
>>>>>  umlaut_dev.AZ_TITLE doesn't exist. Do you create that with your
>>>>>  load_sfx_urls rake task which draws in data directly from the SFX
>>>>>  database? If I set config.app_config.use_umlaut_journal_index to true,
>>>>>  then it fails because acts_as_ferret isn't loaded. I suppose the
>>>>>  needed info hasn't been pre-fetched from the SFX server database. Was
>>>>>  acts_as_ferret used by Ross to do things like categories?
>>>>>
>>>>
>>>> What SFX server are you using?  I didn't directly go off the DB either
>>>> (since SFX was hosted consortially for Tech, this wasn't an option).
>>>> Instead, I had a python script to import the tab delimited export
>>>> files into mysql (doing this in python took about 1/5 the time of the
>>>> equivalent ruby script) and then I'd reindex the ferret index after it
>>>> completed.  At Tech, we didn't use the categories, since the
>>>> librarians didn't like them.
>>>>
>>>
>>> For now I'll probably just skip A-Z functionality and forge ahead.
>>>
>>>
>>>
>>>
>>>>>
>>>>>  Would it be helpful to include a step by step on creating a mysql
>>>>>  development database?
>>>>>
>>>>
>>>> Do the migration scripts not work right?
>>>>
>>>
>>> The migrations seem to work just fine. I was talking more about the
>>> more mundane details of creating the mysql database. For instance I
>>> issued the following so that it wasn't using the root user. I thought
>>> it might be good to include something fuller like this in the docs,
>>> but before I did I wanted another set of eyes on it.
>>>
>>> CREATE DATABASE umlaut_dev DEFAULT CHARACTER SET 'utf8';
>>> CREATE USER umlaut;
>>> SET PASSWORD FOR 'umlaut'@'localhost' = PASSWORD('umlaut');
>>> GRANT ALL ON umlaut_dev.* TO 'umlaut'@'localhost';
>>>
>>>
>>>>>
>>>>>  Now that I've got it up once, I'm going to try installation and
>>>>>  configuration again on a fresh install of Ubuntu.
>>>>>
>>>>
>>>> While you're at it, it might be interesting to try it with postgres or
>>>> firebird or something.  I mean, if you're up to it.
>>>>
>>>
>>> Sounds like a good plan. I'll give postgres a try.
>>>
>>> Jason
>>> _______________________________________________
>>> Umlaut-general mailing list
>>> Umlaut-general at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>>
>>
>> --
>> Jonathan Rochkind
>> Digital Services Software Engineer
>> The Sheridan Libraries
>> Johns Hopkins University
>> 410.516.8886 rochkind (at) jhu.edu
>>
>> _______________________________________________
>> Umlaut-general mailing list
>> Umlaut-general at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/umlaut-general
>>
> _______________________________________________
> Umlaut-general mailing list
> Umlaut-general at rubyforge.org
> http://rubyforge.org/mailman/listinfo/umlaut-general
>

From jronallo at gmail.com  Thu May 22 08:08:17 2008
From: jronallo at gmail.com (Jason Ronallo)
Date: Thu, 22 May 2008 08:08:17 -0400
Subject: [Umlaut-general] Umlaut commits email list
Message-ID: <763570460805220508u2d03a553we519442eb27c5836@mail.gmail.com>

Hi,
We now have a commits email list which you can subscribe to here:
http://rubyforge.org/mailman/listinfo/umlaut-commits

Jason