From std5 at nyu.edu Thu Nov 12 14:00:28 2009 From: std5 at nyu.edu (Scot Dalton) Date: Thu, 12 Nov 2009 14:00:28 -0500 Subject: [Umlaut-general] DissertationCatch Error Message-ID: Hi, We're experiencing an error in the DissertationCatch ReferentFilter. We're getting the following SQL error message when calling 'referent.remove_value("issn")': "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column 'referent_id' cannot be null: UPDATE `referent_values` SET referent_id = NULL WHERE (referent_id = 234925 AND id IN (6543983)) )" It doesn't appear that either JHU or New School are having a similar problem. We've been experiencing some issues with our MySQL instance, which we hope to resolve shortly. That may be causing our problem, but if anyone has any ideas, I'd appreciate it. Not sure if the MySQL instance is even the problem. No problem at JHU and New School: http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 Problem at NYU: http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 Thanks, Scot Dalton Phone: (212) 998-2674 Web Development Division of Libraries New York University From rochkind at jhu.edu Thu Nov 12 15:39:57 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 12 Nov 2009 15:39:57 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: References: Message-ID: <4AFC729D.2070404@jhu.edu> And wow, somehow I missed the fact that New School was even up with Umlaut! Wow! Is someone from New School on this list? Welcome! Jonathan Scot Dalton wrote: > Hi, > > We're experiencing an error in the DissertationCatch ReferentFilter. > We're getting the following SQL error message when calling > 'referent.remove_value("issn")': > "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column > 'referent_id' cannot be null: UPDATE `referent_values` SET referent_id > = NULL WHERE (referent_id = 234925 AND id IN (6543983)) )" > > It doesn't appear that either JHU or New School are having a similar > problem. We've been experiencing some issues with our MySQL instance, > which we hope to resolve shortly. That may be causing our problem, > but if anyone has any ideas, I'd appreciate it. Not sure if the MySQL > instance is even the problem. > > No problem at JHU and New School: > http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 > http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 > > Problem at NYU: > http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 > > > Thanks, > Scot Dalton > Phone: (212) 998-2674 > Web Development > Division of Libraries > New York University > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Thu Nov 12 15:06:27 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 12 Nov 2009 15:06:27 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: References: Message-ID: <4AFC6AC3.8040507@jhu.edu> Hmm, that's annoying! For contextual background, what the DissertationCatch referent filter does is take incoming OpenURLs that reference Dissertation Abstracts, and change them to be of genre=dissertation referencing the actual dissertation, not the abstract of the dissertation in Dissertation Abstracts! Since that leads to the actual desired behavior. So if for some reason you wanted to disable the DissertationCatch, that's all you'd lose. (Incidentally, I recently realized that I could have done that with a plain old Umlaut service, there was no need for the extra ReferentFilter architecture. At some point in the future, I'll probably get rid of ReferentFilters and just use ordinary services; less code, less bugs.) But as far as why you're actually getting that bug.... I'm not sure. Can you actually reproduce it on demand? If so, I'd try to log into your servers to help you debug it. But that might be more trouble than it's worse, since to do that I'd need a) shell access to your server, and b) ideally some high port (like 3000) opened up to me, so I could launch a copy of Umlaut on it with ruby-debug, and then access it with my browser while interactively debugging. As a step before this, can you send the stack trace that goes with this error? Maybe looking at the source that'll give me some clues. I grepped through my own production logs to see if I had that error in them ever, but it doesn't seem to be there. Jonathan Scot Dalton wrote: > Hi, > > We're experiencing an error in the DissertationCatch ReferentFilter. > We're getting the following SQL error message when calling > 'referent.remove_value("issn")': > "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column > 'referent_id' cannot be null: UPDATE `referent_values` SET referent_id > = NULL WHERE (referent_id = 234925 AND id IN (6543983)) )" > > It doesn't appear that either JHU or New School are having a similar > problem. We've been experiencing some issues with our MySQL instance, > which we hope to resolve shortly. That may be causing our problem, > but if anyone has any ideas, I'd appreciate it. Not sure if the MySQL > instance is even the problem. > > No problem at JHU and New School: > http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 > http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 > > Problem at NYU: > http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 > > > Thanks, > Scot Dalton > Phone: (212) 998-2674 > Web Development > Division of Libraries > New York University > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Thu Nov 12 15:21:33 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 12 Nov 2009 15:21:33 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: References: Message-ID: <4AFC6E4D.9030703@jhu.edu> Well, okay, here's some clues. remove_value(issn) does this: def remove_value(key) referent_values.find(:all, :conditions=> ['key_name =?', key]).each do |rv| referent_values.delete(rv) end end So basically, it looks through the to-many referent_values relationship for a ReferentValue whose key_name is "issn", and when it finds one, it uses the standard Rails delete() to delete it. I was initially surprised that was ending up creating SQL ""UPDATE `referent_values` SET referent_id = NULL ...." at all, I thought I was actually deleting the ReferentValue from the db entirely. But then I looked up ActiveRecord documentation: collection.delete(object, ?) Removes one or more objects from the collection by setting their foreign keys to NULL. Objects will be in addition destroyed if they?re associated with :dependent => :destroy, and deleted if they?re associated with :dependent => :delete_all. So that explains why Rails is setting the foreign key to null! Now, why that raises in your Umlaut but not mine, I dunno. When I wrote the code, I thought that I was actually deleting the ReferentValue from the db, not just removing it from the association! But I'm apparently not, it's leaving all those orphaned ReferentValues in the db! Which doesn't sound so good! Except when I look in my actual db, I _don't_ have any ReferentValues with nil referent_ids. So, now I'm confused. I'm not sure how to fix this, I'm not sure what the 'right' way to both delete a ReferentValue in the db _and_ make sure it's removed from the in-memory association is, in Rails? It looks like I'd make sure it was really removed from the db if I continued to delete(), but just changed the referent_values association decleration to be: has_many :referent_values, :dependent => :destroy [ or :dependent => :delete_all] But I think that would still FIRST null out the referent_id, and THEN delete it. I wonder why that triggers MySQL constraint violations in your MySQL and not mine? The referent_values/referent_id column in my db is defined like this: | Field | Type | Null | Key | Default | Extra | | referent_id | int(11) | NO | MUL | 0 | | [ And yes, I just checked to see if I had a bunch of ReferentValues with 0 for referent_id too. I do not. I do not seem to be getting orphaned ReferentValues. ] So hopefully this is some clues. If you figure out what's up and fix it, feel free to commit the changes. Jonathan Scot Dalton wrote: > Hi, > > We're experiencing an error in the DissertationCatch ReferentFilter. > We're getting the following SQL error message when calling > 'referent.remove_value("issn")': > "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column > 'referent_id' cannot be null: UPDATE `referent_values` SET referent_id > = NULL WHERE (referent_id = 234925 AND id IN (6543983)) )" > > It doesn't appear that either JHU or New School are having a similar > problem. We've been experiencing some issues with our MySQL instance, > which we hope to resolve shortly. That may be causing our problem, > but if anyone has any ideas, I'd appreciate it. Not sure if the MySQL > instance is even the problem. > > No problem at JHU and New School: > http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 > http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 > > Problem at NYU: > http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 > > > Thanks, > Scot Dalton > Phone: (212) 998-2674 > Web Development > Division of Libraries > New York University > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From std5 at nyu.edu Thu Nov 12 16:24:58 2009 From: std5 at nyu.edu (Scot Dalton) Date: Thu, 12 Nov 2009 16:24:58 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: <4AFC6E4D.9030703@jhu.edu> References: <4AFC6E4D.9030703@jhu.edu> Message-ID: <7A8FFC74-40D8-42C2-8365-1A543D20854A@nyu.edu> Here's the stack trace from production.log: [12 Nov 16:23:58] (pid:23562) INFO: Processing ResolveController#index (for 128.122.91.179 at 2009-11-12 16:23:58) [GET] [12 Nov 16:23:58] (pid:23562) INFO: Session ID: 257d56120f3a8233dd9db71635cd47b2 [12 Nov 16:23:58] (pid:23562) INFO: Parameters: {"rft.issn"=>"0419-4217", "action"=>"index", "controller"=>"resolve"} [12 Nov 16:23:58] (pid:23562) INFO: HTTP Referer: [none] [12 Nov 16:23:58] (pid:23562) INFO: User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_1; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9 [12 Nov 16:23:58] (pid:23562) FATAL: ActiveRecord::StatementInvalid (Mysql::Error: #23000Column 'referent_id' cannot be null: UPDATE `referent_values` SET referent_id = NULL WHERE (referent_id = 275 AND id IN (6099)) ): /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract_adapter.rb:147:in `log' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/mysql_adapter.rb:302:in `execute' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/database_statements.rb: 167:in `update_sql' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/mysql_adapter.rb:317:in `update_sql' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/database_statements.rb: 49:in `update_without_query_dirty' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/query_cache.rb:19:in `update' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/base.rb:752:in `update_all' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/associations/has_many_association.rb:73:in `delete_records' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/associations/association_collection.rb:138:in `delete' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/database_statements.rb: 66:in `transaction' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/transactions.rb:79:in `transaction' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/transactions.rb:98:in `transaction' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/associations/association_collection.rb:134:in `delete' /app/models/referent.rb:435:in `remove_value' /app/models/referent.rb:434:in `each' /app/models/referent.rb:434:in `remove_value' /lib/referent_filters/dissertation_catch.rb:33:in `filter' /app/models/referent.rb:119:in `create_by_context_object' /app/models/referent.rb:117:in `each' /app/models/referent.rb:117:in `create_by_context_object' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/database_statements.rb: 66:in `transaction' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/transactions.rb:79:in `transaction' /app/models/referent.rb:95:in `create_by_context_object' /app/models/referent.rb:79:in `find_or_create_by_context_object' /app/models/request.rb:411:in `create_new_request!' /app/models/request.rb:70:in `new_request' /app/controllers/resolve_controller.rb:29:in `init_processing' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activesupport-2.1.2/ lib/active_support/callbacks.rb:173:in `send' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activesupport-2.1.2/ lib/active_support/callbacks.rb:173:in `evaluate_method' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activesupport-2.1.2/ lib/active_support/callbacks.rb:161:in `call' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/filters.rb:191:in `call' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/filters.rb:591:in `run_before_filters' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/filters.rb:577:in `call_filters' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/filters.rb:572:in `perform_action_without_benchmark' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/benchmarking.rb:68:in `perform_action_without_rescue' /apps/ruby-1.8.6-p287/lib/ruby/1.8/benchmark.rb:293:in `measure' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/benchmarking.rb:68:in `perform_action_without_rescue' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/rescue.rb:201:in `perform_action_without_caching' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/caching/sql_cache.rb:13:in `perform_action' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/activerecord-2.1.2/ lib/active_record/query_cache.rb:8:in `cache' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/caching/sql_cache.rb:12:in `perform_action' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/base.rb:529:in `send' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/base.rb:529:in `process_without_filters' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/filters.rb:568:in `process_without_session_management_support' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/session_management.rb:130:in `process' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/base.rb:389:in `process' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:149:in `handle_request' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:107:in `dispatch' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:104:in `synchronize' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:104:in `dispatch' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:120:in `dispatch_cgi' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/actionpack-2.1.2/lib/ action_controller/dispatcher.rb:35:in `dispatch' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/rails.rb:76:in `process' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/rails.rb:74:in `synchronize' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/rails.rb:74:in `process' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:159:in `process_client' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:158:in `each' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:158:in `process_client' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:285:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:285:in `initialize' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:285:in `new' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:285:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:268:in `initialize' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:268:in `new' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel.rb:268:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/configurator.rb:282:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/configurator.rb:281:in `each' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/configurator.rb:281:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/ mongrel_rails:128:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../ lib/mongrel/command.rb:212:in `run' /apps/ruby-1.8.6-p287/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/ mongrel_rails:281 /apps/ruby/bin/mongrel_rails:19:in `load' /apps/ruby/bin/mongrel_rails:19 [12 Nov 16:23:58] (pid:23562) INFO: Rendering template within layouts/ local/nyu_resolve [12 Nov 16:23:58] (pid:23562) INFO: Rendering error/resolve_error (500) Thanks, Scot Dalton Phone: (212) 998-2674 Web Development Division of Libraries New York University On Nov 12, 2009, at Nov 12, 3:21 PM, Jonathan Rochkind wrote: > Well, okay, here's some clues. remove_value(issn) does this: > > def remove_value(key) > referent_values.find(:all, :conditions=> ['key_name =?', key]).each > do |rv| > referent_values.delete(rv) > end > end > > > So basically, it looks through the to-many referent_values > relationship for a ReferentValue whose key_name is "issn", and when > it finds one, it uses the standard Rails delete() to delete it. > > I was initially surprised that was ending up creating SQL ""UPDATE > `referent_values` SET referent_id = NULL ...." at all, I thought I > was actually deleting the ReferentValue from the db entirely. But > then I looked up ActiveRecord documentation: > > collection.delete(object, ?) > Removes one or more objects from the collection by setting their > foreign keys to NULL. Objects will be in addition destroyed if > they?re associated with :dependent => :destroy, and deleted if > they?re associated with :dependent => :delete_all. > So that explains why Rails is setting the foreign key to null! > > Now, why that raises in your Umlaut but not mine, I dunno. > > When I wrote the code, I thought that I was actually deleting the > ReferentValue from the db, not just removing it from the > association! But I'm apparently not, it's leaving all those orphaned > ReferentValues in the db! Which doesn't sound so good! Except when I > look in my actual db, I _don't_ have any ReferentValues with nil > referent_ids. So, now I'm confused. > > I'm not sure how to fix this, I'm not sure what the 'right' way to > both delete a ReferentValue in the db _and_ make sure it's removed > from the in-memory association is, in Rails? It looks like I'd make > sure it was really removed from the db if I continued to delete(), > but just changed the referent_values association decleration to be: > > has_many :referent_values, :dependent => :destroy [ or :dependent > => :delete_all] > > But I think that would still FIRST null out the referent_id, and > THEN delete it. I wonder why that triggers MySQL constraint > violations in your MySQL and not mine? The referent_values/ > referent_id column in my db is defined like this: > > | Field | Type | Null | Key | Default | Extra | > | referent_id | int(11) | NO | MUL | 0 | | > > [ And yes, I just checked to see if I had a bunch of ReferentValues > with 0 for referent_id too. I do not. I do not seem to be getting > orphaned ReferentValues. ] > > So hopefully this is some clues. If you figure out what's up and fix > it, feel free to commit the changes. > > Jonathan > > > Scot Dalton wrote: >> Hi, >> >> We're experiencing an error in the DissertationCatch >> ReferentFilter. We're getting the following SQL error message >> when calling 'referent.remove_value("issn")': >> "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column >> 'referent_id' cannot be null: UPDATE `referent_values` SET >> referent_id = NULL WHERE (referent_id = 234925 AND id IN >> (6543983)) )" >> >> It doesn't appear that either JHU or New School are having a >> similar problem. We've been experiencing some issues with our >> MySQL instance, which we hope to resolve shortly. That may be >> causing our problem, but if anyone has any ideas, I'd appreciate >> it. Not sure if the MySQL instance is even the problem. >> >> No problem at JHU and New School: >> http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 >> http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 >> >> Problem at NYU: >> http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 >> >> >> Thanks, >> Scot Dalton >> Phone: (212) 998-2674 >> Web Development >> Division of Libraries >> New York University >> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general From std5 at nyu.edu Thu Nov 12 16:32:04 2009 From: std5 at nyu.edu (Scot Dalton) Date: Thu, 12 Nov 2009 16:32:04 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: <4AFC6E4D.9030703@jhu.edu> References: <4AFC6E4D.9030703@jhu.edu> Message-ID: <69056D57-ADD8-495D-B285-B0F0CDF2BDE1@nyu.edu> Thanks for this. I'll continue to look through it and see what I can find. In the interim, I may just use the ensure_value! method and set the rv to an empty string. That seems to work in my tests. As for why I see the problem and you guys don't, could it be a version thing? (Obviously grasping at straws here.) We're running: ruby 1.8.6 (2008-08-11 patchlevel 287) [sparc-solaris2.10] gem 1.3.3 actionmailer (2.3.2, 2.1.2) actionpack (2.3.2, 2.1.2) activerecord (2.3.2, 2.1.2) activeresource (2.3.2, 2.1.2) activesupport (2.3.2, 2.1.2) cgi_multipart_eof_fix (2.5.0) daemons (1.0.10) fastthread (1.0.7) gem_plugin (0.2.3) hpricot (0.8.1, 0.6) json (1.1.6, 1.1.3) marc (0.2.2) mongrel (1.1.5) mongrel_cluster (1.0.5) mysql (2.7) rails (2.3.2, 2.1.2) rake (0.8.7) ruby-json (1.1.2) rubygems-update (1.3.3) Thanks again, Scot Dalton Phone: (212) 998-2674 Web Development Division of Libraries New York University On Nov 12, 2009, at Nov 12, 3:21 PM, Jonathan Rochkind wrote: > Well, okay, here's some clues. remove_value(issn) does this: > > def remove_value(key) > referent_values.find(:all, :conditions=> ['key_name =?', key]).each > do |rv| > referent_values.delete(rv) > end > end > > > So basically, it looks through the to-many referent_values > relationship for a ReferentValue whose key_name is "issn", and when > it finds one, it uses the standard Rails delete() to delete it. > > I was initially surprised that was ending up creating SQL ""UPDATE > `referent_values` SET referent_id = NULL ...." at all, I thought I > was actually deleting the ReferentValue from the db entirely. But > then I looked up ActiveRecord documentation: > > collection.delete(object, ?) > Removes one or more objects from the collection by setting their > foreign keys to NULL. Objects will be in addition destroyed if > they?re associated with :dependent => :destroy, and deleted if > they?re associated with :dependent => :delete_all. > So that explains why Rails is setting the foreign key to null! > > Now, why that raises in your Umlaut but not mine, I dunno. > > When I wrote the code, I thought that I was actually deleting the > ReferentValue from the db, not just removing it from the > association! But I'm apparently not, it's leaving all those orphaned > ReferentValues in the db! Which doesn't sound so good! Except when I > look in my actual db, I _don't_ have any ReferentValues with nil > referent_ids. So, now I'm confused. > > I'm not sure how to fix this, I'm not sure what the 'right' way to > both delete a ReferentValue in the db _and_ make sure it's removed > from the in-memory association is, in Rails? It looks like I'd make > sure it was really removed from the db if I continued to delete(), > but just changed the referent_values association decleration to be: > > has_many :referent_values, :dependent => :destroy [ or :dependent > => :delete_all] > > But I think that would still FIRST null out the referent_id, and > THEN delete it. I wonder why that triggers MySQL constraint > violations in your MySQL and not mine? The referent_values/ > referent_id column in my db is defined like this: > > | Field | Type | Null | Key | Default | Extra | > | referent_id | int(11) | NO | MUL | 0 | | > > [ And yes, I just checked to see if I had a bunch of ReferentValues > with 0 for referent_id too. I do not. I do not seem to be getting > orphaned ReferentValues. ] > > So hopefully this is some clues. If you figure out what's up and fix > it, feel free to commit the changes. > > Jonathan > > > Scot Dalton wrote: >> Hi, >> >> We're experiencing an error in the DissertationCatch >> ReferentFilter. We're getting the following SQL error message >> when calling 'referent.remove_value("issn")': >> "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column >> 'referent_id' cannot be null: UPDATE `referent_values` SET >> referent_id = NULL WHERE (referent_id = 234925 AND id IN >> (6543983)) )" >> >> It doesn't appear that either JHU or New School are having a >> similar problem. We've been experiencing some issues with our >> MySQL instance, which we hope to resolve shortly. That may be >> causing our problem, but if anyone has any ideas, I'd appreciate >> it. Not sure if the MySQL instance is even the problem. >> >> No problem at JHU and New School: >> http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 >> http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 >> >> Problem at NYU: >> http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 >> >> >> Thanks, >> Scot Dalton >> Phone: (212) 998-2674 >> Web Development >> Division of Libraries >> New York University >> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general From rochkind at jhu.edu Thu Nov 12 16:37:00 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Thu, 12 Nov 2009 16:37:00 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: <69056D57-ADD8-495D-B285-B0F0CDF2BDE1@nyu.edu> References: <4AFC6E4D.9030703@jhu.edu> <69056D57-ADD8-495D-B285-B0F0CDF2BDE1@nyu.edu> Message-ID: <4AFC7FFC.50508@jhu.edu> Also try adding this to Referent.rb: Replace: has_many :referent_values With: has_many :referent_values, :dependent => :delete_all. If that somehow miraculously fixes your problem, that's fine to commit, I think. (Test it a _bit_ first. :) ). Gem versions should all be more or less _fixed_ by Umlaut, it should _insist_ on using the same gem versions for all of us. (Including Rails 2.1.2, that's why you ended up with that one installed). Wonder if it could be different mysql versions. But we don't want to insist on a specific mysql version, we want to fix the code to work with any of em. Jonathan Scot Dalton wrote: > Thanks for this. I'll continue to look through it and see what I can > find. > > In the interim, I may just use the ensure_value! method and set the rv > to an empty string. That seems to work in my tests. > > As for why I see the problem and you guys don't, could it be a version > thing? (Obviously grasping at straws here.) > We're running: > ruby 1.8.6 (2008-08-11 patchlevel 287) [sparc-solaris2.10] > gem 1.3.3 > actionmailer (2.3.2, 2.1.2) > actionpack (2.3.2, 2.1.2) > activerecord (2.3.2, 2.1.2) > activeresource (2.3.2, 2.1.2) > activesupport (2.3.2, 2.1.2) > cgi_multipart_eof_fix (2.5.0) > daemons (1.0.10) > fastthread (1.0.7) > gem_plugin (0.2.3) > hpricot (0.8.1, 0.6) > json (1.1.6, 1.1.3) > marc (0.2.2) > mongrel (1.1.5) > mongrel_cluster (1.0.5) > mysql (2.7) > rails (2.3.2, 2.1.2) > rake (0.8.7) > ruby-json (1.1.2) > rubygems-update (1.3.3) > > Thanks again, > Scot Dalton > Phone: (212) 998-2674 > Web Development > Division of Libraries > New York University > > On Nov 12, 2009, at Nov 12, 3:21 PM, Jonathan Rochkind wrote: > > >> Well, okay, here's some clues. remove_value(issn) does this: >> >> def remove_value(key) >> referent_values.find(:all, :conditions=> ['key_name =?', key]).each >> do |rv| >> referent_values.delete(rv) >> end >> end >> >> >> So basically, it looks through the to-many referent_values >> relationship for a ReferentValue whose key_name is "issn", and when >> it finds one, it uses the standard Rails delete() to delete it. >> >> I was initially surprised that was ending up creating SQL ""UPDATE >> `referent_values` SET referent_id = NULL ...." at all, I thought I >> was actually deleting the ReferentValue from the db entirely. But >> then I looked up ActiveRecord documentation: >> >> collection.delete(object, ?) >> Removes one or more objects from the collection by setting their >> foreign keys to NULL. Objects will be in addition destroyed if >> they?re associated with :dependent => :destroy, and deleted if >> they?re associated with :dependent => :delete_all. >> So that explains why Rails is setting the foreign key to null! >> >> Now, why that raises in your Umlaut but not mine, I dunno. >> >> When I wrote the code, I thought that I was actually deleting the >> ReferentValue from the db, not just removing it from the >> association! But I'm apparently not, it's leaving all those orphaned >> ReferentValues in the db! Which doesn't sound so good! Except when I >> look in my actual db, I _don't_ have any ReferentValues with nil >> referent_ids. So, now I'm confused. >> >> I'm not sure how to fix this, I'm not sure what the 'right' way to >> both delete a ReferentValue in the db _and_ make sure it's removed >> from the in-memory association is, in Rails? It looks like I'd make >> sure it was really removed from the db if I continued to delete(), >> but just changed the referent_values association decleration to be: >> >> has_many :referent_values, :dependent => :destroy [ or :dependent >> => :delete_all] >> >> But I think that would still FIRST null out the referent_id, and >> THEN delete it. I wonder why that triggers MySQL constraint >> violations in your MySQL and not mine? The referent_values/ >> referent_id column in my db is defined like this: >> >> | Field | Type | Null | Key | Default | Extra | >> | referent_id | int(11) | NO | MUL | 0 | | >> >> [ And yes, I just checked to see if I had a bunch of ReferentValues >> with 0 for referent_id too. I do not. I do not seem to be getting >> orphaned ReferentValues. ] >> >> So hopefully this is some clues. If you figure out what's up and fix >> it, feel free to commit the changes. >> >> Jonathan >> >> >> Scot Dalton wrote: >> >>> Hi, >>> >>> We're experiencing an error in the DissertationCatch >>> ReferentFilter. We're getting the following SQL error message >>> when calling 'referent.remove_value("issn")': >>> "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column >>> 'referent_id' cannot be null: UPDATE `referent_values` SET >>> referent_id = NULL WHERE (referent_id = 234925 AND id IN >>> (6543983)) )" >>> >>> It doesn't appear that either JHU or New School are having a >>> similar problem. We've been experiencing some issues with our >>> MySQL instance, which we hope to resolve shortly. That may be >>> causing our problem, but if anyone has any ideas, I'd appreciate >>> it. Not sure if the MySQL instance is even the problem. >>> >>> No problem at JHU and New School: >>> http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 >>> http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 >>> >>> Problem at NYU: >>> http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 >>> >>> >>> Thanks, >>> Scot Dalton >>> Phone: (212) 998-2674 >>> Web Development >>> Division of Libraries >>> New York University >>> >>> _______________________________________________ >>> Umlaut-general mailing list >>> Umlaut-general at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/umlaut-general >>> >>> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From std5 at nyu.edu Thu Nov 12 17:18:34 2009 From: std5 at nyu.edu (Scot Dalton) Date: Thu, 12 Nov 2009 17:18:34 -0500 Subject: [Umlaut-general] DissertationCatch Error In-Reply-To: <4AFC7FFC.50508@jhu.edu> References: <4AFC6E4D.9030703@jhu.edu> <69056D57-ADD8-495D-B285-B0F0CDF2BDE1@nyu.edu> <4AFC7FFC.50508@jhu.edu> Message-ID: <6B10BE2F-2B8F-46E7-AA61-EB9013DE538B@nyu.edu> Wow! That seemed to somehow miraculously solve my problem. I'll continue to test before doing anything. Thanks! Scot Dalton Phone: (212) 998-2674 Web Development Division of Libraries New York University On Nov 12, 2009, at Nov 12, 4:37 PM, Jonathan Rochkind wrote: > Also try adding this to Referent.rb: > > Replace: > > has_many :referent_values > > With: > has_many :referent_values, :dependent => :delete_all. > If that somehow miraculously fixes your problem, that's fine to > commit, I think. (Test it a _bit_ first. :) ). > > Gem versions should all be more or less _fixed_ by Umlaut, it should > _insist_ on using the same gem versions for all of us. (Including > Rails 2.1.2, that's why you ended up with that one installed). > > Wonder if it could be different mysql versions. But we don't want > to insist on a specific mysql version, we want to fix the code to > work with any of em. > > Jonathan > > Scot Dalton wrote: >> Thanks for this. I'll continue to look through it and see what I >> can find. >> >> In the interim, I may just use the ensure_value! method and set the >> rv to an empty string. That seems to work in my tests. >> >> As for why I see the problem and you guys don't, could it be a >> version thing? (Obviously grasping at straws here.) >> We're running: >> ruby 1.8.6 (2008-08-11 patchlevel 287) [sparc-solaris2.10] >> gem 1.3.3 >> actionmailer (2.3.2, 2.1.2) >> actionpack (2.3.2, 2.1.2) >> activerecord (2.3.2, 2.1.2) >> activeresource (2.3.2, 2.1.2) >> activesupport (2.3.2, 2.1.2) >> cgi_multipart_eof_fix (2.5.0) >> daemons (1.0.10) >> fastthread (1.0.7) >> gem_plugin (0.2.3) >> hpricot (0.8.1, 0.6) >> json (1.1.6, 1.1.3) >> marc (0.2.2) >> mongrel (1.1.5) >> mongrel_cluster (1.0.5) >> mysql (2.7) >> rails (2.3.2, 2.1.2) >> rake (0.8.7) >> ruby-json (1.1.2) >> rubygems-update (1.3.3) >> >> Thanks again, >> Scot Dalton >> Phone: (212) 998-2674 >> Web Development >> Division of Libraries >> New York University >> >> On Nov 12, 2009, at Nov 12, 3:21 PM, Jonathan Rochkind wrote: >> >> >>> Well, okay, here's some clues. remove_value(issn) does this: >>> >>> def remove_value(key) >>> referent_values.find(:all, :conditions=> ['key_name =?', >>> key]).each do |rv| >>> referent_values.delete(rv) >>> end >>> end >>> >>> >>> So basically, it looks through the to-many referent_values >>> relationship for a ReferentValue whose key_name is "issn", and >>> when it finds one, it uses the standard Rails delete() to delete >>> it. >>> >>> I was initially surprised that was ending up creating SQL >>> ""UPDATE `referent_values` SET referent_id = NULL ...." at all, I >>> thought I was actually deleting the ReferentValue from the db >>> entirely. But then I looked up ActiveRecord documentation: >>> >>> collection.delete(object, ?) >>> Removes one or more objects from the collection by setting their >>> foreign keys to NULL. Objects will be in addition destroyed if >>> they?re associated with :dependent => :destroy, and deleted if >>> they?re associated with :dependent => :delete_all. >>> So that explains why Rails is setting the foreign key to null! >>> >>> Now, why that raises in your Umlaut but not mine, I dunno. >>> >>> When I wrote the code, I thought that I was actually deleting the >>> ReferentValue from the db, not just removing it from the >>> association! But I'm apparently not, it's leaving all those >>> orphaned ReferentValues in the db! Which doesn't sound so good! >>> Except when I look in my actual db, I _don't_ have any >>> ReferentValues with nil referent_ids. So, now I'm confused. >>> >>> I'm not sure how to fix this, I'm not sure what the 'right' way >>> to both delete a ReferentValue in the db _and_ make sure it's >>> removed from the in-memory association is, in Rails? It looks >>> like I'd make sure it was really removed from the db if I >>> continued to delete(), but just changed the referent_values >>> association decleration to be: >>> >>> has_many :referent_values, :dependent => :destroy [ or :dependent >>> => :delete_all] >>> >>> But I think that would still FIRST null out the referent_id, and >>> THEN delete it. I wonder why that triggers MySQL constraint >>> violations in your MySQL and not mine? The referent_values/ >>> referent_id column in my db is defined like this: >>> >>> | Field | Type | Null | Key | Default | Extra | >>> | referent_id | int(11) | NO | MUL | 0 | | >>> >>> [ And yes, I just checked to see if I had a bunch of >>> ReferentValues with 0 for referent_id too. I do not. I do not >>> seem to be getting orphaned ReferentValues. ] >>> >>> So hopefully this is some clues. If you figure out what's up and >>> fix it, feel free to commit the changes. >>> >>> Jonathan >>> >>> >>> Scot Dalton wrote: >>> >>>> Hi, >>>> >>>> We're experiencing an error in the DissertationCatch >>>> ReferentFilter. We're getting the following SQL error message >>>> when calling 'referent.remove_value("issn")': >>>> "FATAL: ActiveRecord::StatementInvalid (Mysql::Error: Column >>>> 'referent_id' cannot be null: UPDATE `referent_values` SET >>>> referent_id = NULL WHERE (referent_id = 234925 AND id IN >>>> (6543983)) )" >>>> >>>> It doesn't appear that either JHU or New School are having a >>>> similar problem. We've been experiencing some issues with our >>>> MySQL instance, which we hope to resolve shortly. That may be >>>> causing our problem, but if anyone has any ideas, I'd >>>> appreciate it. Not sure if the MySQL instance is even the >>>> problem. >>>> >>>> No problem at JHU and New School: >>>> http://findit.library.jhu.edu/resolve?rft.issn=0419-4217 >>>> http://getit.library.newschool.edu/resolve?rft.issn=0419-4217 >>>> >>>> Problem at NYU: >>>> http://getit.library.nyu.edu/resolve?rft.issn=0419-4217 >>>> >>>> >>>> Thanks, >>>> Scot Dalton >>>> Phone: (212) 998-2674 >>>> Web Development >>>> Division of Libraries >>>> New York University >>>> >>>> _______________________________________________ >>>> Umlaut-general mailing list >>>> Umlaut-general at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/umlaut-general >>>> >>>> >>> _______________________________________________ >>> Umlaut-general mailing list >>> Umlaut-general at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/umlaut-general >>> >> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general From std5 at nyu.edu Thu Nov 12 18:26:57 2009 From: std5 at nyu.edu (Scot Dalton) Date: Thu, 12 Nov 2009 18:26:57 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems Message-ID: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> Hi, I was having a problem with our SFX MySQL instance not liking the SQL the Umlaut search_controller was generating to render the A-Z list. The SFX database seemed to get confused by table aliasing in some cases. I updated the search controller to not use table aliases and was going to email the list when I came across a weirder case. Basically the following seems to be a valid scenario in the SFX database: a journal exists in the AZ_TITLE_VER3 table with a non-null OBJECT_ID, but there is no corresponding record in the OBJECT table. I don't manage our SFX instance, but my understanding is that the AZ_TITLE_VER3 table gets populated by a periodic cron and that "orphaned title" scenario can occur when objects are removed before the next running of the cron. Anyway, the Umlaut search controller SQL doesn't account for this scenario, presumably because the extra join to the OBJECT table would slow things down. Rails didn't like this scenario at all. ActiveRecord reported a hit mismatch when Umlaut passed in ten IDs to the find method, but could only retrieve nine (or fewer) back. Umlaut rendered the error page. I don't think this scenario occurs that often or for extended periods (we happened to have turned the cron off for a few days and the orphaned A-Z objects built up), so it may not be necessary to make the extra join. So my questions are: 1. Should I commit the un-table aliased SQL to svn? 2. Is the above scenario a valid concern that Umlaut should account for and make the additional join? 3. If not, should we at NYU consider increasing the frequency that we rebuild the A-Z index? We're currently running it nightly. Let me know if this makes sense. It may be easier to just send code snippets. Thanks, Scot Dalton Phone: (212) 998-2674 Web Development Division of Libraries New York University From rochkind at jhu.edu Mon Nov 16 16:23:08 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Mon, 16 Nov 2009 16:23:08 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> Message-ID: <4B01C2BC.1010808@jhu.edu> Yeah, the direct access to the SFX db is a bit weird in general, as it's based on reverse engineering of the SFX db without much documentation, and is subject to break if SFX updates change the SFX db schema. I'm not sure if the current problems you're seeing are consequences of an SFX update, or just edge cases I never noticed before. I don't understand your case where a journal exists in the AZ_TITLE_VER3 table without a corresponding record in the SFX Object table. I suspect this is actually database corruption on the SFX side (not neccesarily local corruption, it could be shared corruption in the shared Ex Libris-provided data). I don't think there's anything that Umlaut (or SFX itself) can usefully do with such an entry -- when the user clicks on it, what are they going to get? Since there is in fact no corresonding SFX object. So _something_ needs to be changed, since Umlaut is raising on it. If I'm right that these things should in fact be ignored, than we should perhaps, as you suggest, change our SQL query to join on the Object table, such that rows without a corresponding Object will be _left out_ of the query. I'm not sure what the correct 'spec' here is, but it sounds like both of us suspect that would be the correct behavior. So if that seems like an appropriate solution to you, and you're willing to write and test the code, then yes, please feel free to go ahead and commit it! One part of 'testing' should be making sure performance isn't horribly impacted by the extra join -- that the SQL query isn't much slower becuase of it. Likewise, if the SQL without table aliases fixes your problem and as far as you can tell works identically in all cases (it's just another way of writing the exact same SQL), feel free to commit that. I'm not entirely sure what's going on either! But I trust you, Scot. Meanwhile, some comments on an alternate possible future approach for A-Z list in general.... Long term, it might make sense to move away from having Umlaut directly query the SFX MySQL db. An alternate solution is to have Umlaut periodically take an 'export' of the SFX db, and cache it locally, for A-Z lookup functionality. That's in fact what Umlaut used to do, way back in the day. It might make more sense. But there are some tricks there to, as far as how Umlaut is going to get that data export. What Umlaut used to do is actually use the SFX "Export" feature, but the problem is there's no good way to automate that remotely, so the Umlaut local cache of SFX A-Z data had to be done manually. There might be a way around that by having Umlaut do some hacky ssh things in system calls. I'm also not sure the SFX manual "export" actually includes all the neccesary data to mimic A-Z functionality (like alternate names fo titles and such). Another option is having Umlaut use the export XML file that SFX prepares for Google Scholar. This is the approach David Walker takes with Xerxes, to create a local cache of SFX title information. I'm cautious of that because I don't entirely trust the Google Scholar XML export -- anecdotally, ti's seemed to me to often be wrong or incomplete, but I've never had the time to debug it to figure out why. I'm also not sure if the Google Scholar XML export again has all the neccesary data to mimic SFX A-Z functionality. And a last option would be to continue to get the data by a direct SQL connection, but to do it in bulk, and cache all the data in a local Umlaut table. This would still require an SQL connection, but the SQL connection would happen "off line" at regular times, not "on line" as part of every individual A-Z transaction, so the dependencies would be more minimal. I lay these out just to share my thinking on the subject; I don't think I'll have time to do any of those options any time soon, but if Scot or anyone else is interested in investigating, it could be useful. Jonathan Scot Dalton wrote: > Hi, > > I was having a problem with our SFX MySQL instance not liking the SQL > the Umlaut search_controller was generating to render the A-Z list. > The SFX database seemed to get confused by table aliasing in some > cases. I updated the search controller to not use table aliases and > was going to email the list when I came across a weirder case. > > Basically the following seems to be a valid scenario in the SFX > database: a journal exists in the AZ_TITLE_VER3 table with a non-null > OBJECT_ID, but there is no corresponding record in the OBJECT table. > I don't manage our SFX instance, but my understanding is that the > AZ_TITLE_VER3 table gets populated by a periodic cron and that > "orphaned title" scenario can occur when objects are removed before > the next running of the cron. > > Anyway, the Umlaut search controller SQL doesn't account for this > scenario, presumably because the extra join to the OBJECT table would > slow things down. Rails didn't like this scenario at all. > ActiveRecord reported a hit mismatch when Umlaut passed in ten IDs to > the find method, but could only retrieve nine (or fewer) back. Umlaut > rendered the error page. > > I don't think this scenario occurs that often or for extended periods > (we happened to have turned the cron off for a few days and the > orphaned A-Z objects built up), so it may not be necessary to make the > extra join. > > So my questions are: > 1. Should I commit the un-table aliased SQL to svn? > 2. Is the above scenario a valid concern that Umlaut should account > for and make the additional join? > 3. If not, should we at NYU consider increasing the frequency that we > rebuild the A-Z index? We're currently running it nightly. > > > Let me know if this makes sense. It may be easier to just send code > snippets. > > Thanks, > Scot Dalton > Phone: (212) 998-2674 > Web Development > Division of Libraries > New York University > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Tue Nov 17 16:17:52 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Tue, 17 Nov 2009 16:17:52 -0500 Subject: [Umlaut-general] Fwd: Gathering Xerxes impact stats WAS: [xerxes-portal] Impact on SFX statistics In-Reply-To: References: <6E8B5FF82A76874DAF6BDD684C9F76D65F44104BED@COWEWEXMB01.csuco> Message-ID: <4B031300.7090104@jhu.edu> Okay, there are a couple different things. I'm going to forward this reply to the Umlaut listserv too, since it's of possibly general interest. So if you're using Umlaut, then all link resolver menu displays that go through Umlaut will make it to SFX as "API" requests. This has two consequences: 1) In your SFX report generation, you need to make sure to select "multi_obj_xml" under "Formats". The default is just having "html" selected here, but that means your report will only include browsers that went straight to SFX, it won't include any Umlaut requests. Selecting both "html" and "multi_obj_request" will give you both. 2) So that's not so bad, but it gets worse. Extensive use of the SFX API completely messes up SFX "clickthrough" reports. Every single API request is recorded by SFX as both a "request" and a simultaneous "clickthrough" for every single target included in the menu generated. This is not unique to Umlaut, it's true of any SFX API use. This means that if you are looking at format:multi_obj_xml in your report, clickthroughs are entirely unreliable, there is no easy way to get anything useful out of them. And if you're using Umlaut, then all (or nearly all) of your SFX use is multi_obj_xml, so you don't have reliable clickthrough stats. The "request" stats, as opposed to "clickthrough" stats are, as far as I know, still reliable. But you need to make sure to include format:multi_obj_xml in your query when you set up your report. I have reported the bug with SFX API use and clickthrough statistics to Ex Libris, but they didn't seem very concerned about it, and didn't even really want to admit it was a bug. Perhaps if NYU conveyed their dissatisfaction with this too, Ex Libris might consider addressing it -- but I think there's no way it'll be addressed before SFX 4.0. If we're lucky, it won't be a problem in SFX 4.0. Jonathan Gloria Rohmann wrote: > Begin forwarded message: > Hi-- > > We're using umlaut, and it seems to be skewing our SFX stats (ie, very few Primo article SIDS are making it into the SFX requests by source report). Yet the number of remote searches in Primo has not changed (according to Primo stats). > > Is there something we can do about that? > > Thanks. > > > From: "Walker, David" > > Date: November 17, 2009 3:22:32 PM EST > To: Gloria Rohmann > > Cc: Nadaleen F Tempelman-Kluit > > Subject: RE: Gathering Xerxes impact stats WAS: [xerxes-portal] Impact on SFX statistics > > Just to complicate things further, we're also using > umlaut to deliver SFX services. > > Yeah, Jonathan Rochkind just mentioned to me in a separate email that using Umlaut in some way skews the stats reports in SFX. Not sure how or why exactly, but he might be able to shed some light on the numbers. > > --Dave > > ================== > David Walker > Library Web Services Manager > California State University > http://xerxes.calstate.edu > ________________________________________ > From: Gloria Rohmann [gr1 at nyu.edu] > Sent: Tuesday, November 17, 2009 11:42 AM > To: Walker, David > Cc: Nadaleen F Tempelman-Kluit > Subject: Re: Gathering Xerxes impact stats WAS: [xerxes-portal] Impact on SFX statistics > > Yes, we're still using Primo with the ML search provided via a > separate search tab. We have recently reconfigured and renamed our > Quick Sets to mirror our categories as shown in Xerxes. It will be > interesting to see if this changes the picture. > > Just to complicate things further, we're also using umlaut to deliver > SFX services. This was also rolled out this summer. It's possible that > this is having an effect on what SIDs get logged. A quick check of our > Primo stats shows that the number of "remote" searches done in Feb is > about the same as in October! > > Here are the 2 reports; Feb 2009 & Oct 2009. > > --Gloria > > Gloria Rohmann > Digital Access Librarian > NYU Libraries > gr1 at nyu.edu > 212-998-2534 > > > From rossfsinger at gmail.com Wed Nov 18 10:39:31 2009 From: rossfsinger at gmail.com (Ross Singer) Date: Wed, 18 Nov 2009 10:39:31 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <4B01C2BC.1010808@jhu.edu> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> Message-ID: <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind wrote: > Long term, it might make sense to move away from having Umlaut directly > query the SFX MySQL db. ?An alternate solution is to have Umlaut > periodically take an 'export' of the SFX db, and cache it locally, for A-Z > lookup functionality. ?That's in fact what Umlaut used to do, way back in > the day. It might make more sense. ?But there are some tricks there to, as > far as how Umlaut is going to get that data export. on would be to continue to get the data by a direct SQL > connection, but to do it in bulk, and cache all the data in a local Umlaut > table. ?This would still require an SQL connection, but the SQL connection > would happen "off line" at regular times, not "on line" as part of every > individual A-Z transaction, so the dependencies would be more minimal. Since this was my contribution, I might as well weigh in on it, since there is some real-world justification in keeping such a mechanism. The reason why we used it at GaTech was because: 1) SFX was hosted consortially: we had no MySQL access (which would be true of EL-hosted SFX, as well) 2) Our SFX instance actually managed a subset of the resources actually resolved: SFX would merge the results of the consortial instance (the resources available statewide, provided by GALILEO) and our local resources. So this wasn't the result of a query on a single table, no matter what. We were able to set up a nightly cron that exported the full title list from the GaTech instance (GALILEO's was monthly, because they did no editing between KB loads) and checked to see if there were any changes from day to day. From that, we pulled (in Python, Ruby was too slow for this) into the local database structure and provided a fulltext index with Ferret. To answer your question, Jonathan, yes it contains all of the alternate titles. With regards to David Walker's Scholar export solution, we were unable to use this, either. By default, SFX ships with a .htaccess only allowing Google and EL (maybe) access to the export file. GALILEO either didn't know how or wouldn't change this for us. Of course, on the other hand, we had strong suspicions that the Scholar export file wasn't properly integrating the GALILEO and GaTech holdings, anyway, so this might have been a non-starter (but I'll never know). I guess what I'm getting at is that Umlaut was originally created, more or less, as a reaction to having very little control over how we could make SFX work the way we wanted: both as a result of GALILEO and Ex Libris. I don't think this is an edge-case. I understand that given the fact that all of the current implementors have local SFX instances this isn't a top priority, but I do think it should be given some consideration in the future. -Ross. From rochkind at jhu.edu Wed Nov 18 11:16:28 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 18 Nov 2009 11:16:28 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> Message-ID: <4B041DDC.9050006@jhu.edu> Basically, the work I do on Umlaut is largely based on what I need locally. Same for Scot at NYU, I'd assume. Of course, as everyone has hopefully noticed, I try to spend time helping other people address their needs too. But at the time I worked on it, SQL database access seemed preferable to me. If someone else wants/needs another form of access and wants to code it, I'm happy to help/advise to help them figure out how to fit that into Umlaut properly, but at the moment I'm unlikely to be prioritizing it myself any time soon. While I recognize that many Library IT departments do behave irrationally and unreasonably.... I think it's arguably unreasonable refuse to provide a read-only MySQL account for email; and I think it's _decidedly_ unreasonable to refuse to edit the IP based restrictions to give Umlaut access to the Google Scholar export. (This latter one can not _possibly_ effect SFX functionality). But yeah, a lot of people live with unreasonableness. I tend to spend time writing code to deal with organizational unreasonableness with great reluctance, rather than with excitement though. Jonathan Ross Singer wrote: > On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind wrote: > >> Long term, it might make sense to move away from having Umlaut directly >> query the SFX MySQL db. An alternate solution is to have Umlaut >> periodically take an 'export' of the SFX db, and cache it locally, for A-Z >> lookup functionality. That's in fact what Umlaut used to do, way back in >> the day. It might make more sense. But there are some tricks there to, as >> far as how Umlaut is going to get that data export. >> > on would be to continue to get the data by a direct SQL > >> connection, but to do it in bulk, and cache all the data in a local Umlaut >> table. This would still require an SQL connection, but the SQL connection >> would happen "off line" at regular times, not "on line" as part of every >> individual A-Z transaction, so the dependencies would be more minimal. >> > > Since this was my contribution, I might as well weigh in on it, since > there is some real-world justification in keeping such a mechanism. > > The reason why we used it at GaTech was because: > > 1) SFX was hosted consortially: we had no MySQL access (which would > be true of EL-hosted SFX, as well) > 2) Our SFX instance actually managed a subset of the resources > actually resolved: SFX would merge the results of the consortial > instance (the resources available statewide, provided by GALILEO) and > our local resources. So this wasn't the result of a query on a single > table, no matter what. > > We were able to set up a nightly cron that exported the full title > list from the GaTech instance (GALILEO's was monthly, because they did > no editing between KB loads) and checked to see if there were any > changes from day to day. From that, we pulled (in Python, Ruby was > too slow for this) into the local database structure and provided a > fulltext index with Ferret. > > To answer your question, Jonathan, yes it contains all of the alternate titles. > > With regards to David Walker's Scholar export solution, we were unable > to use this, either. By default, SFX ships with a .htaccess only > allowing Google and EL (maybe) access to the export file. GALILEO > either didn't know how or wouldn't change this for us. > > Of course, on the other hand, we had strong suspicions that the > Scholar export file wasn't properly integrating the GALILEO and GaTech > holdings, anyway, so this might have been a non-starter (but I'll > never know). > > I guess what I'm getting at is that Umlaut was originally created, > more or less, as a reaction to having very little control over how we > could make SFX work the way we wanted: both as a result of GALILEO and > Ex Libris. I don't think this is an edge-case. I understand that > given the fact that all of the current implementors have local SFX > instances this isn't a top priority, but I do think it should be given > some consideration in the future. > > -Ross. > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rossfsinger at gmail.com Wed Nov 18 13:15:18 2009 From: rossfsinger at gmail.com (Ross Singer) Date: Wed, 18 Nov 2009 13:15:18 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <4B041DDC.9050006@jhu.edu> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> <4B041DDC.9050006@jhu.edu> Message-ID: <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> Well, I think I made the point that I didn't figure this was an immediate priority. But it does provide an appropriate level of abstraction for: 1. SFX Consortial setups like GALILEO (which I assume aren't uncommon) -- your solution won't work even with IT cooperation 2. Changes in SFX's database structure. 3. Non-SFX based resolvers. -Ross. On Wed, Nov 18, 2009 at 11:16 AM, Jonathan Rochkind wrote: > Basically, the work I do on Umlaut is largely based on what I need locally. > Same for Scot at NYU, I'd assume. > > Of course, as everyone has hopefully noticed, I try to spend time helping > other people address their needs too. > > But at the time I worked on it, SQL database access seemed preferable to me. > ?If someone else wants/needs another form of access and wants to code it, > I'm happy to help/advise to help them figure out how to fit that into Umlaut > properly, but at the moment I'm unlikely to be prioritizing it myself any > time soon. > > While I recognize that many Library IT departments do behave irrationally > and unreasonably.... ?I think it's arguably unreasonable refuse to provide a > read-only MySQL account for email; and I think it's _decidedly_ unreasonable > to refuse to edit the IP based restrictions to give Umlaut access to the > Google Scholar export. ?(This latter one can not _possibly_ effect SFX > functionality). ?But yeah, a lot of people live with unreasonableness. ? I > tend to spend time writing code to deal with organizational unreasonableness > with great reluctance, rather than with excitement though. > > Jonathan > > Ross Singer wrote: >> >> On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind >> wrote: >> >>> >>> Long term, it might make sense to move away from having Umlaut directly >>> query the SFX MySQL db. ?An alternate solution is to have Umlaut >>> periodically take an 'export' of the SFX db, and cache it locally, for >>> A-Z >>> lookup functionality. ?That's in fact what Umlaut used to do, way back in >>> the day. It might make more sense. ?But there are some tricks there to, >>> as >>> far as how Umlaut is going to get that data export. >>> >> >> on would be to continue to get the data by a direct SQL >> >>> >>> connection, but to do it in bulk, and cache all the data in a local >>> Umlaut >>> table. ?This would still require an SQL connection, but the SQL >>> connection >>> would happen "off line" at regular times, not "on line" as part of every >>> individual A-Z transaction, so the dependencies would be more minimal. >>> >> >> Since this was my contribution, I might as well weigh in on it, since >> there is some real-world justification in keeping such a mechanism. >> >> The reason why we used it at GaTech was because: >> >> 1) SFX was hosted consortially: ?we had no MySQL access (which would >> be true of EL-hosted SFX, as well) >> 2) Our SFX instance actually managed a subset of the resources >> actually resolved: ?SFX would merge the results of the consortial >> instance (the resources available statewide, provided by GALILEO) and >> our local resources. ?So this wasn't the result of a query on a single >> table, no matter what. >> >> We were able to set up a nightly cron that exported the full title >> list from the GaTech instance (GALILEO's was monthly, because they did >> no editing between KB loads) and checked to see if there were any >> changes from day to day. ?From that, we pulled (in Python, Ruby was >> too slow for this) into the local database structure and provided a >> fulltext index with Ferret. >> >> To answer your question, Jonathan, yes it contains all of the alternate >> titles. >> >> With regards to David Walker's Scholar export solution, we were unable >> to use this, either. ?By default, SFX ships with a .htaccess only >> allowing Google and EL (maybe) access to the export file. ?GALILEO >> either didn't know how or wouldn't change this for us. >> >> Of course, on the other hand, we had strong suspicions that the >> Scholar export file wasn't properly integrating the GALILEO and GaTech >> holdings, anyway, so this might have been a non-starter (but I'll >> never know). >> >> I guess what I'm getting at is that Umlaut was originally created, >> more or less, as a reaction to having very little control over how we >> could make SFX work the way we wanted: both as a result of GALILEO and >> Ex Libris. ?I don't think this is an edge-case. ?I understand that >> given the fact that all of the current implementors have local SFX >> instances this isn't a top priority, but I do think it should be given >> some consideration in the future. >> >> -Ross. >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Wed Nov 18 13:19:41 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 18 Nov 2009 13:19:41 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> <4B041DDC.9050006@jhu.edu> <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> Message-ID: <4B043ABD.1090208@jhu.edu> Yep, you're right especially on #2 and #3. Ross Singer wrote: > Well, I think I made the point that I didn't figure this was an > immediate priority. > > But it does provide an appropriate level of abstraction for: > 1. SFX Consortial setups like GALILEO (which I assume aren't uncommon) > -- your solution won't work even with IT cooperation > 2. Changes in SFX's database structure. > 3. Non-SFX based resolvers. > > -Ross. > > On Wed, Nov 18, 2009 at 11:16 AM, Jonathan Rochkind wrote: > >> Basically, the work I do on Umlaut is largely based on what I need locally. >> Same for Scot at NYU, I'd assume. >> >> Of course, as everyone has hopefully noticed, I try to spend time helping >> other people address their needs too. >> >> But at the time I worked on it, SQL database access seemed preferable to me. >> If someone else wants/needs another form of access and wants to code it, >> I'm happy to help/advise to help them figure out how to fit that into Umlaut >> properly, but at the moment I'm unlikely to be prioritizing it myself any >> time soon. >> >> While I recognize that many Library IT departments do behave irrationally >> and unreasonably.... I think it's arguably unreasonable refuse to provide a >> read-only MySQL account for email; and I think it's _decidedly_ unreasonable >> to refuse to edit the IP based restrictions to give Umlaut access to the >> Google Scholar export. (This latter one can not _possibly_ effect SFX >> functionality). But yeah, a lot of people live with unreasonableness. I >> tend to spend time writing code to deal with organizational unreasonableness >> with great reluctance, rather than with excitement though. >> >> Jonathan >> >> Ross Singer wrote: >> >>> On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind >>> wrote: >>> >>> >>>> Long term, it might make sense to move away from having Umlaut directly >>>> query the SFX MySQL db. An alternate solution is to have Umlaut >>>> periodically take an 'export' of the SFX db, and cache it locally, for >>>> A-Z >>>> lookup functionality. That's in fact what Umlaut used to do, way back in >>>> the day. It might make more sense. But there are some tricks there to, >>>> as >>>> far as how Umlaut is going to get that data export. >>>> >>>> >>> on would be to continue to get the data by a direct SQL >>> >>> >>>> connection, but to do it in bulk, and cache all the data in a local >>>> Umlaut >>>> table. This would still require an SQL connection, but the SQL >>>> connection >>>> would happen "off line" at regular times, not "on line" as part of every >>>> individual A-Z transaction, so the dependencies would be more minimal. >>>> >>>> >>> Since this was my contribution, I might as well weigh in on it, since >>> there is some real-world justification in keeping such a mechanism. >>> >>> The reason why we used it at GaTech was because: >>> >>> 1) SFX was hosted consortially: we had no MySQL access (which would >>> be true of EL-hosted SFX, as well) >>> 2) Our SFX instance actually managed a subset of the resources >>> actually resolved: SFX would merge the results of the consortial >>> instance (the resources available statewide, provided by GALILEO) and >>> our local resources. So this wasn't the result of a query on a single >>> table, no matter what. >>> >>> We were able to set up a nightly cron that exported the full title >>> list from the GaTech instance (GALILEO's was monthly, because they did >>> no editing between KB loads) and checked to see if there were any >>> changes from day to day. From that, we pulled (in Python, Ruby was >>> too slow for this) into the local database structure and provided a >>> fulltext index with Ferret. >>> >>> To answer your question, Jonathan, yes it contains all of the alternate >>> titles. >>> >>> With regards to David Walker's Scholar export solution, we were unable >>> to use this, either. By default, SFX ships with a .htaccess only >>> allowing Google and EL (maybe) access to the export file. GALILEO >>> either didn't know how or wouldn't change this for us. >>> >>> Of course, on the other hand, we had strong suspicions that the >>> Scholar export file wasn't properly integrating the GALILEO and GaTech >>> holdings, anyway, so this might have been a non-starter (but I'll >>> never know). >>> >>> I guess what I'm getting at is that Umlaut was originally created, >>> more or less, as a reaction to having very little control over how we >>> could make SFX work the way we wanted: both as a result of GALILEO and >>> Ex Libris. I don't think this is an edge-case. I understand that >>> given the fact that all of the current implementors have local SFX >>> instances this isn't a top priority, but I do think it should be given >>> some consideration in the future. >>> >>> -Ross. >>> _______________________________________________ >>> Umlaut-general mailing list >>> Umlaut-general at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/umlaut-general >>> >>> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rochkind at jhu.edu Wed Nov 18 13:24:48 2009 From: rochkind at jhu.edu (Jonathan Rochkind) Date: Wed, 18 Nov 2009 13:24:48 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> <4B041DDC.9050006@jhu.edu> <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> Message-ID: <4B043BF0.6010801@jhu.edu> On #1, I'm still suspicious, but, hey, I'm not in that situation, if someone else shows up in that situation who wants to use Umalut... it would give me more encouragement to work on it to meet their real-world needs. Xerxes has been adopted by a bunch of institutions, including consortial institutions, the need to access the Google Scholar XML import doesn't seem to have been a barrier. And I _think_ there will be different google scholar XML exports for each institution in a consortium, if that's what's desired. But of course Umlaut doesn't do google scholar right now either! And the direct database access _could_ be modified to deal with consortial stuff but that would be code too. But you're right in general. Even when I added the direct db stuff in Umlaut, I always intended that to be an option, and allow the way you had been doing it too. There are config parameters still there that theoretically allow you to use your original way. It's just that they probably don't work, since I never really used them or tested them, and I'm not sure if the 90% finished Umlaut2 code I inherited ever had that actually complete. But I tried to leave the hooks there for future expansion. I definitely agree that ultimately Umlaut should support various means of bulk-loading A-Z data to a local Umlaut db, from various external sources. I just personally probably won't attend to it until I have a use case I need. Jonathan Ross Singer wrote: > Well, I think I made the point that I didn't figure this was an > immediate priority. > > But it does provide an appropriate level of abstraction for: > 1. SFX Consortial setups like GALILEO (which I assume aren't uncommon) > -- your solution won't work even with IT cooperation > 2. Changes in SFX's database structure. > 3. Non-SFX based resolvers. > > -Ross. > > On Wed, Nov 18, 2009 at 11:16 AM, Jonathan Rochkind wrote: > >> Basically, the work I do on Umlaut is largely based on what I need locally. >> Same for Scot at NYU, I'd assume. >> >> Of course, as everyone has hopefully noticed, I try to spend time helping >> other people address their needs too. >> >> But at the time I worked on it, SQL database access seemed preferable to me. >> If someone else wants/needs another form of access and wants to code it, >> I'm happy to help/advise to help them figure out how to fit that into Umlaut >> properly, but at the moment I'm unlikely to be prioritizing it myself any >> time soon. >> >> While I recognize that many Library IT departments do behave irrationally >> and unreasonably.... I think it's arguably unreasonable refuse to provide a >> read-only MySQL account for email; and I think it's _decidedly_ unreasonable >> to refuse to edit the IP based restrictions to give Umlaut access to the >> Google Scholar export. (This latter one can not _possibly_ effect SFX >> functionality). But yeah, a lot of people live with unreasonableness. I >> tend to spend time writing code to deal with organizational unreasonableness >> with great reluctance, rather than with excitement though. >> >> Jonathan >> >> Ross Singer wrote: >> >>> On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind >>> wrote: >>> >>> >>>> Long term, it might make sense to move away from having Umlaut directly >>>> query the SFX MySQL db. An alternate solution is to have Umlaut >>>> periodically take an 'export' of the SFX db, and cache it locally, for >>>> A-Z >>>> lookup functionality. That's in fact what Umlaut used to do, way back in >>>> the day. It might make more sense. But there are some tricks there to, >>>> as >>>> far as how Umlaut is going to get that data export. >>>> >>>> >>> on would be to continue to get the data by a direct SQL >>> >>> >>>> connection, but to do it in bulk, and cache all the data in a local >>>> Umlaut >>>> table. This would still require an SQL connection, but the SQL >>>> connection >>>> would happen "off line" at regular times, not "on line" as part of every >>>> individual A-Z transaction, so the dependencies would be more minimal. >>>> >>>> >>> Since this was my contribution, I might as well weigh in on it, since >>> there is some real-world justification in keeping such a mechanism. >>> >>> The reason why we used it at GaTech was because: >>> >>> 1) SFX was hosted consortially: we had no MySQL access (which would >>> be true of EL-hosted SFX, as well) >>> 2) Our SFX instance actually managed a subset of the resources >>> actually resolved: SFX would merge the results of the consortial >>> instance (the resources available statewide, provided by GALILEO) and >>> our local resources. So this wasn't the result of a query on a single >>> table, no matter what. >>> >>> We were able to set up a nightly cron that exported the full title >>> list from the GaTech instance (GALILEO's was monthly, because they did >>> no editing between KB loads) and checked to see if there were any >>> changes from day to day. From that, we pulled (in Python, Ruby was >>> too slow for this) into the local database structure and provided a >>> fulltext index with Ferret. >>> >>> To answer your question, Jonathan, yes it contains all of the alternate >>> titles. >>> >>> With regards to David Walker's Scholar export solution, we were unable >>> to use this, either. By default, SFX ships with a .htaccess only >>> allowing Google and EL (maybe) access to the export file. GALILEO >>> either didn't know how or wouldn't change this for us. >>> >>> Of course, on the other hand, we had strong suspicions that the >>> Scholar export file wasn't properly integrating the GALILEO and GaTech >>> holdings, anyway, so this might have been a non-starter (but I'll >>> never know). >>> >>> I guess what I'm getting at is that Umlaut was originally created, >>> more or less, as a reaction to having very little control over how we >>> could make SFX work the way we wanted: both as a result of GALILEO and >>> Ex Libris. I don't think this is an edge-case. I understand that >>> given the fact that all of the current implementors have local SFX >>> instances this isn't a top priority, but I do think it should be given >>> some consideration in the future. >>> >>> -Ross. >>> _______________________________________________ >>> Umlaut-general mailing list >>> Umlaut-general at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/umlaut-general >>> >>> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> >> > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general > From rossfsinger at gmail.com Wed Nov 18 13:28:05 2009 From: rossfsinger at gmail.com (Ross Singer) Date: Wed, 18 Nov 2009 13:28:05 -0500 Subject: [Umlaut-general] SFX A-Z Ver 3 Problems In-Reply-To: <4B043BF0.6010801@jhu.edu> References: <016C56CE-98CB-4E3D-BB13-0D5DA0DFED65@nyu.edu> <4B01C2BC.1010808@jhu.edu> <23b83f160911180739l162d4b0bnec8691c5df97c6c3@mail.gmail.com> <4B041DDC.9050006@jhu.edu> <23b83f160911181015g72a163aj3240f83608509ec4@mail.gmail.com> <4B043BF0.6010801@jhu.edu> Message-ID: <23b83f160911181028o2fec300dxe61577ccbf54ee86@mail.gmail.com> I'm not trying to argue with you, I just explained the rationale for why GaTech housed the A-Z list internally in Umlaut :) -Ross. On Wed, Nov 18, 2009 at 1:24 PM, Jonathan Rochkind wrote: > On #1, I'm still suspicious, but, hey, I'm not in that situation, if someone > else shows up in that situation who wants to use Umalut... it would give me > more encouragement to work on it to meet their real-world needs. > > Xerxes has been adopted by a bunch of institutions, including consortial > institutions, the need to access the Google Scholar XML import doesn't seem > to have been a barrier. And I _think_ there will be different google scholar > XML exports for each institution in a consortium, if that's what's desired. > > But of course Umlaut doesn't do google scholar right now either! And the > direct database access _could_ be modified to deal with consortial stuff but > that would be code too. > > But you're right in general. Even when I added the direct db stuff in > Umlaut, I always intended that to be an option, and allow the way you had > been doing it too. There are config parameters still there that > theoretically allow you to use your original way. It's just that they > probably don't work, since I never really used them or tested them, and I'm > not sure if the 90% finished Umlaut2 code I inherited ever had that actually > complete. ?But I tried to leave the hooks there for future expansion. > > I definitely agree that ultimately Umlaut should support various means of > bulk-loading A-Z data to a local Umlaut db, from various external sources. I > just personally probably won't attend to it until I have a use case I need. > > Jonathan > > Ross Singer wrote: >> >> Well, I think I made the point that I didn't figure this was an >> immediate priority. >> >> But it does provide an appropriate level of abstraction for: >> 1. SFX Consortial setups like GALILEO (which I assume aren't uncommon) >> -- your solution won't work even with IT cooperation >> 2. Changes in SFX's database structure. >> 3. Non-SFX based resolvers. >> >> -Ross. >> >> On Wed, Nov 18, 2009 at 11:16 AM, Jonathan Rochkind >> wrote: >> >>> >>> Basically, the work I do on Umlaut is largely based on what I need >>> locally. >>> Same for Scot at NYU, I'd assume. >>> >>> Of course, as everyone has hopefully noticed, I try to spend time helping >>> other people address their needs too. >>> >>> But at the time I worked on it, SQL database access seemed preferable to >>> me. >>> ?If someone else wants/needs another form of access and wants to code it, >>> I'm happy to help/advise to help them figure out how to fit that into >>> Umlaut >>> properly, but at the moment I'm unlikely to be prioritizing it myself any >>> time soon. >>> >>> While I recognize that many Library IT departments do behave irrationally >>> and unreasonably.... ?I think it's arguably unreasonable refuse to >>> provide a >>> read-only MySQL account for email; and I think it's _decidedly_ >>> unreasonable >>> to refuse to edit the IP based restrictions to give Umlaut access to the >>> Google Scholar export. ?(This latter one can not _possibly_ effect SFX >>> functionality). ?But yeah, a lot of people live with unreasonableness. >>> I >>> tend to spend time writing code to deal with organizational >>> unreasonableness >>> with great reluctance, rather than with excitement though. >>> >>> Jonathan >>> >>> Ross Singer wrote: >>> >>>> >>>> On Mon, Nov 16, 2009 at 4:23 PM, Jonathan Rochkind >>>> wrote: >>>> >>>> >>>>> >>>>> Long term, it might make sense to move away from having Umlaut directly >>>>> query the SFX MySQL db. ?An alternate solution is to have Umlaut >>>>> periodically take an 'export' of the SFX db, and cache it locally, for >>>>> A-Z >>>>> lookup functionality. ?That's in fact what Umlaut used to do, way back >>>>> in >>>>> the day. It might make more sense. ?But there are some tricks there to, >>>>> as >>>>> far as how Umlaut is going to get that data export. >>>>> >>>>> >>>> >>>> on would be to continue to get the data by a direct SQL >>>> >>>> >>>>> >>>>> connection, but to do it in bulk, and cache all the data in a local >>>>> Umlaut >>>>> table. ?This would still require an SQL connection, but the SQL >>>>> connection >>>>> would happen "off line" at regular times, not "on line" as part of >>>>> every >>>>> individual A-Z transaction, so the dependencies would be more minimal. >>>>> >>>>> >>>> >>>> Since this was my contribution, I might as well weigh in on it, since >>>> there is some real-world justification in keeping such a mechanism. >>>> >>>> The reason why we used it at GaTech was because: >>>> >>>> 1) SFX was hosted consortially: ?we had no MySQL access (which would >>>> be true of EL-hosted SFX, as well) >>>> 2) Our SFX instance actually managed a subset of the resources >>>> actually resolved: ?SFX would merge the results of the consortial >>>> instance (the resources available statewide, provided by GALILEO) and >>>> our local resources. ?So this wasn't the result of a query on a single >>>> table, no matter what. >>>> >>>> We were able to set up a nightly cron that exported the full title >>>> list from the GaTech instance (GALILEO's was monthly, because they did >>>> no editing between KB loads) and checked to see if there were any >>>> changes from day to day. ?From that, we pulled (in Python, Ruby was >>>> too slow for this) into the local database structure and provided a >>>> fulltext index with Ferret. >>>> >>>> To answer your question, Jonathan, yes it contains all of the alternate >>>> titles. >>>> >>>> With regards to David Walker's Scholar export solution, we were unable >>>> to use this, either. ?By default, SFX ships with a .htaccess only >>>> allowing Google and EL (maybe) access to the export file. ?GALILEO >>>> either didn't know how or wouldn't change this for us. >>>> >>>> Of course, on the other hand, we had strong suspicions that the >>>> Scholar export file wasn't properly integrating the GALILEO and GaTech >>>> holdings, anyway, so this might have been a non-starter (but I'll >>>> never know). >>>> >>>> I guess what I'm getting at is that Umlaut was originally created, >>>> more or less, as a reaction to having very little control over how we >>>> could make SFX work the way we wanted: both as a result of GALILEO and >>>> Ex Libris. ?I don't think this is an edge-case. ?I understand that >>>> given the fact that all of the current implementors have local SFX >>>> instances this isn't a top priority, but I do think it should be given >>>> some consideration in the future. >>>> >>>> -Ross. >>>> _______________________________________________ >>>> Umlaut-general mailing list >>>> Umlaut-general at rubyforge.org >>>> http://rubyforge.org/mailman/listinfo/umlaut-general >>>> >>>> >>> >>> _______________________________________________ >>> Umlaut-general mailing list >>> Umlaut-general at rubyforge.org >>> http://rubyforge.org/mailman/listinfo/umlaut-general >>> >>> >> >> _______________________________________________ >> Umlaut-general mailing list >> Umlaut-general at rubyforge.org >> http://rubyforge.org/mailman/listinfo/umlaut-general >> > > _______________________________________________ > Umlaut-general mailing list > Umlaut-general at rubyforge.org > http://rubyforge.org/mailman/listinfo/umlaut-general >